Compare commits

..

No commits in common. "v2.11" and "master" have entirely different histories.

30 changed files with 230 additions and 1050 deletions

View File

@ -5,7 +5,7 @@ name: Test CI
on: on:
push: push:
branches: [main, repo-1, stable, maint] branches: [master, repo-1, stable, maint]
tags: [v*] tags: [v*]
jobs: jobs:

1
.gitignore vendored
View File

@ -7,7 +7,6 @@ __pycache__
.repopickle_* .repopickle_*
/repoc /repoc
/.tox /.tox
/.venv
# PyCharm related # PyCharm related
/.idea/ /.idea/

View File

@ -10,7 +10,7 @@
- Make corrections if requested. - Make corrections if requested.
- Verify your changes on gerrit so they can be submitted. - Verify your changes on gerrit so they can be submitted.
`git push https://gerrit-review.googlesource.com/git-repo HEAD:refs/for/main` `git push https://gerrit-review.googlesource.com/git-repo HEAD:refs/for/master`
# Long Version # Long Version
@ -150,7 +150,7 @@ Push your patches over HTTPS to the review server, possibly through
a remembered remote to make this easier in the future: a remembered remote to make this easier in the future:
git config remote.review.url https://gerrit-review.googlesource.com/git-repo git config remote.review.url https://gerrit-review.googlesource.com/git-repo
git config remote.review.push HEAD:refs/for/main git config remote.review.push HEAD:refs/for/master
git push review git push review

View File

@ -106,7 +106,7 @@ support, see the [manifest-format.md] file.
setting in the manifest (i.e. the path on the remote server) with a `.git` setting in the manifest (i.e. the path on the remote server) with a `.git`
suffix. This allows for multiple checkouts of the same remote git repo to suffix. This allows for multiple checkouts of the same remote git repo to
share their objects. For example, you could have different branches of share their objects. For example, you could have different branches of
`foo/bar.git` checked out to `foo/bar-main`, `foo/bar-release`, etc... `foo/bar.git` checked out to `foo/bar-master`, `foo/bar-release`, etc...
There will be multiple trees under `projects/` for each one, but only one There will be multiple trees under `projects/` for each one, but only one
under `project-objects/`. under `project-objects/`.

View File

@ -100,7 +100,6 @@ following DTD:
<!ELEMENT include EMPTY> <!ELEMENT include EMPTY>
<!ATTLIST include name CDATA #REQUIRED> <!ATTLIST include name CDATA #REQUIRED>
<!ATTLIST include groups CDATA #IMPLIED>
]> ]>
``` ```
@ -111,10 +110,6 @@ A description of the elements and their attributes follows.
The root element of the file. The root element of the file.
### Element notice
Arbitrary text that is displayed to users whenever `repo sync` finishes.
The content is simply passed through as it exists in the manifest.
### Element remote ### Element remote
@ -147,8 +142,8 @@ Attribute `review`: Hostname of the Gerrit server where reviews
are uploaded to by `repo upload`. This attribute is optional; are uploaded to by `repo upload`. This attribute is optional;
if not specified then `repo upload` will not function. if not specified then `repo upload` will not function.
Attribute `revision`: Name of a Git branch (e.g. `main` or Attribute `revision`: Name of a Git branch (e.g. `master` or
`refs/heads/main`). Remotes with their own revision will override `refs/heads/master`). Remotes with their own revision will override
the default revision. the default revision.
### Element default ### Element default
@ -161,11 +156,11 @@ Attribute `remote`: Name of a previously defined remote element.
Project elements lacking a remote attribute of their own will use Project elements lacking a remote attribute of their own will use
this remote. this remote.
Attribute `revision`: Name of a Git branch (e.g. `main` or Attribute `revision`: Name of a Git branch (e.g. `master` or
`refs/heads/main`). Project elements lacking their own `refs/heads/master`). Project elements lacking their own
revision attribute will use this revision. revision attribute will use this revision.
Attribute `dest-branch`: Name of a Git branch (e.g. `main`). Attribute `dest-branch`: Name of a Git branch (e.g. `master`).
Project elements not setting their own `dest-branch` will inherit Project elements not setting their own `dest-branch` will inherit
this value. If this value is not set, projects will use `revision` this value. If this value is not set, projects will use `revision`
by default instead. by default instead.
@ -252,13 +247,13 @@ If not supplied the remote given by the default element is used.
Attribute `revision`: Name of the Git branch the manifest wants Attribute `revision`: Name of the Git branch the manifest wants
to track for this project. Names can be relative to refs/heads to track for this project. Names can be relative to refs/heads
(e.g. just "main") or absolute (e.g. "refs/heads/main"). (e.g. just "master") or absolute (e.g. "refs/heads/master").
Tags and/or explicit SHA-1s should work in theory, but have not Tags and/or explicit SHA-1s should work in theory, but have not
been extensively tested. If not supplied the revision given by been extensively tested. If not supplied the revision given by
the remote element is used if applicable, else the default the remote element is used if applicable, else the default
element is used. element is used.
Attribute `dest-branch`: Name of a Git branch (e.g. `main`). Attribute `dest-branch`: Name of a Git branch (e.g. `master`).
When using `repo upload`, changes will be submitted for code When using `repo upload`, changes will be submitted for code
review on this branch. If unspecified both here and in the review on this branch. If unspecified both here and in the
default element, `revision` is used instead. default element, `revision` is used instead.
@ -267,7 +262,7 @@ Attribute `groups`: List of groups to which this project belongs,
whitespace or comma separated. All projects belong to the group whitespace or comma separated. All projects belong to the group
"all", and each project automatically belongs to a group of "all", and each project automatically belongs to a group of
its name:`name` and path:`path`. E.g. for its name:`name` and path:`path`. E.g. for
`<project name="monkeys" path="barrel-of"/>`, that project <project name="monkeys" path="barrel-of"/>, that project
definition is implicitly in the following manifest groups: definition is implicitly in the following manifest groups:
default, name:monkeys, and path:barrel-of. If you place a project in the default, name:monkeys, and path:barrel-of. If you place a project in the
group "notdefault", it will not be automatically downloaded by repo. group "notdefault", it will not be automatically downloaded by repo.
@ -364,19 +359,6 @@ This element is mostly useful in a local manifest file, where
the user can remove a project, and possibly replace it with their the user can remove a project, and possibly replace it with their
own definition. own definition.
### Element repo-hooks
NB: See the [practical documentation](./repo-hooks.md) for using repo hooks.
Only one repo-hooks element may be specified at a time.
Attempting to redefine it will fail to parse.
Attribute `in-project`: The project where the hooks are defined. The value
must match the `name` attribute (**not** the `path` attribute) of a previously
defined `project` element.
Attribute `enabled-list`: List of hooks to use, whitespace or comma separated.
### Element include ### Element include
This element provides the capability of including another manifest This element provides the capability of including another manifest
@ -386,10 +368,6 @@ target manifest to include - it must be a usable manifest on its own.
Attribute `name`: the manifest to include, specified relative to Attribute `name`: the manifest to include, specified relative to
the manifest repository's root. the manifest repository's root.
Attribute `groups`: List of additional groups to which all projects
in the included manifest belong. This appends and recurses, meaning
all projects in sub-manifests carry all parent include groups.
Same syntax as the corresponding element of `project`.
## Local Manifests ## Local Manifests

View File

@ -18,13 +18,13 @@ Bugfixes may be added on a best-effort basis or from the community, but largely
no new features will be added, nor is support guaranteed. no new features will be added, nor is support guaranteed.
Users can select this during `repo init` time via the [repo launcher]. Users can select this during `repo init` time via the [repo launcher].
Otherwise the default branches (e.g. stable & main) will be used which will Otherwise the default branches (e.g. stable & master) will be used which will
require Python 3. require Python 3.
This means the [repo launcher] needs to support both Python 2 & Python 3, but This means the [repo launcher] needs to support both Python 2 & Python 3, but
since it doesn't import any other repo code, this shouldn't be too problematic. since it doesn't import any other repo code, this shouldn't be too problematic.
The main branch will require Python 3.6 at a minimum. The master branch will require Python 3.6 at a minimum.
If the system has an older version of Python 3, then users will have to select If the system has an older version of Python 3, then users will have to select
the legacy Python 2 branch instead. the legacy Python 2 branch instead.

View File

@ -97,7 +97,7 @@ If that tag cannot be verified, it gives up and forces the user to resolve.
## Branch management ## Branch management
All development happens on the `main` branch and should generally be stable. All development happens on the `master` branch and should generally be stable.
Since the repo launcher defaults to tracking the `stable` branch, it is not Since the repo launcher defaults to tracking the `stable` branch, it is not
normally updated until a new release is available. normally updated until a new release is available.
@ -112,7 +112,7 @@ For example, when `stable` moves from `v1.10.x` to `v1.11.x`, then the `maint`
branch will be updated from `v1.9.x` to `v1.10.x`. branch will be updated from `v1.9.x` to `v1.10.x`.
We don't have parallel release branches/series. We don't have parallel release branches/series.
Typically all tags are made against the `main` branch and then pushed to the Typically all tags are made against the `master` branch and then pushed to the
`stable` branch to make it available to the rest of the world. `stable` branch to make it available to the rest of the world.
Since repo doesn't typically see a lot of changes, this tends to be OK. Since repo doesn't typically see a lot of changes, this tends to be OK.
@ -120,10 +120,10 @@ Since repo doesn't typically see a lot of changes, this tends to be OK.
When you want to create a new release, you'll need to select a good version and When you want to create a new release, you'll need to select a good version and
create a signed tag using a key registered in repo itself. create a signed tag using a key registered in repo itself.
Typically we just tag the latest version of the `main` branch. Typically we just tag the latest version of the `master` branch.
The tag could be pushed now, but it won't be used by clients normally (since the The tag could be pushed now, but it won't be used by clients normally (since the
default `repo-rev` setting is `stable`). default `repo-rev` setting is `stable`).
This would allow some early testing on systems who explicitly select `main`. This would allow some early testing on systems who explicitly select `master`.
### Creating a signed tag ### Creating a signed tag
@ -144,7 +144,7 @@ $ export GNUPGHOME=~/.gnupg/repo/
$ gpg -K $ gpg -K
# Pick whatever branch or commit you want to tag. # Pick whatever branch or commit you want to tag.
$ r=main $ r=master
# Pick the new version. # Pick the new version.
$ t=1.12.10 $ t=1.12.10

View File

@ -27,7 +27,7 @@ repohooks project is updated and a hook is triggered.
For the full syntax, see the [repo manifest format](./manifest-format.md). For the full syntax, see the [repo manifest format](./manifest-format.md).
Here's a short example from Here's a short example from
[Android](https://android.googlesource.com/platform/manifest/+/HEAD/default.xml). [Android](https://android.googlesource.com/platform/manifest/+/master/default.xml).
The `<project>` line checks out the repohooks git repo to the local The `<project>` line checks out the repohooks git repo to the local
`tools/repohooks/` path. The `<repo-hooks>` line says to look in the project `tools/repohooks/` path. The `<repo-hooks>` line says to look in the project
with the name `platform/tools/repohooks` for hooks to run during the with the name `platform/tools/repohooks` for hooks to run during the

View File

@ -1,197 +0,0 @@
# Copyright (C) 2020 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Provide event logging in the git trace2 EVENT format.
The git trace2 EVENT format is defined at:
https://www.kernel.org/pub/software/scm/git/docs/technical/api-trace2.html#_event_format
https://git-scm.com/docs/api-trace2#_the_event_format_target
Usage:
git_trace_log = EventLog()
git_trace_log.StartEvent()
...
git_trace_log.ExitEvent()
git_trace_log.Write()
"""
import datetime
import json
import os
import sys
import tempfile
import threading
from git_command import GitCommand, RepoSourceVersion
class EventLog(object):
"""Event log that records events that occurred during a repo invocation.
Events are written to the log as a consecutive JSON entries, one per line.
Entries follow the git trace2 EVENT format.
Each entry contains the following common keys:
- event: The event name
- sid: session-id - Unique string to allow process instance to be identified.
- thread: The thread name.
- time: is the UTC time of the event.
Valid 'event' names and event specific fields are documented here:
https://git-scm.com/docs/api-trace2#_event_format
"""
def __init__(self, env=None):
"""Initializes the event log."""
self._log = []
# Try to get session-id (sid) from environment (setup in repo launcher).
KEY = 'GIT_TRACE2_PARENT_SID'
if env is None:
env = os.environ
now = datetime.datetime.utcnow()
# Save both our sid component and the complete sid.
# We use our sid component (self._sid) as the unique filename prefix and
# the full sid (self._full_sid) in the log itself.
self._sid = 'repo-%s-P%08x' % (now.strftime('%Y%m%dT%H%M%SZ'), os.getpid())
parent_sid = env.get(KEY)
# Append our sid component to the parent sid (if it exists).
if parent_sid is not None:
self._full_sid = parent_sid + '/' + self._sid
else:
self._full_sid = self._sid
# Set/update the environment variable.
# Environment handling across systems is messy.
try:
env[KEY] = self._full_sid
except UnicodeEncodeError:
env[KEY] = self._full_sid.encode()
# Add a version event to front of the log.
self._AddVersionEvent()
@property
def full_sid(self):
return self._full_sid
def _AddVersionEvent(self):
"""Adds a 'version' event at the beginning of current log."""
version_event = self._CreateEventDict('version')
version_event['evt'] = 2
version_event['exe'] = RepoSourceVersion()
self._log.insert(0, version_event)
def _CreateEventDict(self, event_name):
"""Returns a dictionary with the common keys/values for git trace2 events.
Args:
event_name: The event name.
Returns:
Dictionary with the common event fields populated.
"""
return {
'event': event_name,
'sid': self._full_sid,
'thread': threading.currentThread().getName(),
'time': datetime.datetime.utcnow().isoformat() + 'Z',
}
def StartEvent(self):
"""Append a 'start' event to the current log."""
start_event = self._CreateEventDict('start')
start_event['argv'] = sys.argv
self._log.append(start_event)
def ExitEvent(self, result):
"""Append an 'exit' event to the current log.
Args:
result: Exit code of the event
"""
exit_event = self._CreateEventDict('exit')
# Consider 'None' success (consistent with event_log result handling).
if result is None:
result = 0
exit_event['code'] = result
self._log.append(exit_event)
def Write(self, path=None):
"""Writes the log out to a file.
Log is only written if 'path' or 'git config --get trace2.eventtarget'
provide a valid path to write logs to.
Logging filename format follows the git trace2 style of being a unique
(exclusive writable) file.
Args:
path: Path to where logs should be written.
Returns:
log_path: Path to the log file if log is written, otherwise None
"""
log_path = None
# If no logging path is specified, get the path from 'trace2.eventtarget'.
if path is None:
cmd = ['config', '--get', 'trace2.eventtarget']
# TODO(https://crbug.com/gerrit/13706): Use GitConfig when it supports
# system git config variables.
p = GitCommand(None, cmd, capture_stdout=True, capture_stderr=True,
bare=True)
retval = p.Wait()
if retval == 0:
# Strip trailing carriage-return in path.
path = p.stdout.rstrip('\n')
elif retval != 1:
# `git config --get` is documented to produce an exit status of `1` if
# the requested variable is not present in the configuration. Report any
# other return value as an error.
print("repo: error: 'git config --get' call failed with return code: %r, stderr: %r" % (
retval, p.stderr), file=sys.stderr)
if isinstance(path, str):
# Get absolute path.
path = os.path.abspath(os.path.expanduser(path))
else:
raise TypeError('path: str required but got %s.' % type(path))
# Git trace2 requires a directory to write log to.
# TODO(https://crbug.com/gerrit/13706): Support file (append) mode also.
if not os.path.isdir(path):
return None
# Use NamedTemporaryFile to generate a unique filename as required by git trace2.
try:
with tempfile.NamedTemporaryFile(mode='x', prefix=self._sid, dir=path,
delete=False) as f:
# TODO(https://crbug.com/gerrit/13706): Support writing events as they
# occur.
for e in self._log:
# Dump in compact encoding mode.
# See 'Compact encoding' in Python docs:
# https://docs.python.org/3/library/json.html#module-json
json.dump(e, f, indent=None, separators=(',', ':'))
f.write('\n')
log_path = f.name
except FileExistsError as err:
print('repo: warning: git trace2 logging failed: %r' % err,
file=sys.stderr)
return None
return log_path

View File

@ -45,8 +45,7 @@ def _set_project_revisions(projects):
should not be overly large. Recommend calling this function multiple times should not be overly large. Recommend calling this function multiple times
with each call not exceeding NUM_BATCH_RETRIEVE_REVISIONID projects. with each call not exceeding NUM_BATCH_RETRIEVE_REVISIONID projects.
Args: @param projects: List of project objects to set the revionExpr for.
projects: List of project objects to set the revionExpr for.
""" """
# Retrieve the commit id for each project based off of it's current # Retrieve the commit id for each project based off of it's current
# revisionExpr and it is not already a commit id. # revisionExpr and it is not already a commit id.
@ -74,8 +73,7 @@ def _manifest_groups(manifest):
This is the same logic used by Command.GetProjects(), which is used during This is the same logic used by Command.GetProjects(), which is used during
repo sync repo sync
Args: @param manifest: The XmlManifest object
manifest: The XmlManifest object
""" """
mp = manifest.manifestProject mp = manifest.manifestProject
groups = mp.config.GetString('manifest.groups') groups = mp.config.GetString('manifest.groups')
@ -87,10 +85,9 @@ def _manifest_groups(manifest):
def generate_gitc_manifest(gitc_manifest, manifest, paths=None): def generate_gitc_manifest(gitc_manifest, manifest, paths=None):
"""Generate a manifest for shafsd to use for this GITC client. """Generate a manifest for shafsd to use for this GITC client.
Args: @param gitc_manifest: Current gitc manifest, or None if there isn't one yet.
gitc_manifest: Current gitc manifest, or None if there isn't one yet. @param manifest: A GitcManifest object loaded with the current repo manifest.
manifest: A GitcManifest object loaded with the current repo manifest. @param paths: List of project paths we want to update.
paths: List of project paths we want to update.
""" """
print('Generating GITC Manifest by fetching revision SHAs for each ' print('Generating GITC Manifest by fetching revision SHAs for each '
@ -152,15 +149,12 @@ def generate_gitc_manifest(gitc_manifest, manifest, paths=None):
def save_manifest(manifest, client_dir=None): def save_manifest(manifest, client_dir=None):
"""Save the manifest file in the client_dir. """Save the manifest file in the client_dir.
Args: @param client_dir: Client directory to save the manifest in.
manifest: Manifest object to save. @param manifest: Manifest object to save.
client_dir: Client directory to save the manifest in.
""" """
if not client_dir: if not client_dir:
manifest_file = manifest.manifestFile client_dir = manifest.gitc_client_dir
else: with open(os.path.join(client_dir, '.manifest'), 'w') as f:
manifest_file = os.path.join(client_dir, '.manifest')
with open(manifest_file, 'w') as f:
manifest.Save(f, groups=_manifest_groups(manifest)) manifest.Save(f, groups=_manifest_groups(manifest))
# TODO(sbasi/jorg): Come up with a solution to remove the sleep below. # TODO(sbasi/jorg): Come up with a solution to remove the sleep below.
# Give the GITC filesystem time to register the manifest changes. # Give the GITC filesystem time to register the manifest changes.

139
hooks.py
View File

@ -14,11 +14,9 @@
# See the License for the specific language governing permissions and # See the License for the specific language governing permissions and
# limitations under the License. # limitations under the License.
import errno
import json import json
import os import os
import re import re
import subprocess
import sys import sys
import traceback import traceback
@ -35,7 +33,6 @@ else:
urllib.parse = urlparse urllib.parse = urlparse
input = raw_input # noqa: F821 input = raw_input # noqa: F821
class RepoHook(object): class RepoHook(object):
"""A RepoHook contains information about a script to run as a hook. """A RepoHook contains information about a script to run as a hook.
@ -48,29 +45,13 @@ class RepoHook(object):
Hooks are always python. When a hook is run, we will load the hook into the Hooks are always python. When a hook is run, we will load the hook into the
interpreter and execute its main() function. interpreter and execute its main() function.
Combinations of hook option flags:
- no-verify=False, verify=False (DEFAULT):
If stdout is a tty, can prompt about running hooks if needed.
If user denies running hooks, the action is cancelled. If stdout is
not a tty and we would need to prompt about hooks, action is
cancelled.
- no-verify=False, verify=True:
Always run hooks with no prompt.
- no-verify=True, verify=False:
Never run hooks, but run action anyway (AKA bypass hooks).
- no-verify=True, verify=True:
Invalid
""" """
def __init__(self, def __init__(self,
hook_type, hook_type,
hooks_project, hooks_project,
repo_topdir, topdir,
manifest_url, manifest_url,
bypass_hooks=False,
allow_all_hooks=False,
ignore_hooks=False,
abort_if_user_denies=False): abort_if_user_denies=False):
"""RepoHook constructor. """RepoHook constructor.
@ -78,27 +59,20 @@ class RepoHook(object):
hook_type: A string representing the type of hook. This is also used hook_type: A string representing the type of hook. This is also used
to figure out the name of the file containing the hook. For to figure out the name of the file containing the hook. For
example: 'pre-upload'. example: 'pre-upload'.
hooks_project: The project containing the repo hooks. hooks_project: The project containing the repo hooks. If you have a
If you have a manifest, this is manifest.repo_hooks_project. manifest, this is manifest.repo_hooks_project. OK if this is None,
OK if this is None, which will make the hook a no-op. which will make the hook a no-op.
repo_topdir: The top directory of the repo client checkout. topdir: Repo's top directory (the one containing the .repo directory).
This is the one containing the .repo directory. Scripts will Scripts will run with CWD as this directory. If you have a manifest,
run with CWD as this directory. this is manifest.topdir
If you have a manifest, this is manifest.topdir.
manifest_url: The URL to the manifest git repo. manifest_url: The URL to the manifest git repo.
bypass_hooks: If True, then 'Do not run the hook'. abort_if_user_denies: If True, we'll throw a HookError() if the user
allow_all_hooks: If True, then 'Run the hook without prompting'.
ignore_hooks: If True, then 'Do not abort action if hooks fail'.
abort_if_user_denies: If True, we'll abort running the hook if the user
doesn't allow us to run the hook. doesn't allow us to run the hook.
""" """
self._hook_type = hook_type self._hook_type = hook_type
self._hooks_project = hooks_project self._hooks_project = hooks_project
self._repo_topdir = repo_topdir
self._manifest_url = manifest_url self._manifest_url = manifest_url
self._bypass_hooks = bypass_hooks self._topdir = topdir
self._allow_all_hooks = allow_all_hooks
self._ignore_hooks = ignore_hooks
self._abort_if_user_denies = abort_if_user_denies self._abort_if_user_denies = abort_if_user_denies
# Store the full path to the script for convenience. # Store the full path to the script for convenience.
@ -134,7 +108,7 @@ class RepoHook(object):
# NOTE: Local (non-committed) changes will not be factored into this hash. # NOTE: Local (non-committed) changes will not be factored into this hash.
# I think this is OK, since we're really only worried about warning the user # I think this is OK, since we're really only worried about warning the user
# about upstream changes. # about upstream changes.
return self._hooks_project.work_git.rev_parse(HEAD) return self._hooks_project.work_git.rev_parse('HEAD')
def _GetMustVerb(self): def _GetMustVerb(self):
"""Return 'must' if the hook is required; 'should' if not.""" """Return 'must' if the hook is required; 'should' if not."""
@ -373,7 +347,7 @@ context['main'](**kwargs)
try: try:
# Always run hooks with CWD as topdir. # Always run hooks with CWD as topdir.
os.chdir(self._repo_topdir) os.chdir(self._topdir)
# Put the hook dir as the first item of sys.path so hooks can do # Put the hook dir as the first item of sys.path so hooks can do
# relative imports. We want to replace the repo dir as [0] so # relative imports. We want to replace the repo dir as [0] so
@ -423,12 +397,7 @@ context['main'](**kwargs)
sys.path = orig_syspath sys.path = orig_syspath
os.chdir(orig_path) os.chdir(orig_path)
def _CheckHook(self): def Run(self, user_allows_all_hooks, **kwargs):
# Bail with a nice error if we can't find the hook.
if not os.path.isfile(self._script_fullpath):
raise HookError('Couldn\'t find repo hook: %s' % self._script_fullpath)
def Run(self, **kwargs):
"""Run the hook. """Run the hook.
If the hook doesn't exist (because there is no hooks project or because If the hook doesn't exist (because there is no hooks project or because
@ -441,80 +410,22 @@ context['main'](**kwargs)
to the hook type. For instance, pre-upload hooks will contain to the hook type. For instance, pre-upload hooks will contain
a project_list. a project_list.
Returns: Raises:
True: On success or ignore hooks by user-request HookError: If there was a problem finding the hook or the user declined
False: The hook failed. The caller should respond with aborting the action. to run a required hook (from _CheckForHookApproval).
Some examples in which False is returned:
* Finding the hook failed while it was enabled, or
* the user declined to run a required hook (from _CheckForHookApproval)
In all these cases the user did not pass the proper arguments to
ignore the result through the option combinations as listed in
AddHookOptionGroup().
""" """
# Do not do anything in case bypass_hooks is set, or # No-op if there is no hooks project or if hook is disabled.
# no-op if there is no hooks project or if hook is disabled. if ((not self._hooks_project) or (self._hook_type not in
if (self._bypass_hooks or self._hooks_project.enabled_repo_hooks)):
not self._hooks_project or return
self._hook_type not in self._hooks_project.enabled_repo_hooks):
return True
passed = True # Bail with a nice error if we can't find the hook.
try: if not os.path.isfile(self._script_fullpath):
self._CheckHook() raise HookError('Couldn\'t find repo hook: "%s"' % self._script_fullpath)
# Make sure the user is OK with running the hook. # Make sure the user is OK with running the hook.
if self._allow_all_hooks or self._CheckForHookApproval(): if (not user_allows_all_hooks) and (not self._CheckForHookApproval()):
return
# Run the hook with the same version of python we're using. # Run the hook with the same version of python we're using.
self._ExecuteHook(**kwargs) self._ExecuteHook(**kwargs)
except SystemExit as e:
passed = False
print('ERROR: %s hooks exited with exit code: %s' % (self._hook_type, str(e)),
file=sys.stderr)
except HookError as e:
passed = False
print('ERROR: %s' % str(e), file=sys.stderr)
if not passed and self._ignore_hooks:
print('\nWARNING: %s hooks failed, but continuing anyways.' % self._hook_type,
file=sys.stderr)
passed = True
return passed
@classmethod
def FromSubcmd(cls, manifest, opt, *args, **kwargs):
"""Method to construct the repo hook class
Args:
manifest: The current active manifest for this command from which we
extract a couple of fields.
opt: Contains the commandline options for the action of this hook.
It should contain the options added by AddHookOptionGroup() in which
we are interested in RepoHook execution.
"""
for key in ('bypass_hooks', 'allow_all_hooks', 'ignore_hooks'):
kwargs.setdefault(key, getattr(opt, key))
kwargs.update({
'hooks_project': manifest.repo_hooks_project,
'repo_topdir': manifest.topdir,
'manifest_url': manifest.manifestProject.GetRemote('origin').url,
})
return cls(*args, **kwargs)
@staticmethod
def AddOptionGroup(parser, name):
"""Help options relating to the various hooks."""
# Note that verify and no-verify are NOT opposites of each other, which
# is why they store to different locations. We are using them to match
# 'git commit' syntax.
group = parser.add_option_group(name + ' hooks')
group.add_option('--no-verify',
dest='bypass_hooks', action='store_true',
help='Do not run the %s hook.' % name)
group.add_option('--verify',
dest='allow_all_hooks', action='store_true',
help='Run the %s hook without prompting.' % name)
group.add_option('--ignore-hooks',
action='store_true',
help='Do not abort if %s hooks fail.' % name)

29
main.py
View File

@ -1,4 +1,4 @@
#!/usr/bin/env python3 #!/usr/bin/env python
# -*- coding:utf-8 -*- # -*- coding:utf-8 -*-
# #
# Copyright (C) 2008 The Android Open Source Project # Copyright (C) 2008 The Android Open Source Project
@ -50,7 +50,6 @@ import event_log
from repo_trace import SetTrace from repo_trace import SetTrace
from git_command import user_agent from git_command import user_agent
from git_config import init_ssh, close_ssh, RepoConfig from git_config import init_ssh, close_ssh, RepoConfig
from git_trace2_event_log import EventLog
from command import InteractiveCommand from command import InteractiveCommand
from command import MirrorSafeCommand from command import MirrorSafeCommand
from command import GitcAvailableCommand, GitcClientCommand from command import GitcAvailableCommand, GitcClientCommand
@ -64,7 +63,7 @@ from error import NoManifestException
from error import NoSuchProjectError from error import NoSuchProjectError
from error import RepoChangedException from error import RepoChangedException
import gitc_utils import gitc_utils
from manifest_xml import GitcClient, RepoClient from manifest_xml import GitcManifest, XmlManifest
from pager import RunPager, TerminatePager from pager import RunPager, TerminatePager
from wrapper import WrapperPath, Wrapper from wrapper import WrapperPath, Wrapper
@ -83,13 +82,12 @@ if not is_python3():
# #
# python-3.6 is in Ubuntu Bionic. # python-3.6 is in Ubuntu Bionic.
MIN_PYTHON_VERSION_SOFT = (3, 6) MIN_PYTHON_VERSION_SOFT = (3, 6)
MIN_PYTHON_VERSION_HARD = (3, 5) MIN_PYTHON_VERSION_HARD = (3, 4)
if sys.version_info.major < 3: if sys.version_info.major < 3:
print('repo: error: Python 2 is no longer supported; ' print('repo: warning: Python 2 is no longer supported; '
'Please upgrade to Python {}.{}+.'.format(*MIN_PYTHON_VERSION_SOFT), 'Please upgrade to Python {}.{}+.'.format(*MIN_PYTHON_VERSION_SOFT),
file=sys.stderr) file=sys.stderr)
sys.exit(1)
else: else:
if sys.version_info < MIN_PYTHON_VERSION_HARD: if sys.version_info < MIN_PYTHON_VERSION_HARD:
print('repo: error: Python 3 version is too old; ' print('repo: error: Python 3 version is too old; '
@ -131,8 +129,6 @@ global_options.add_option('--version',
global_options.add_option('--event-log', global_options.add_option('--event-log',
dest='event_log', action='store', dest='event_log', action='store',
help='filename of event log to append timeline to') help='filename of event log to append timeline to')
global_options.add_option('--git-trace2-event-log', action='store',
help='directory to write git trace2 event log to')
class _Repo(object): class _Repo(object):
@ -214,17 +210,15 @@ class _Repo(object):
file=sys.stderr) file=sys.stderr)
return 1 return 1
git_trace2_event_log = EventLog()
cmd.repodir = self.repodir cmd.repodir = self.repodir
cmd.client = RepoClient(cmd.repodir) cmd.manifest = XmlManifest(cmd.repodir)
cmd.manifest = cmd.client.manifest
cmd.gitc_manifest = None cmd.gitc_manifest = None
gitc_client_name = gitc_utils.parse_clientdir(os.getcwd()) gitc_client_name = gitc_utils.parse_clientdir(os.getcwd())
if gitc_client_name: if gitc_client_name:
cmd.gitc_manifest = GitcClient(cmd.repodir, gitc_client_name) cmd.gitc_manifest = GitcManifest(cmd.repodir, gitc_client_name)
cmd.client.isGitcClient = True cmd.manifest.isGitcClient = True
Editor.globalConfig = cmd.client.globalConfig Editor.globalConfig = cmd.manifest.globalConfig
if not isinstance(cmd, MirrorSafeCommand) and cmd.manifest.IsMirror: if not isinstance(cmd, MirrorSafeCommand) and cmd.manifest.IsMirror:
print("fatal: '%s' requires a working directory" % name, print("fatal: '%s' requires a working directory" % name,
@ -252,7 +246,7 @@ class _Repo(object):
return 1 return 1
if gopts.pager is not False and not isinstance(cmd, InteractiveCommand): if gopts.pager is not False and not isinstance(cmd, InteractiveCommand):
config = cmd.client.globalConfig config = cmd.manifest.globalConfig
if gopts.pager: if gopts.pager:
use_pager = True use_pager = True
else: else:
@ -265,8 +259,6 @@ class _Repo(object):
start = time.time() start = time.time()
cmd_event = cmd.event_log.Add(name, event_log.TASK_COMMAND, start) cmd_event = cmd.event_log.Add(name, event_log.TASK_COMMAND, start)
cmd.event_log.SetParent(cmd_event) cmd.event_log.SetParent(cmd_event)
git_trace2_event_log.StartEvent()
try: try:
cmd.ValidateOptions(copts, cargs) cmd.ValidateOptions(copts, cargs)
result = cmd.Execute(copts, cargs) result = cmd.Execute(copts, cargs)
@ -309,13 +301,10 @@ class _Repo(object):
cmd.event_log.FinishEvent(cmd_event, finish, cmd.event_log.FinishEvent(cmd_event, finish,
result is None or result == 0) result is None or result == 0)
git_trace2_event_log.ExitEvent(result)
if gopts.event_log: if gopts.event_log:
cmd.event_log.Write(os.path.abspath( cmd.event_log.Write(os.path.abspath(
os.path.expanduser(gopts.event_log))) os.path.expanduser(gopts.event_log)))
git_trace2_event_log.Write(gopts.git_trace2_event_log)
return result return result

View File

@ -187,24 +187,12 @@ class _XmlRemote(object):
class XmlManifest(object): class XmlManifest(object):
"""manages the repo configuration file""" """manages the repo configuration file"""
def __init__(self, repodir, manifest_file, local_manifests=None): def __init__(self, repodir):
"""Initialize.
Args:
repodir: Path to the .repo/ dir for holding all internal checkout state.
It must be in the top directory of the repo client checkout.
manifest_file: Full path to the manifest file to parse. This will usually
be |repodir|/|MANIFEST_FILE_NAME|.
local_manifests: Full path to the directory of local override manifests.
This will usually be |repodir|/|LOCAL_MANIFESTS_DIR_NAME|.
"""
# TODO(vapier): Move this out of this class.
self.globalConfig = GitConfig.ForUser()
self.repodir = os.path.abspath(repodir) self.repodir = os.path.abspath(repodir)
self.topdir = os.path.dirname(self.repodir) self.topdir = os.path.dirname(self.repodir)
self.manifestFile = manifest_file self.manifestFile = os.path.join(self.repodir, MANIFEST_FILE_NAME)
self.local_manifests = local_manifests self.globalConfig = GitConfig.ForUser()
self.isGitcClient = False
self._load_local_manifests = True self._load_local_manifests = True
self.repoProject = MetaProject(self, 'repo', self.repoProject = MetaProject(self, 'repo',
@ -292,21 +280,18 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
if r.revision is not None: if r.revision is not None:
e.setAttribute('revision', r.revision) e.setAttribute('revision', r.revision)
def _ParseList(self, field): def _ParseGroups(self, groups):
"""Parse fields that contain flattened lists. return [x for x in re.split(r'[,\s]+', groups) if x]
These are whitespace & comma separated. Empty elements will be discarded. def Save(self, fd, peg_rev=False, peg_rev_upstream=True, peg_rev_dest_branch=True, groups=None):
"""Write the current manifest out to the given file descriptor.
""" """
return [x for x in re.split(r'[,\s]+', field) if x]
def ToXml(self, peg_rev=False, peg_rev_upstream=True, peg_rev_dest_branch=True, groups=None):
"""Return the current manifest XML."""
mp = self.manifestProject mp = self.manifestProject
if groups is None: if groups is None:
groups = mp.config.GetString('manifest.groups') groups = mp.config.GetString('manifest.groups')
if groups: if groups:
groups = self._ParseList(groups) groups = self._ParseGroups(groups)
doc = xml.dom.minidom.Document() doc = xml.dom.minidom.Document()
root = doc.createElement('manifest') root = doc.createElement('manifest')
@ -474,56 +459,6 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
' '.join(self._repo_hooks_project.enabled_repo_hooks)) ' '.join(self._repo_hooks_project.enabled_repo_hooks))
root.appendChild(e) root.appendChild(e)
return doc
def ToDict(self, **kwargs):
"""Return the current manifest as a dictionary."""
# Elements that may only appear once.
SINGLE_ELEMENTS = {
'notice',
'default',
'manifest-server',
'repo-hooks',
}
# Elements that may be repeated.
MULTI_ELEMENTS = {
'remote',
'remove-project',
'project',
'extend-project',
'include',
# These are children of 'project' nodes.
'annotation',
'project',
'copyfile',
'linkfile',
}
doc = self.ToXml(**kwargs)
ret = {}
def append_children(ret, node):
for child in node.childNodes:
if child.nodeType == xml.dom.Node.ELEMENT_NODE:
attrs = child.attributes
element = dict((attrs.item(i).localName, attrs.item(i).value)
for i in range(attrs.length))
if child.nodeName in SINGLE_ELEMENTS:
ret[child.nodeName] = element
elif child.nodeName in MULTI_ELEMENTS:
ret.setdefault(child.nodeName, []).append(element)
else:
raise ManifestParseError('Unhandled element "%s"' % (child.nodeName,))
append_children(element, child)
append_children(ret, doc.firstChild)
return ret
def Save(self, fd, **kwargs):
"""Write the current manifest out to the given file descriptor."""
doc = self.ToXml(**kwargs)
doc.writexml(fd, '', ' ', '\n', 'UTF-8') doc.writexml(fd, '', ' ', '\n', 'UTF-8')
def _output_manifest_project_extras(self, p, e): def _output_manifest_project_extras(self, p, e):
@ -618,11 +553,20 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
nodes.append(self._ParseManifestXml(self.manifestFile, nodes.append(self._ParseManifestXml(self.manifestFile,
self.manifestProject.worktree)) self.manifestProject.worktree))
if self._load_local_manifests and self.local_manifests: if self._load_local_manifests:
if os.path.exists(os.path.join(self.repodir, LOCAL_MANIFEST_NAME)):
print('error: %s is not supported; put local manifests in `%s`'
'instead' % (LOCAL_MANIFEST_NAME,
os.path.join(self.repodir, LOCAL_MANIFESTS_DIR_NAME)),
file=sys.stderr)
sys.exit(1)
local_dir = os.path.abspath(os.path.join(self.repodir,
LOCAL_MANIFESTS_DIR_NAME))
try: try:
for local_file in sorted(platform_utils.listdir(self.local_manifests)): for local_file in sorted(platform_utils.listdir(local_dir)):
if local_file.endswith('.xml'): if local_file.endswith('.xml'):
local = os.path.join(self.local_manifests, local_file) local = os.path.join(local_dir, local_file)
nodes.append(self._ParseManifestXml(local, self.repodir)) nodes.append(self._ParseManifestXml(local, self.repodir))
except OSError: except OSError:
pass pass
@ -641,7 +585,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
self._loaded = True self._loaded = True
def _ParseManifestXml(self, path, include_root, parent_groups=''): def _ParseManifestXml(self, path, include_root):
try: try:
root = xml.dom.minidom.parse(path) root = xml.dom.minidom.parse(path)
except (OSError, xml.parsers.expat.ExpatError) as e: except (OSError, xml.parsers.expat.ExpatError) as e:
@ -660,17 +604,12 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
for node in manifest.childNodes: for node in manifest.childNodes:
if node.nodeName == 'include': if node.nodeName == 'include':
name = self._reqatt(node, 'name') name = self._reqatt(node, 'name')
include_groups = ''
if parent_groups:
include_groups = parent_groups
if node.hasAttribute('groups'):
include_groups = node.getAttribute('groups') + ',' + include_groups
fp = os.path.join(include_root, name) fp = os.path.join(include_root, name)
if not os.path.isfile(fp): if not os.path.isfile(fp):
raise ManifestParseError("include %s doesn't exist or isn't a file" raise ManifestParseError("include %s doesn't exist or isn't a file"
% (name,)) % (name,))
try: try:
nodes.extend(self._ParseManifestXml(fp, include_root, include_groups)) nodes.extend(self._ParseManifestXml(fp, include_root))
# should isolate this to the exact exception, but that's # should isolate this to the exact exception, but that's
# tricky. actual parsing implementation may vary. # tricky. actual parsing implementation may vary.
except (KeyboardInterrupt, RuntimeError, SystemExit): except (KeyboardInterrupt, RuntimeError, SystemExit):
@ -679,11 +618,6 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
raise ManifestParseError( raise ManifestParseError(
"failed parsing included manifest %s: %s" % (name, e)) "failed parsing included manifest %s: %s" % (name, e))
else: else:
if parent_groups and node.nodeName == 'project':
nodeGroups = parent_groups
if node.hasAttribute('groups'):
nodeGroups = node.getAttribute('groups') + ',' + nodeGroups
node.setAttribute('groups', nodeGroups)
nodes.append(node) nodes.append(node)
return nodes return nodes
@ -758,7 +692,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
path = node.getAttribute('path') path = node.getAttribute('path')
groups = node.getAttribute('groups') groups = node.getAttribute('groups')
if groups: if groups:
groups = self._ParseList(groups) groups = self._ParseGroups(groups)
revision = node.getAttribute('revision') revision = node.getAttribute('revision')
remote = node.getAttribute('remote') remote = node.getAttribute('remote')
if remote: if remote:
@ -780,7 +714,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
if node.nodeName == 'repo-hooks': if node.nodeName == 'repo-hooks':
# Get the name of the project and the (space-separated) list of enabled. # Get the name of the project and the (space-separated) list of enabled.
repo_hooks_project = self._reqatt(node, 'in-project') repo_hooks_project = self._reqatt(node, 'in-project')
enabled_repo_hooks = self._ParseList(self._reqatt(node, 'enabled-list')) enabled_repo_hooks = self._reqatt(node, 'enabled-list').split()
# Only one project can be the hooks project # Only one project can be the hooks project
if self._repo_hooks_project is not None: if self._repo_hooks_project is not None:
@ -993,7 +927,7 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
groups = '' groups = ''
if node.hasAttribute('groups'): if node.hasAttribute('groups'):
groups = node.getAttribute('groups') groups = node.getAttribute('groups')
groups = self._ParseList(groups) groups = self._ParseGroups(groups)
if parent is None: if parent is None:
relpath, worktree, gitdir, objdir, use_git_worktrees = \ relpath, worktree, gitdir, objdir, use_git_worktrees = \
@ -1270,7 +1204,15 @@ https://gerrit.googlesource.com/git-repo/+/HEAD/docs/manifest-format.md
class GitcManifest(XmlManifest): class GitcManifest(XmlManifest):
"""Parser for GitC (git-in-the-cloud) manifests."""
def __init__(self, repodir, gitc_client_name):
"""Initialize the GitcManifest object."""
super(GitcManifest, self).__init__(repodir)
self.isGitcClient = True
self.gitc_client_name = gitc_client_name
self.gitc_client_dir = os.path.join(gitc_utils.get_gitc_manifest_dir(),
gitc_client_name)
self.manifestFile = os.path.join(self.gitc_client_dir, '.manifest')
def _ParseProject(self, node, parent=None): def _ParseProject(self, node, parent=None):
"""Override _ParseProject and add support for GITC specific attributes.""" """Override _ParseProject and add support for GITC specific attributes."""
@ -1281,38 +1223,3 @@ class GitcManifest(XmlManifest):
"""Output GITC Specific Project attributes""" """Output GITC Specific Project attributes"""
if p.old_revision: if p.old_revision:
e.setAttribute('old-revision', str(p.old_revision)) e.setAttribute('old-revision', str(p.old_revision))
class RepoClient(XmlManifest):
"""Manages a repo client checkout."""
def __init__(self, repodir, manifest_file=None):
self.isGitcClient = False
if os.path.exists(os.path.join(repodir, LOCAL_MANIFEST_NAME)):
print('error: %s is not supported; put local manifests in `%s` instead' %
(LOCAL_MANIFEST_NAME, os.path.join(repodir, LOCAL_MANIFESTS_DIR_NAME)),
file=sys.stderr)
sys.exit(1)
if manifest_file is None:
manifest_file = os.path.join(repodir, MANIFEST_FILE_NAME)
local_manifests = os.path.abspath(os.path.join(repodir, LOCAL_MANIFESTS_DIR_NAME))
super(RepoClient, self).__init__(repodir, manifest_file, local_manifests)
# TODO: Completely separate manifest logic out of the client.
self.manifest = self
class GitcClient(RepoClient, GitcManifest):
"""Manages a GitC client checkout."""
def __init__(self, repodir, gitc_client_name):
"""Initialize the GitcManifest object."""
self.gitc_client_name = gitc_client_name
self.gitc_client_dir = os.path.join(gitc_utils.get_gitc_manifest_dir(),
gitc_client_name)
super(GitcManifest, self).__init__(
repodir, os.path.join(self.gitc_client_dir, '.manifest'))
self.isGitcClient = True

View File

@ -62,8 +62,7 @@ RETRY_JITTER_PERCENT = 0.1
def _lwrite(path, content): def _lwrite(path, content):
lock = '%s.lock' % path lock = '%s.lock' % path
# Maintain Unix line endings on all OS's to match git behavior. with open(lock, 'w') as fd:
with open(lock, 'w', newline='\n') as fd:
fd.write(content) fd.write(content)
try: try:
@ -511,7 +510,7 @@ class Project(object):
with exponential backoff and jitter. with exponential backoff and jitter.
old_revision: saved git commit id for open GITC projects. old_revision: saved git commit id for open GITC projects.
""" """
self.client = self.manifest = manifest self.manifest = manifest
self.name = name self.name = name
self.remote = remote self.remote = remote
self.gitdir = gitdir.replace('\\', '/') self.gitdir = gitdir.replace('\\', '/')
@ -552,7 +551,7 @@ class Project(object):
self.linkfiles = [] self.linkfiles = []
self.annotations = [] self.annotations = []
self.config = GitConfig.ForRepository(gitdir=self.gitdir, self.config = GitConfig.ForRepository(gitdir=self.gitdir,
defaults=self.client.globalConfig) defaults=self.manifest.globalConfig)
if self.worktree: if self.worktree:
self.work_git = self._GitGetByExec(self, bare=False, gitdir=gitdir) self.work_git = self._GitGetByExec(self, bare=False, gitdir=gitdir)
@ -1027,7 +1026,6 @@ class Project(object):
if GitCommand(self, cmd, bare=True).Wait() != 0: if GitCommand(self, cmd, bare=True).Wait() != 0:
raise UploadError('Upload failed') raise UploadError('Upload failed')
if not dryrun:
msg = "posted to %s for %s" % (branch.remote.review, dest_branch) msg = "posted to %s for %s" % (branch.remote.review, dest_branch)
self.bare_git.UpdateRef(R_PUB + branch.name, self.bare_git.UpdateRef(R_PUB + branch.name,
R_HEADS + branch.name, R_HEADS + branch.name,
@ -1170,7 +1168,7 @@ class Project(object):
self._InitHooks() self._InitHooks()
def _CopyAndLinkFiles(self): def _CopyAndLinkFiles(self):
if self.client.isGitcClient: if self.manifest.isGitcClient:
return return
for copyfile in self.copyfiles: for copyfile in self.copyfiles:
copyfile._Copy() copyfile._Copy()
@ -2558,8 +2556,6 @@ class Project(object):
base = R_WORKTREE_M base = R_WORKTREE_M
active_git = self.work_git active_git = self.work_git
self._InitAnyMRef(HEAD, self.bare_git, detach=True)
else: else:
base = R_M base = R_M
active_git = self.bare_git active_git = self.bare_git
@ -2569,7 +2565,7 @@ class Project(object):
def _InitMirrorHead(self): def _InitMirrorHead(self):
self._InitAnyMRef(HEAD, self.bare_git) self._InitAnyMRef(HEAD, self.bare_git)
def _InitAnyMRef(self, ref, active_git, detach=False): def _InitAnyMRef(self, ref, active_git):
cur = self.bare_ref.symref(ref) cur = self.bare_ref.symref(ref)
if self.revisionId: if self.revisionId:
@ -2582,9 +2578,6 @@ class Project(object):
dst = remote.ToLocal(self.revisionExpr) dst = remote.ToLocal(self.revisionExpr)
if cur != dst: if cur != dst:
msg = 'manifest set to %s' % self.revisionExpr msg = 'manifest set to %s' % self.revisionExpr
if detach:
active_git.UpdateRef(ref, dst, message=msg, detach=True)
else:
active_git.symbolic_ref('-m', msg, ref, dst) active_git.symbolic_ref('-m', msg, ref, dst)
def _CheckDirReference(self, srcdir, destdir, share_refs): def _CheckDirReference(self, srcdir, destdir, share_refs):
@ -2708,14 +2701,12 @@ class Project(object):
# Some platforms (e.g. Windows) won't let us update dotgit in situ because # Some platforms (e.g. Windows) won't let us update dotgit in situ because
# of file permissions. Delete it and recreate it from scratch to avoid. # of file permissions. Delete it and recreate it from scratch to avoid.
platform_utils.remove(dotgit) platform_utils.remove(dotgit)
# Use relative path from checkout->worktree & maintain Unix line endings # Use relative path from checkout->worktree.
# on all OS's to match git behavior. with open(dotgit, 'w') as fp:
with open(dotgit, 'w', newline='\n') as fp:
print('gitdir:', os.path.relpath(git_worktree_path, self.worktree), print('gitdir:', os.path.relpath(git_worktree_path, self.worktree),
file=fp) file=fp)
# Use relative path from worktree->checkout & maintain Unix line endings # Use relative path from worktree->checkout.
# on all OS's to match git behavior. with open(os.path.join(git_worktree_path, 'gitdir'), 'w') as fp:
with open(os.path.join(git_worktree_path, 'gitdir'), 'w', newline='\n') as fp:
print(os.path.relpath(dotgit, git_worktree_path), file=fp) print(os.path.relpath(dotgit, git_worktree_path), file=fp)
self._InitMRef() self._InitMRef()

46
repo
View File

@ -32,13 +32,6 @@ import subprocess
import sys import sys
# These should never be newer than the main.py version since this needs to be a
# bit more flexible with older systems. See that file for more details on the
# versions we select.
MIN_PYTHON_VERSION_SOFT = (3, 6)
MIN_PYTHON_VERSION_HARD = (3, 5)
# Keep basic logic in sync with repo_trace.py. # Keep basic logic in sync with repo_trace.py.
class Trace(object): class Trace(object):
"""Trace helper logic.""" """Trace helper logic."""
@ -77,6 +70,8 @@ def check_python_version():
def reexec(prog): def reexec(prog):
exec_command([prog] + sys.argv) exec_command([prog] + sys.argv)
MIN_PYTHON_VERSION = (3, 6)
ver = sys.version_info ver = sys.version_info
major = ver.major major = ver.major
minor = ver.minor minor = ver.minor
@ -85,26 +80,19 @@ def check_python_version():
if (major, minor) < (2, 7): if (major, minor) < (2, 7):
print('repo: error: Your Python version is too old. ' print('repo: error: Your Python version is too old. '
'Please use Python {}.{} or newer instead.'.format( 'Please use Python {}.{} or newer instead.'.format(
*MIN_PYTHON_VERSION_SOFT), file=sys.stderr) *MIN_PYTHON_VERSION), file=sys.stderr)
sys.exit(1) sys.exit(1)
# Try to re-exec the version specific Python 3 if needed. # Try to re-exec the version specific Python 3 if needed.
if (major, minor) < MIN_PYTHON_VERSION_SOFT: if (major, minor) < MIN_PYTHON_VERSION:
# Python makes releases ~once a year, so try our min version +10 to help # Python makes releases ~once a year, so try our min version +10 to help
# bridge the gap. This is the fallback anyways so perf isn't critical. # bridge the gap. This is the fallback anyways so perf isn't critical.
min_major, min_minor = MIN_PYTHON_VERSION_SOFT min_major, min_minor = MIN_PYTHON_VERSION
for inc in range(0, 10): for inc in range(0, 10):
reexec('python{}.{}'.format(min_major, min_minor + inc)) reexec('python{}.{}'.format(min_major, min_minor + inc))
# Fallback to older versions if possible. # Try the generic Python 3 wrapper, but only if it's new enough. We don't
for inc in range(MIN_PYTHON_VERSION_SOFT[1] - MIN_PYTHON_VERSION_HARD[1], 0, -1): # want to go from (still supported) Python 2.7 to (unsupported) Python 3.5.
# Don't downgrade, and don't reexec ourselves (which would infinite loop).
if (min_major, min_minor - inc) <= (major, minor):
break
reexec('python{}.{}'.format(min_major, min_minor - inc))
# Try the generic Python 3 wrapper, but only if it's new enough. If it
# isn't, we want to just give up below and make the user resolve things.
try: try:
proc = subprocess.Popen( proc = subprocess.Popen(
['python3', '-c', 'import sys; ' ['python3', '-c', 'import sys; '
@ -115,20 +103,18 @@ def check_python_version():
except (OSError, subprocess.CalledProcessError): except (OSError, subprocess.CalledProcessError):
python3_ver = None python3_ver = None
# If the python3 version looks like it's new enough, give it a try. # The python3 version looks like it's new enough, so give it a try.
if (python3_ver and python3_ver >= MIN_PYTHON_VERSION_HARD if python3_ver and python3_ver >= MIN_PYTHON_VERSION:
and python3_ver != (major, minor)):
reexec('python3') reexec('python3')
# We're still here, so diagnose things for the user. # We're still here, so diagnose things for the user.
if major < 3: if major < 3:
print('repo: error: Python 2 is no longer supported; ' print('repo: warning: Python 2 is no longer supported; '
'Please upgrade to Python {}.{}+.'.format(*MIN_PYTHON_VERSION_HARD), 'Please upgrade to Python {}.{}+.'.format(*MIN_PYTHON_VERSION),
file=sys.stderr) file=sys.stderr)
sys.exit(1) else:
elif (major, minor) < MIN_PYTHON_VERSION_HARD:
print('repo: error: Python 3 version is too old; ' print('repo: error: Python 3 version is too old; '
'Please use Python {}.{} or newer.'.format(*MIN_PYTHON_VERSION_HARD), 'Please use Python {}.{} or newer.'.format(*MIN_PYTHON_VERSION),
file=sys.stderr) file=sys.stderr)
sys.exit(1) sys.exit(1)
@ -453,11 +439,9 @@ def get_gitc_manifest_dir():
def gitc_parse_clientdir(gitc_fs_path): def gitc_parse_clientdir(gitc_fs_path):
"""Parse a path in the GITC FS and return its client name. """Parse a path in the GITC FS and return its client name.
Args: @param gitc_fs_path: A subdirectory path within the GITC_FS_ROOT_DIR.
gitc_fs_path: A subdirectory path within the GITC_FS_ROOT_DIR.
Returns: @returns: The GITC client name
The GITC client name.
""" """
if gitc_fs_path == GITC_FS_ROOT_DIR: if gitc_fs_path == GITC_FS_ROOT_DIR:
return None return None

View File

@ -1,4 +1,4 @@
#!/usr/bin/env python3 #!/usr/bin/env python
# -*- coding:utf-8 -*- # -*- coding:utf-8 -*-
# Copyright 2019 The Android Open Source Project # Copyright 2019 The Android Open Source Project
# #
@ -20,26 +20,22 @@ from __future__ import print_function
import errno import errno
import os import os
import shutil
import subprocess import subprocess
import sys import sys
def find_pytest(): def run_pytest(cmd, argv):
"""Try to locate a good version of pytest.""" """Run the unittests via |cmd|."""
# Use the Python 3 version if available. try:
ret = shutil.which('pytest-3') return subprocess.call([cmd] + argv)
if ret: except OSError as e:
return ret if e.errno == errno.ENOENT:
print('%s: unable to run `%s`: %s' % (__file__, cmd, e), file=sys.stderr)
# Hopefully this is a Python 3 version. print('%s: Try installing pytest: sudo apt-get install python-pytest' %
ret = shutil.which('pytest') (__file__,), file=sys.stderr)
if ret: return 127
return ret else:
raise
print(f'{__file__}: unable to find pytest.', file=sys.stderr)
print(f'{__file__}: Try installing: sudo apt-get install python-pytest',
file=sys.stderr)
def main(argv): def main(argv):
@ -52,8 +48,7 @@ def main(argv):
pythonpath += os.pathsep + oldpythonpath pythonpath += os.pathsep + oldpythonpath
os.environ['PYTHONPATH'] = pythonpath os.environ['PYTHONPATH'] = pythonpath
pytest = find_pytest() return run_pytest('pytest', argv)
return subprocess.run([pytest] + argv, check=True)
if __name__ == '__main__': if __name__ == '__main__':

View File

@ -1,4 +1,4 @@
#!/usr/bin/env python3 #!/usr/bin/env python
# -*- coding:utf-8 -*- # -*- coding:utf-8 -*-
# Copyright 2019 The Android Open Source Project # Copyright 2019 The Android Open Source Project
# #
@ -55,10 +55,9 @@ setuptools.setup(
'Operating System :: MacOS :: MacOS X', 'Operating System :: MacOS :: MacOS X',
'Operating System :: Microsoft :: Windows :: Windows 10', 'Operating System :: Microsoft :: Windows :: Windows 10',
'Operating System :: POSIX :: Linux', 'Operating System :: POSIX :: Linux',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3 :: Only',
'Topic :: Software Development :: Version Control :: Git', 'Topic :: Software Development :: Version Control :: Git',
], ],
python_requires='>=3.6', # We support Python 2.7 and Python 3.6+.
python_requires='>=2.7, ' + ', '.join('!=3.%i.*' % x for x in range(0, 6)),
packages=['subcmds'], packages=['subcmds'],
) )

View File

@ -15,20 +15,10 @@
# limitations under the License. # limitations under the License.
from __future__ import print_function from __future__ import print_function
import itertools
import multiprocessing
import sys import sys
from color import Coloring from color import Coloring
from command import Command from command import Command
# Number of projects to submit to a single worker process at a time.
# This number represents a tradeoff between the overhead of IPC and finer
# grained opportunity for parallelism. This particular value was chosen by
# iterating through powers of two until the overall performance no longer
# improved. The performance of this batch size is not a function of the
# number of cores on the system.
WORKER_BATCH_SIZE = 32
class BranchColoring(Coloring): class BranchColoring(Coloring):
def __init__(self, config): def __init__(self, config):
@ -107,32 +97,20 @@ is shown, then the branch appears in all projects.
""" """
def _Options(self, p):
"""Add flags to CLI parser for this subcommand."""
default_jobs = min(multiprocessing.cpu_count(), 8)
p.add_option(
'-j',
'--jobs',
type=int,
default=default_jobs,
help='Number of worker processes to spawn '
'(default: %s)' % default_jobs)
def Execute(self, opt, args): def Execute(self, opt, args):
projects = self.GetProjects(args) projects = self.GetProjects(args)
out = BranchColoring(self.manifest.manifestProject.config) out = BranchColoring(self.manifest.manifestProject.config)
all_branches = {} all_branches = {}
project_cnt = len(projects) project_cnt = len(projects)
with multiprocessing.Pool(processes=opt.jobs) as pool:
project_branches = pool.imap_unordered(
expand_project_to_branches, projects, chunksize=WORKER_BATCH_SIZE)
for name, b in itertools.chain.from_iterable(project_branches): for project in projects:
for name, b in project.GetBranches().items():
b.project = project
if name not in all_branches: if name not in all_branches:
all_branches[name] = BranchInfo(name) all_branches[name] = BranchInfo(name)
all_branches[name].add(b) all_branches[name].add(b)
names = sorted(all_branches) names = list(sorted(all_branches))
if not names: if not names:
print(' (no branches)', file=sys.stderr) print(' (no branches)', file=sys.stderr)
@ -202,19 +180,3 @@ is shown, then the branch appears in all projects.
else: else:
out.write(' in all projects') out.write(' in all projects')
out.nl() out.nl()
def expand_project_to_branches(project):
"""Expands a project into a list of branch names & associated information.
Args:
project: project.Project
Returns:
List[Tuple[str, git_config.Branch]]
"""
branches = []
for name, b in project.GetBranches().items():
b.project = project
branches.append((name, b))
return branches

View File

@ -16,7 +16,7 @@
from color import Coloring from color import Coloring
from command import PagedCommand from command import PagedCommand
from manifest_xml import RepoClient from manifest_xml import XmlManifest
class _Coloring(Coloring): class _Coloring(Coloring):
@ -183,7 +183,7 @@ synced and their revisions won't be found.
self.OptionParser.error('missing manifests to diff') self.OptionParser.error('missing manifests to diff')
def Execute(self, opt, args): def Execute(self, opt, args):
self.out = _Coloring(self.client.globalConfig) self.out = _Coloring(self.manifest.globalConfig)
self.printText = self.out.nofmt_printer('text') self.printText = self.out.nofmt_printer('text')
if opt.color: if opt.color:
self.printProject = self.out.nofmt_printer('project', attr='bold') self.printProject = self.out.nofmt_printer('project', attr='bold')
@ -193,12 +193,12 @@ synced and their revisions won't be found.
else: else:
self.printProject = self.printAdded = self.printRemoved = self.printRevision = self.printText self.printProject = self.printAdded = self.printRemoved = self.printRevision = self.printText
manifest1 = RepoClient(self.manifest.repodir) manifest1 = XmlManifest(self.manifest.repodir)
manifest1.Override(args[0], load_local_manifests=False) manifest1.Override(args[0], load_local_manifests=False)
if len(args) == 1: if len(args) == 1:
manifest2 = self.manifest manifest2 = self.manifest
else: else:
manifest2 = RepoClient(self.manifest.repodir) manifest2 = XmlManifest(self.manifest.repodir)
manifest2.Override(args[1], load_local_manifests=False) manifest2.Override(args[1], load_local_manifests=False)
diff = manifest1.projectsDiff(manifest2) diff = manifest1.projectsDiff(manifest2)

View File

@ -65,7 +65,7 @@ Displays detailed usage information about a command.
def gitc_supported(cmd): def gitc_supported(cmd):
if not isinstance(cmd, GitcAvailableCommand) and not isinstance(cmd, GitcClientCommand): if not isinstance(cmd, GitcAvailableCommand) and not isinstance(cmd, GitcClientCommand):
return True return True
if self.client.isGitcClient: if self.manifest.isGitcClient:
return True return True
if isinstance(cmd, GitcClientCommand): if isinstance(cmd, GitcClientCommand):
return False return False
@ -127,7 +127,7 @@ Displays detailed usage information about a command.
self.wrap.end_paragraph(1) self.wrap.end_paragraph(1)
self.wrap.end_paragraph(0) self.wrap.end_paragraph(0)
out = _Out(self.client.globalConfig) out = _Out(self.manifest.globalConfig)
out._PrintSection('Summary', 'helpSummary') out._PrintSection('Summary', 'helpSummary')
cmd.OptionParser.print_help() cmd.OptionParser.print_help()
out._PrintSection('Description', 'helpDescription') out._PrintSection('Description', 'helpDescription')

View File

@ -44,7 +44,7 @@ class Info(PagedCommand):
help="Disable all remote operations") help="Disable all remote operations")
def Execute(self, opt, args): def Execute(self, opt, args):
self.out = _Coloring(self.client.globalConfig) self.out = _Coloring(self.manifest.globalConfig)
self.heading = self.out.printer('heading', attr='bold') self.heading = self.out.printer('heading', attr='bold')
self.headtext = self.out.nofmt_printer('headtext', fg='yellow') self.headtext = self.out.nofmt_printer('headtext', fg='yellow')
self.redtext = self.out.printer('redtext', fg='red') self.redtext = self.out.printer('redtext', fg='red')

View File

@ -365,7 +365,7 @@ to update the working directory files.
return a return a
def _ShouldConfigureUser(self, opt): def _ShouldConfigureUser(self, opt):
gc = self.client.globalConfig gc = self.manifest.globalConfig
mp = self.manifest.manifestProject mp = self.manifest.manifestProject
# If we don't have local settings, get from global. # If we don't have local settings, get from global.
@ -414,7 +414,7 @@ to update the working directory files.
return False return False
def _ConfigureColor(self): def _ConfigureColor(self):
gc = self.client.globalConfig gc = self.manifest.globalConfig
if self._HasColorSet(gc): if self._HasColorSet(gc):
return return
@ -521,7 +521,7 @@ to update the working directory files.
rp.gitdir, opt.repo_rev, repo_verify=opt.repo_verify, quiet=opt.quiet) rp.gitdir, opt.repo_rev, repo_verify=opt.repo_verify, quiet=opt.quiet)
branch = rp.GetBranch('default') branch = rp.GetBranch('default')
branch.merge = remote_ref branch.merge = remote_ref
rp.work_git.reset('--hard', rev) rp.work_git.update_ref('refs/heads/default', rev)
branch.Save() branch.Save()
if opt.worktree: if opt.worktree:

View File

@ -15,8 +15,6 @@
# limitations under the License. # limitations under the License.
from __future__ import print_function from __future__ import print_function
import json
import os import os
import sys import sys
@ -70,10 +68,6 @@ to indicate the remote ref to push changes to via 'repo upload'.
help='If in -r mode, do not write the dest-branch field. ' help='If in -r mode, do not write the dest-branch field. '
'Only of use if the branch names for a sha1 manifest are ' 'Only of use if the branch names for a sha1 manifest are '
'sensitive.') 'sensitive.')
p.add_option('--json', default=False, action='store_true',
help='Output manifest in JSON format (experimental).')
p.add_option('--pretty', default=False, action='store_true',
help='Format output for humans to read.')
p.add_option('-o', '--output-file', p.add_option('-o', '--output-file',
dest='output_file', dest='output_file',
default='-', default='-',
@ -89,22 +83,6 @@ to indicate the remote ref to push changes to via 'repo upload'.
fd = sys.stdout fd = sys.stdout
else: else:
fd = open(opt.output_file, 'w') fd = open(opt.output_file, 'w')
if opt.json:
print('warning: --json is experimental!', file=sys.stderr)
doc = self.manifest.ToDict(peg_rev=opt.peg_rev,
peg_rev_upstream=opt.peg_rev_upstream,
peg_rev_dest_branch=opt.peg_rev_dest_branch)
json_settings = {
# JSON style guide says Uunicode characters are fully allowed.
'ensure_ascii': False,
# We use 2 space indent to match JSON style guide.
'indent': 2 if opt.pretty else None,
'separators': (',', ': ') if opt.pretty else (',', ':'),
'sort_keys': True,
}
fd.write(json.dumps(doc, **json_settings))
else:
self.manifest.Save(fd, self.manifest.Save(fd,
peg_rev=opt.peg_rev, peg_rev=opt.peg_rev,
peg_rev_upstream=opt.peg_rev_upstream, peg_rev_upstream=opt.peg_rev_upstream,

View File

@ -165,7 +165,7 @@ the following meanings:
proj_dirs, proj_dirs_parents, outstring) proj_dirs, proj_dirs_parents, outstring)
if outstring: if outstring:
output = StatusColoring(self.client.globalConfig) output = StatusColoring(self.manifest.globalConfig)
output.project('Objects not within a project (orphans)') output.project('Objects not within a project (orphans)')
output.nl() output.nl()
for entry in outstring: for entry in outstring:

View File

@ -780,7 +780,6 @@ later is required to fix a server side protocol bug.
start = time.time() start = time.time()
success = mp.Sync_NetworkHalf(quiet=opt.quiet, verbose=opt.verbose, success = mp.Sync_NetworkHalf(quiet=opt.quiet, verbose=opt.verbose,
current_branch_only=opt.current_branch_only, current_branch_only=opt.current_branch_only,
force_sync=opt.force_sync,
tags=opt.tags, tags=opt.tags,
optimized_fetch=opt.optimized_fetch, optimized_fetch=opt.optimized_fetch,
retry_fetches=opt.retry_fetches, retry_fetches=opt.retry_fetches,

View File

@ -21,7 +21,7 @@ import sys
from command import InteractiveCommand from command import InteractiveCommand
from editor import Editor from editor import Editor
from error import UploadError from error import HookError, UploadError
from git_command import GitCommand from git_command import GitCommand
from git_refs import R_HEADS from git_refs import R_HEADS
from hooks import RepoHook from hooks import RepoHook
@ -205,7 +205,33 @@ Gerrit Code Review: https://www.gerritcodereview.com/
p.add_option('--no-cert-checks', p.add_option('--no-cert-checks',
dest='validate_certs', action='store_false', default=True, dest='validate_certs', action='store_false', default=True,
help='Disable verifying ssl certs (unsafe).') help='Disable verifying ssl certs (unsafe).')
RepoHook.AddOptionGroup(p, 'pre-upload')
# Options relating to upload hook. Note that verify and no-verify are NOT
# opposites of each other, which is why they store to different locations.
# We are using them to match 'git commit' syntax.
#
# Combinations:
# - no-verify=False, verify=False (DEFAULT):
# If stdout is a tty, can prompt about running upload hooks if needed.
# If user denies running hooks, the upload is cancelled. If stdout is
# not a tty and we would need to prompt about upload hooks, upload is
# cancelled.
# - no-verify=False, verify=True:
# Always run upload hooks with no prompt.
# - no-verify=True, verify=False:
# Never run upload hooks, but upload anyway (AKA bypass hooks).
# - no-verify=True, verify=True:
# Invalid
g = p.add_option_group('Upload hooks')
g.add_option('--no-verify',
dest='bypass_hooks', action='store_true',
help='Do not run the upload hook.')
g.add_option('--verify',
dest='allow_all_hooks', action='store_true',
help='Run the upload hook without prompting.')
g.add_option('--ignore-hooks',
dest='ignore_hooks', action='store_true',
help='Do not abort uploading if upload hooks fail.')
def _SingleBranch(self, opt, branch, people): def _SingleBranch(self, opt, branch, people):
project = branch.project project = branch.project
@ -528,10 +554,10 @@ Gerrit Code Review: https://www.gerritcodereview.com/
avail = [up_branch] avail = [up_branch]
else: else:
avail = None avail = None
print('repo: error: Unable to upload branch "%s". ' print('ERROR: Current branch (%s) not uploadable. '
'You might be able to fix the branch by running:\n' 'You may be able to type '
' git branch --set-upstream-to m/%s' % '"git branch --set-upstream-to m/master" to fix '
(str(cbr), self.manifest.branch), 'your branch.' % str(cbr),
file=sys.stderr) file=sys.stderr)
else: else:
avail = project.GetUploadableBranches(branch) avail = project.GetUploadableBranches(branch)
@ -546,14 +572,30 @@ Gerrit Code Review: https://www.gerritcodereview.com/
(branch,), file=sys.stderr) (branch,), file=sys.stderr)
return 1 return 1
if not opt.bypass_hooks:
hook = RepoHook('pre-upload', self.manifest.repo_hooks_project,
self.manifest.topdir,
self.manifest.manifestProject.GetRemote('origin').url,
abort_if_user_denies=True)
pending_proj_names = [project.name for (project, available) in pending] pending_proj_names = [project.name for (project, available) in pending]
pending_worktrees = [project.worktree for (project, available) in pending] pending_worktrees = [project.worktree for (project, available) in pending]
hook = RepoHook.FromSubcmd( passed = True
hook_type='pre-upload', manifest=self.manifest, try:
opt=opt, abort_if_user_denies=True) hook.Run(opt.allow_all_hooks, project_list=pending_proj_names,
if not hook.Run( worktree_list=pending_worktrees)
project_list=pending_proj_names, except SystemExit:
worktree_list=pending_worktrees): passed = False
if not opt.ignore_hooks:
raise
except HookError as e:
passed = False
print("ERROR: %s" % str(e), file=sys.stderr)
if not passed:
if opt.ignore_hooks:
print('\nWARNING: pre-upload hooks failed, but uploading anyways.',
file=sys.stderr)
else:
return 1 return 1
if opt.reviewers: if opt.reviewers:

View File

@ -1,169 +0,0 @@
# Copyright (C) 2020 The Android Open Source Project
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Unittests for the git_trace2_event_log.py module."""
import json
import tempfile
import unittest
import git_trace2_event_log
class EventLogTestCase(unittest.TestCase):
"""TestCase for the EventLog module."""
PARENT_SID_KEY = 'GIT_TRACE2_PARENT_SID'
PARENT_SID_VALUE = 'parent_sid'
SELF_SID_REGEX = r'repo-\d+T\d+Z-.*'
FULL_SID_REGEX = r'^%s/%s' % (PARENT_SID_VALUE, SELF_SID_REGEX)
def setUp(self):
"""Load the event_log module every time."""
self._event_log_module = None
# By default we initialize with the expected case where
# repo launches us (so GIT_TRACE2_PARENT_SID is set).
env = {
self.PARENT_SID_KEY: self.PARENT_SID_VALUE,
}
self._event_log_module = git_trace2_event_log.EventLog(env=env)
self._log_data = None
def verifyCommonKeys(self, log_entry, expected_event_name, full_sid=True):
"""Helper function to verify common event log keys."""
self.assertIn('event', log_entry)
self.assertIn('sid', log_entry)
self.assertIn('thread', log_entry)
self.assertIn('time', log_entry)
# Do basic data format validation.
self.assertEqual(expected_event_name, log_entry['event'])
if full_sid:
self.assertRegex(log_entry['sid'], self.FULL_SID_REGEX)
else:
self.assertRegex(log_entry['sid'], self.SELF_SID_REGEX)
self.assertRegex(log_entry['time'], r'^\d+-\d+-\d+T\d+:\d+:\d+\.\d+Z$')
def readLog(self, log_path):
"""Helper function to read log data into a list."""
log_data = []
with open(log_path, mode='rb') as f:
for line in f:
log_data.append(json.loads(line))
return log_data
def test_initial_state_with_parent_sid(self):
"""Test initial state when 'GIT_TRACE2_PARENT_SID' is set by parent."""
self.assertRegex(self._event_log_module.full_sid, self.FULL_SID_REGEX)
def test_initial_state_no_parent_sid(self):
"""Test initial state when 'GIT_TRACE2_PARENT_SID' is not set."""
# Setup an empty environment dict (no parent sid).
self._event_log_module = git_trace2_event_log.EventLog(env={})
self.assertRegex(self._event_log_module.full_sid, self.SELF_SID_REGEX)
def test_version_event(self):
"""Test 'version' event data is valid.
Verify that the 'version' event is written even when no other
events are addded.
Expected event log:
<version event>
"""
with tempfile.TemporaryDirectory(prefix='event_log_tests') as tempdir:
log_path = self._event_log_module.Write(path=tempdir)
self._log_data = self.readLog(log_path)
# A log with no added events should only have the version entry.
self.assertEqual(len(self._log_data), 1)
version_event = self._log_data[0]
self.verifyCommonKeys(version_event, expected_event_name='version')
# Check for 'version' event specific fields.
self.assertIn('evt', version_event)
self.assertIn('exe', version_event)
def test_start_event(self):
"""Test and validate 'start' event data is valid.
Expected event log:
<version event>
<start event>
"""
self._event_log_module.StartEvent()
with tempfile.TemporaryDirectory(prefix='event_log_tests') as tempdir:
log_path = self._event_log_module.Write(path=tempdir)
self._log_data = self.readLog(log_path)
self.assertEqual(len(self._log_data), 2)
start_event = self._log_data[1]
self.verifyCommonKeys(self._log_data[0], expected_event_name='version')
self.verifyCommonKeys(start_event, expected_event_name='start')
# Check for 'start' event specific fields.
self.assertIn('argv', start_event)
self.assertTrue(isinstance(start_event['argv'], list))
def test_exit_event_result_none(self):
"""Test 'exit' event data is valid when result is None.
We expect None result to be converted to 0 in the exit event data.
Expected event log:
<version event>
<exit event>
"""
self._event_log_module.ExitEvent(None)
with tempfile.TemporaryDirectory(prefix='event_log_tests') as tempdir:
log_path = self._event_log_module.Write(path=tempdir)
self._log_data = self.readLog(log_path)
self.assertEqual(len(self._log_data), 2)
exit_event = self._log_data[1]
self.verifyCommonKeys(self._log_data[0], expected_event_name='version')
self.verifyCommonKeys(exit_event, expected_event_name='exit')
# Check for 'exit' event specific fields.
self.assertIn('code', exit_event)
# 'None' result should convert to 0 (successful) return code.
self.assertEqual(exit_event['code'], 0)
def test_exit_event_result_integer(self):
"""Test 'exit' event data is valid when result is an integer.
Expected event log:
<version event>
<exit event>
"""
self._event_log_module.ExitEvent(2)
with tempfile.TemporaryDirectory(prefix='event_log_tests') as tempdir:
log_path = self._event_log_module.Write(path=tempdir)
self._log_data = self.readLog(log_path)
self.assertEqual(len(self._log_data), 2)
exit_event = self._log_data[1]
self.verifyCommonKeys(self._log_data[0], expected_event_name='version')
self.verifyCommonKeys(exit_event, expected_event_name='exit')
# Check for 'exit' event specific fields.
self.assertIn('code', exit_event)
self.assertEqual(exit_event['code'], 2)
# TODO(https://crbug.com/gerrit/13706): Add additional test coverage for
# Write() where:
# - path=None (using git config call)
# - path=<Non-String type> (raises TypeError)
# - path=<Non-Directory> (should return None)
# - tempfile.NamedTemporaryFile errors with FileExistsError (should return None)
if __name__ == '__main__':
unittest.main()

View File

@ -19,8 +19,6 @@
from __future__ import print_function from __future__ import print_function
import os import os
import shutil
import tempfile
import unittest import unittest
import xml.dom.minidom import xml.dom.minidom
@ -148,146 +146,3 @@ class ValueTests(unittest.TestCase):
with self.assertRaises(error.ManifestParseError): with self.assertRaises(error.ManifestParseError):
node = self._get_node('<node a="xx"/>') node = self._get_node('<node a="xx"/>')
manifest_xml.XmlInt(node, 'a') manifest_xml.XmlInt(node, 'a')
class XmlManifestTests(unittest.TestCase):
"""Check manifest processing."""
def setUp(self):
self.tempdir = tempfile.mkdtemp(prefix='repo_tests')
self.repodir = os.path.join(self.tempdir, '.repo')
self.manifest_dir = os.path.join(self.repodir, 'manifests')
self.manifest_file = os.path.join(
self.repodir, manifest_xml.MANIFEST_FILE_NAME)
self.local_manifest_dir = os.path.join(
self.repodir, manifest_xml.LOCAL_MANIFESTS_DIR_NAME)
os.mkdir(self.repodir)
os.mkdir(self.manifest_dir)
# The manifest parsing really wants a git repo currently.
gitdir = os.path.join(self.repodir, 'manifests.git')
os.mkdir(gitdir)
with open(os.path.join(gitdir, 'config'), 'w') as fp:
fp.write("""[remote "origin"]
url = https://localhost:0/manifest
""")
def tearDown(self):
shutil.rmtree(self.tempdir, ignore_errors=True)
def getXmlManifest(self, data):
"""Helper to initialize a manifest for testing."""
with open(self.manifest_file, 'w') as fp:
fp.write(data)
return manifest_xml.XmlManifest(self.repodir, self.manifest_file)
def test_empty(self):
"""Parse an 'empty' manifest file."""
manifest = self.getXmlManifest(
'<?xml version="1.0" encoding="UTF-8"?>'
'<manifest></manifest>')
self.assertEqual(manifest.remotes, {})
self.assertEqual(manifest.projects, [])
def test_link(self):
"""Verify Link handling with new names."""
manifest = manifest_xml.XmlManifest(self.repodir, self.manifest_file)
with open(os.path.join(self.manifest_dir, 'foo.xml'), 'w') as fp:
fp.write('<manifest></manifest>')
manifest.Link('foo.xml')
with open(self.manifest_file) as fp:
self.assertIn('<include name="foo.xml" />', fp.read())
def test_toxml_empty(self):
"""Verify the ToXml() helper."""
manifest = self.getXmlManifest(
'<?xml version="1.0" encoding="UTF-8"?>'
'<manifest></manifest>')
self.assertEqual(manifest.ToXml().toxml(), '<?xml version="1.0" ?><manifest/>')
def test_todict_empty(self):
"""Verify the ToDict() helper."""
manifest = self.getXmlManifest(
'<?xml version="1.0" encoding="UTF-8"?>'
'<manifest></manifest>')
self.assertEqual(manifest.ToDict(), {})
def test_repo_hooks(self):
"""Check repo-hooks settings."""
manifest = self.getXmlManifest("""
<manifest>
<remote name="test-remote" fetch="http://localhost" />
<default remote="test-remote" revision="refs/heads/main" />
<project name="repohooks" path="src/repohooks"/>
<repo-hooks in-project="repohooks" enabled-list="a, b"/>
</manifest>
""")
self.assertEqual(manifest.repo_hooks_project.name, 'repohooks')
self.assertEqual(manifest.repo_hooks_project.enabled_repo_hooks, ['a', 'b'])
def test_project_group(self):
"""Check project group settings."""
manifest = self.getXmlManifest("""
<manifest>
<remote name="test-remote" fetch="http://localhost" />
<default remote="test-remote" revision="refs/heads/main" />
<project name="test-name" path="test-path"/>
<project name="extras" path="path" groups="g1,g2,g1"/>
</manifest>
""")
self.assertEqual(len(manifest.projects), 2)
# Ordering isn't guaranteed.
result = {
manifest.projects[0].name: manifest.projects[0].groups,
manifest.projects[1].name: manifest.projects[1].groups,
}
project = manifest.projects[0]
self.assertCountEqual(
result['test-name'],
['name:test-name', 'all', 'path:test-path'])
self.assertCountEqual(
result['extras'],
['g1', 'g2', 'g1', 'name:extras', 'all', 'path:path'])
def test_include_levels(self):
root_m = os.path.join(self.manifest_dir, 'root.xml')
with open(root_m, 'w') as fp:
fp.write("""
<manifest>
<remote name="test-remote" fetch="http://localhost" />
<default remote="test-remote" revision="refs/heads/main" />
<include name="level1.xml" groups="level1-group" />
<project name="root-name1" path="root-path1" />
<project name="root-name2" path="root-path2" groups="r2g1,r2g2" />
</manifest>
""")
with open(os.path.join(self.manifest_dir, 'level1.xml'), 'w') as fp:
fp.write("""
<manifest>
<include name="level2.xml" groups="level2-group" />
<project name="level1-name1" path="level1-path1" />
</manifest>
""")
with open(os.path.join(self.manifest_dir, 'level2.xml'), 'w') as fp:
fp.write("""
<manifest>
<project name="level2-name1" path="level2-path1" groups="l2g1,l2g2" />
</manifest>
""")
include_m = manifest_xml.XmlManifest(self.repodir, root_m)
for proj in include_m.projects:
if proj.name == 'root-name1':
# Check include group not set on root level proj.
self.assertNotIn('level1-group', proj.groups)
if proj.name == 'root-name2':
# Check root proj group not removed.
self.assertIn('r2g1', proj.groups)
if proj.name == 'level1-name1':
# Check level1 proj has inherited group level 1.
self.assertIn('level1-group', proj.groups)
if proj.name == 'level2-name1':
# Check level2 proj has inherited group levels 1 and 2.
self.assertIn('level1-group', proj.groups)
self.assertIn('level2-group', proj.groups)
# Check level2 proj group not removed.
self.assertIn('l2g1', proj.groups)

View File

@ -26,7 +26,6 @@ import tempfile
import unittest import unittest
import error import error
import git_command
import git_config import git_config
import platform_utils import platform_utils
import project import project
@ -39,19 +38,7 @@ def TempGitTree():
# Python 2 support entirely. # Python 2 support entirely.
try: try:
tempdir = tempfile.mkdtemp(prefix='repo-tests') tempdir = tempfile.mkdtemp(prefix='repo-tests')
subprocess.check_call(['git', 'init'], cwd=tempdir)
# Tests need to assume, that main is default branch at init,
# which is not supported in config until 2.28.
cmd = ['git', 'init']
if git_command.git_require((2, 28, 0)):
cmd += ['--initial-branch=main']
else:
# Use template dir for init.
templatedir = tempfile.mkdtemp(prefix='.test-template')
with open(os.path.join(templatedir, 'HEAD'), 'w') as fp:
fp.write('ref: refs/heads/main\n')
cmd += ['--template=', templatedir]
subprocess.check_call(cmd, cwd=tempdir)
yield tempdir yield tempdir
finally: finally:
platform_utils.rmtree(tempdir) platform_utils.rmtree(tempdir)
@ -90,7 +77,7 @@ class ReviewableBranchTests(unittest.TestCase):
# Start off with the normal details. # Start off with the normal details.
rb = project.ReviewableBranch( rb = project.ReviewableBranch(
fakeproj, fakeproj.config.GetBranch('work'), 'main') fakeproj, fakeproj.config.GetBranch('work'), 'master')
self.assertEqual('work', rb.name) self.assertEqual('work', rb.name)
self.assertEqual(1, len(rb.commits)) self.assertEqual(1, len(rb.commits))
self.assertIn('Del file', rb.commits[0]) self.assertIn('Del file', rb.commits[0])
@ -103,9 +90,9 @@ class ReviewableBranchTests(unittest.TestCase):
self.assertTrue(rb.date) self.assertTrue(rb.date)
# Now delete the tracking branch! # Now delete the tracking branch!
fakeproj.work_git.branch('-D', 'main') fakeproj.work_git.branch('-D', 'master')
rb = project.ReviewableBranch( rb = project.ReviewableBranch(
fakeproj, fakeproj.config.GetBranch('work'), 'main') fakeproj, fakeproj.config.GetBranch('work'), 'master')
self.assertEqual(0, len(rb.commits)) self.assertEqual(0, len(rb.commits))
self.assertFalse(rb.base_exists) self.assertFalse(rb.base_exists)
# Hard to assert anything useful about this. # Hard to assert anything useful about this.

View File

@ -25,8 +25,6 @@ import shutil
import tempfile import tempfile
import unittest import unittest
import git_command
import main
import platform_utils import platform_utils
from pyversion import is_python3 from pyversion import is_python3
import wrapper import wrapper
@ -84,16 +82,6 @@ class RepoWrapperUnitTest(RepoWrapperTestCase):
self.assertEqual('', stderr.getvalue()) self.assertEqual('', stderr.getvalue())
self.assertIn('repo launcher version', stdout.getvalue()) self.assertIn('repo launcher version', stdout.getvalue())
def test_python_constraints(self):
"""The launcher should never require newer than main.py."""
self.assertGreaterEqual(main.MIN_PYTHON_VERSION_HARD,
wrapper.MIN_PYTHON_VERSION_HARD)
self.assertGreaterEqual(main.MIN_PYTHON_VERSION_SOFT,
wrapper.MIN_PYTHON_VERSION_SOFT)
# Make sure the versions are themselves in sync.
self.assertGreaterEqual(wrapper.MIN_PYTHON_VERSION_SOFT,
wrapper.MIN_PYTHON_VERSION_HARD)
def test_init_parser(self): def test_init_parser(self):
"""Make sure 'init' GetParser works.""" """Make sure 'init' GetParser works."""
parser = self.wrapper.GetParser(gitc_init=False) parser = self.wrapper.GetParser(gitc_init=False)
@ -369,19 +357,7 @@ class GitCheckoutTestCase(RepoWrapperTestCase):
remote = os.path.join(cls.GIT_DIR, 'remote') remote = os.path.join(cls.GIT_DIR, 'remote')
os.mkdir(remote) os.mkdir(remote)
run_git('init', cwd=remote)
# Tests need to assume, that main is default branch at init,
# which is not supported in config until 2.28.
if git_command.git_require((2, 28, 0)):
initstr = '--initial-branch=main'
else:
# Use template dir for init.
templatedir = tempfile.mkdtemp(prefix='.test-template')
with open(os.path.join(templatedir, 'HEAD'), 'w') as fp:
fp.write('ref: refs/heads/main\n')
initstr = '--template=' + templatedir
run_git('init', initstr, cwd=remote)
run_git('commit', '--allow-empty', '-minit', cwd=remote) run_git('commit', '--allow-empty', '-minit', cwd=remote)
run_git('branch', 'stable', cwd=remote) run_git('branch', 'stable', cwd=remote)
run_git('tag', 'v1.0', cwd=remote) run_git('tag', 'v1.0', cwd=remote)
@ -426,8 +402,8 @@ class ResolveRepoRev(GitCheckoutTestCase):
self.assertEqual('refs/heads/stable', rrev) self.assertEqual('refs/heads/stable', rrev)
self.assertEqual(self.REV_LIST[1], lrev) self.assertEqual(self.REV_LIST[1], lrev)
rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, 'main') rrev, lrev = self.wrapper.resolve_repo_rev(self.GIT_DIR, 'master')
self.assertEqual('refs/heads/main', rrev) self.assertEqual('refs/heads/master', rrev)
self.assertEqual(self.REV_LIST[0], lrev) self.assertEqual(self.REV_LIST[0], lrev)
def test_tag_name(self): def test_tag_name(self):