Since the --manifest-url flag is always required when creating a new
checkout, allow the url to be specified via a positional argument.
This brings it a little closer to the `git clone` UI.
Change-Id: Iaf18e794ae2fa38b20579243d067205cae5fae2f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297322
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
We only provide input to GitCommand in one place, so inline the logic
to be more synchronous and similar to subprocess.run. This makes the
code simpler and easier to understand.
Change-Id: Ibe498fedf608774bae1f807fc301eb67841c468b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/297142
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Added --no-use-superproject to repo and init.py to disable use of
manifest superprojects.
Replaced the term "sha" with "commit id".
Added _GetBranch method to Superproject object.
Moved shared code between init and sync into SyncSuperproject function.
This function either does git clone or git fetch. If git fetch fails
it does git clone.
Changed Superproject constructor to accept manifest, repodir and branch
to avoid passing them to multiple functions as argument.
Changed functions that were raising exceptions to return either True
or False.
Saved the --use-superproject option in config as repo.superproject.
Updated internal-fs-layout.md document.
Updated the tests to work with the new API changes in Superproject.
Performance for the first time sync has improved from 20 minutes to
around 15 minutes.
Tested the code with the following commands.
$ ./run_tests -v
Tested the sync code by using repo_dev alias and pointing to this CL.
$ repo init took around 20 seconds longer because of cloning of superproject.
$ time repo_dev init -u sso://android.git.corp.google.com/platform/manifest -b master --partial-clone --clone-filter=blob:limit=10M --repo-rev=main --use-superproject
...
real 0m35.919s
user 0m21.947s
sys 0m8.977s
First run
$ time repo sync --use-superproject
...
real 16m41.982s
user 100m6.916s
sys 19m18.753s
No difference in repo sync time after the first run.
Bug: [google internal] b/179090734
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Change-Id: I12df92112f46e001dfbc6f12cd633c3a15cf924b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296382
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Changed "git pull" to "git fetch" as we are using --bare option. Used the
following command to fetch:
git fetch origin +refs/heads/*:refs/heads/* --prune
Pass --branch argument to Superproject's UpdateProjectsRevisionId function.
Returned False/None when directories don't exist instead of raise
GitError exception from _Fetch and _LsTree functions. The caller of Fetch
does Clone if Fetch fails.
Tested the code with the following commands.
$ ./run_tests -v
Tested the init and sync code by copying all the repo changes into my Android
AOSP checkout and running repo sync with --use-superproject option.
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: I3e441ecdfc87c735f46eff0eb98efa63cc2eb22a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296222
Reviewed-by: Mike Frysinger <vapier@google.com>
After updating all project’s revsionIds with the SHAs from superproject,
write the updated manifest into superproject_override.xml file. Reload
that file for future Reloads. This file is created in exp-superproject
directory.
Moved most of the code that is superproject specific into
git_superproject.py and wrote test code.
If git pull fails, did a git clone of the superproject.
We saw performance gains for consecutive repo sync's. The time to sync
went down from around 120 secs to 40 secs when repo sync is executed
consecutively.
Tested the code with the following commands.
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and doing a repo sync --use-superproject twice.
First run
$ time repo sync --use-superproject
...
real 21m3.745s
user 97m59.380s
sys 19m11.286s
After two consecutive sync runs
$ time repo sync -c -j8 --use-superproject
real 0m39.626s
user 0m29.937s
sys 0m38.155s
Bug: https://crbug.com/gerrit/13709
Bug: https://crbug.com/gerrit/13707
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id79a0d7c4d20babd65e9bd485196c6f8fbe9de5e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/296082
Reviewed-by: Ian Kasprzak <iankaz@google.com>
Tested-by: Raman Tenneti <rtenneti@google.com>
Added "--use-superporject" option to sync.py to fetch project SHAs from
superproject. If there are any missing projects in superprojects, it
prints the missing entries and exits. If there are no missing entries,
it will use SHAs from superproject to fetch the projects from git.
Tested the code with the following commands.
$ ./run_tests tests/test_manifest_xml.py
$ ./run_tests -v tests/test_git_superproject.py
$ ./run_tests -v
Tested the sync code by copying all the repo changes into my Android
AOSP checkout and adding <superporject> tag to default.xml. With
local modification to the code to print the status,
.../WORKING_DIRECTORY$ repo sync --use-superproject
repo: executing 'git clone' url: sso://android/platform/superproject
repo: executing 'git ls-tree'
Success: []
Bug: https://crbug.com/gerrit/13709
Tested-by: Raman Tenneti <rtenneti@google.com>
Change-Id: Id18665992428dd684c04b0e0b3a52f46316873a0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/293822
Reviewed-by: Mike Frysinger <vapier@google.com>
Spread the operation of querying which local branches exist across a
pool of processes and build the name map of projects -> branches as
these tasks finish rather than blocking on the entire query. The search
operations are submitted in batches to reduce the overhead of interprocess
communication. The `chunksize` argument used to control this batch size
was selected by incrementing through powers of two until it stopped being
faster.
Change-Id: Ie3d7f799ee8e83e5058536caf53e2979175408b7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/291342
Tested-by: Chris Mcdonald <cjmcdonald@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
When updating the tracking ref to whatever the user requested,
make sure we reset state completely rather than trying to update
the ref to it. This avoids confusing git as to the current state
of the tree, and is more inline with user intentions: if they made
a local change to the checkout, but ran repo init with a specific
rev, we shouldn't stay wedged forever until they manually clean it
all up.
Bug: https://crbug.com/gerrit/12801
Change-Id: Ieba8d9c15781b4d0649bf01c7460694da63387b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/290923
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
The knowledge about running hooks and all its exception handling
is scattered over multiple files. This makes the code harder
to read, but also it requires duplication of logic in case
other RepoHooks are added to different commands.
This refactoring also creates uniform behavior of the hooks
across multiple commands and it guarantees the re-use of the same
arguments on all of them.
Signed-off-by: Remy Bohmer <github@bohmer.net>
Change-Id: Ia4d90eab429e4af00943306e89faec8db35ba29d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/277562
Tested-by: Remy Bohmer <oss@bohmer.net>
Reviewed-by: Mike Frysinger <vapier@google.com>
We conflate the manifest & parsing logic with the management of the
repo client checkout in a single class. This makes testing just one
part (the manifest parsing) hard as it requires a full checkout too.
Start splitting the two apart into separate classes to make it easy
to reason about & test.
Change-Id: Iaf897c93db9c724baba6044bfe7a589c024523b2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/288682
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Instead of hardcoding "master" as our default, use the remote server's
default branch instead. For most people, this should be the same as
"master" already. For projects moving to "main", it means we'll use
the new name automatically rather than forcing people to use -b main.
For repositories that never set up a default HEAD, we should still use
the historical "master" default.
Bug: https://crbug.com/gerrit/13339
Change-Id: I4117c81a760c9495f98dbb1111a3e6c127f45eba
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/280799
Reviewed-by: Michael Mortensen <mmortensen@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
This change increases the speed of the command with parallelization with
processes. The parallelization with threads doesn't work well, and
increasing the number of jobs to many (8 threads ~) didn't increase the speed.
Possibly, the global interpreter lock of Python affects.
Bug: https://crbug.com/gerrit/12389
Change-Id: Icbe5df8ba037dd91422b96f4e43708068d7be924
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279936
Tested-by: Kimiyuki Onaka <kimiyuki@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
If you pass args to `repo init` when first creating a checkout, the
repo launcher throws an error. But the init subcommand that runs in
an existing checkout silently ignores them. Throw a proper error.
Change-Id: I433bfcc73902d25f6b6a2974e77f6a977a75ed16
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/279696
Reviewed-by: Jonathan Nieder <jrn@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
For large projects, clone bundle is useful because it provided a way to
efficiently transfer a large portion of git objects through CDN, without
needing to interact with git server. However, with partial clones, the
intention is to not download most of the objects, so the use of clone
bundles would defeat the space savings normally seen with partial
clones, as they are downloaded before the first fetch.
A new option, --clone-bundle is added to override this behavior.
Add a new repo.clonebundle variable which remembers the choice if
explicitly given from command line at repo init.
Change-Id: I03638474af303a82af34579e16cd4700690b5f43
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268452
Tested-by: Xin Li <delphij@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
If the dest-branch attribute is set in the project manifest, then
we need to push to that branch. Previously, we would unconditionally
pre-pend the refs/heads prefix to it. The dest-branch attribute is
allowed to be a ref expression though, so it may already have it.
Simple fix is to check if it already has the prefix before adding it.
Bug: crbug.com/gerrit/12770
Change-Id: I45d6107ed6cf305cf223023b0ddad4278f7f4146
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/268152
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sean McAllister <smcallis@google.com>
When running repo info -d an error would be thrown saying:
fatal: bad revision 'refs/remotes/m/refs/heads/master..'
Using the short branch name here instead, like 'refs/remotes/m/master..'
resolves this issue.
Signed-off-by: Daniel Kutik <daniel.kutik@lavawerk.com>
Change-Id: I50ea92c45c011b2c3e3a63803decb88e7837a380
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/266578
Reviewed-by: Mike Frysinger <vapier@google.com>
When generating a revision locked manifest, we need to know what
ref to push changes to when doing 'repo upload'. This information
is lost when we lock the revision attribute to a particular commit
hash, so we need to expose it through the dest-branch attribute.
Bug: https://crbug.com/1005103
Test: manual execution
Change-Id: Ib31fd77ad8c9379759c4181dac1ea97de43eec35
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263572
Tested-by: Sean McAllister <smcallis@google.com>
Reviewed-by: Mike Frysinger <vapier@google.com>
Recent changes in ChromeOS Infra to ensure we're reading from
snapshot manifests properly have exposed several bugs in our
assumptions about manifest files. Mainly that the revision field
for a project does _not_ have to refer to a ref, it can just be
a commit hash.
Several places assume that the revision field can be parsed as a
ref to get the branch the project is on, which isn't true. To fix
this we need to be able to look at the upstream and dest-branch
attributes of the repo, so we expose them through the environment
variables set in `repo forall`.
Test: manual 'repo forall' run
Bug: https://crbug.com/1032441
Change-Id: I2c039e0f4b2e0f430602932e91b782edb6f9b1ed
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/263132
Reviewed-by: Mike Frysinger <vapier@google.com>
Tested-by: Sean McAllister <smcallis@google.com>
We respect this option when running the first `repo init`, but then
silently ignore it once the initial sync is done. Make sure users
are able to change things on the fly.
We refactor the wrapper API to allow reuse between the two init's.
Bug: https://crbug.com/gerrit/11045
Change-Id: Icb89a8cddca32f39a760a6283152457810b2392d
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/260032
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Jonathan Nieder <jrn@google.com>
We gracefully handle cherry-pick errors, but none of the others
which means people get confusing Python tracebacks. Move the
main logic in a single GitError try block so we can show pretty
error messages for all of them.
Change-Id: I52cdf6468d21a98de7f65b86d5267b3caabd5af8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259854
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The git cherry-pick already supports this, so plumb the existing repo
option down. Otherwise it's confusing when people use -c --ff and it
doesn't use that behavior.
Change-Id: Id68932ffa09204bb30b92a21aff185c00394a520
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259852
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
Tested-by: Mike Frysinger <vapier@google.com>
The current subcmds design has singletons in all_commands. This isn't
exactly unusual, but the fact that our main & help subcommand will then
attach members to the classes before invoking them is. This makes it
hard to keep track of what members a command has access to, and the two
code paths (main & help) attach different members depending on what APIs
they then invoke.
Lets pull this back a step by storing classes in all_commands and leave
the instantiation step to when they're used. This doesn't fully clean
up the confusion, but gets us closer.
Change-Id: I6a768ff97fe541e6f3228358dba04ed66c4b070a
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259154
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
The branch->branches alias is setup in the main module when that
really belongs in the existing all_commands setup.
For help, rather than monkey patching all_commands to the class,
switch it to use the state directly from the module. This makes
it a bit more obvious where it's coming from rather than this one
subcommand having a |commands| member added externally to it.
Change-Id: I0200def09bf4774cad8012af0f4ae60ea3089dc0
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/259153
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: David Pursehouse <dpursehouse@collab.net>
