ingydotnet / git-subrepo Goto Github PK

In cases you have existing projects with git subtree, its not plainly obvious how to migrate to git subrepo.

$ git subrepo clone https://github.com/kentfredric/travis-scripts.git maint-travis-ci/
# Error: subdir 'maint-travis-ci/' exists and is not empty

All output and errors should end in period

Either in the main documentation or as a separate document.

The status command can report how many new commits on either side, since the
last pull or clone. It can also give a log listing with --pretty=oneline of
those commits. The logs should probably be limited to 5 lines or so. If there
are more than we show we can print the command to show them all.

I did a - make install, and I got a cannot stat ‘doc/git-subrepo.1’: No such file or directory error message.

I am on ubuntu 14.04 Linux #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 GNU/Linux

Here's the output after a make install
~GIT_SUBREPO_HOME (master ✔) ᐅ sudo make install
[sudo] password for chungonn:
install -C -m 0755 /home/chungonn/development/git/git-subrepo/lib/git-subrepo /usr/lib/git-core/
install -C -d -m 0755 /usr/lib/git-core/git-subrepo.d/
install -C -m 0755 /home/chungonn/development/git/git-subrepo/lib/git-subrepo.d/bash+.bash /usr/lib/git-core/git-subrepo.d/
install -C -d -m 0755 /usr/local/share/man/man1
install -C -m 0644 doc/git-subrepo.1 /usr/local/share/man/man1
install: cannot stat ‘doc/git-subrepo.1’: No such file or directory
make: *** [install-doc] Error 1

$ git subrepo push ext/bashplus/ -b testing
git-subrepo: Command failed: 'git fetch --quiet [email protected]:ingydotnet/bashplus.git testing'.
$ git subrepo push ext/bashplus/
git-subrepo: There is a previous push branch 'subrepo-push/ext/bashplus'. Delete it first.

Maybe we just branch after the fetch.

In my experience, pull is too fragile and confusing.

I'd like to reimplement it made out of these:

git subrepo checkout <subdir>
git subrepo merge
git subrepo commit

And when the merge fails, tell the user to:

git subrepo checkout <subdir>
# hand merge/rebase/cherrypick/whatever and commit
git subrepo commit

The checkout command will:

• git subrepo fetch <subdir> # Fetch the subrepo's upstream
• Create refs/subrepo/remote/<subdir> → FETCH_HEAD
• filter-branch the subdir commits
• Create a branch subrepo/<subdir> → subdir history
• git checkout subrepo/<subdir>

The merge command can try simple merges of the subrepo branch and the
upstream head, but it should reset on failure, and tell the user to hand
merge.

The commit command is new. it takes the tree object of the successfully
merged subrepo branch, creates a new commit with no parents, appends on the
.gitrepo file, and attaches itself to the top of the mainline branch.

So when you are all done you just have a single new non-merge commit that
represents the pulled subrepo content.

I think we can ditch the --continue option. I really like the idea of having
more simple git-like subcommands that just do specific things.

Since a subrepo push should really just be a subrepo pull and a normal
push to the subrepo's remote, I think this should be everything.

@AndrewSmart, @kentfredric: Please review and comment. Also discuss on
irc.freenode.net#git-commands.

Other related changes I'd like:

• git subrepo status within a subrepo branch
• git subrepo log <subdir> show real commit log for subrepo
• git subrepo reset reset and checkout mainline branch

I'll start doing the new work on the push-commands branch. Feel free to
test, contribute and complain.

As someone who is tired of the submodules/subtree shenanigans, and eager to see an easier way to install this and get this to a wider audience, could there be a formal beta release?

I'm asking because that seems to be a requirement for getting it into mac homebrew: Homebrew/legacy-homebrew#27484

This is mostly because I simply expected that subsequent "Pulls" would create differential squashes like git subtree does, ie:

*     - Merge ( applies squash delta )
| \ 
|  *  - Pull ( squash delta vs clone )
|  | 
*  |   - Merge 
| \
|  *   - Clone
|

Maybe this is by-design, hard to tell.

Either way:

* - Merge ( applies squash delta )
| \
|  *  - Pull ( basically a clone with a different name )
|   
*    - Merge 
| \
|  *   - Clone
|

Is what I see.

The body of subrepo commit messages is a JSON object somewhat pretty-printed
with all the info. It looks ugly and we don't use it. Moreover, it gives the
false impression that we do use it for something.

I'd like the info to go into the body but let's use a nice simple format.

The git-subrepo internal commands like RUN can be used in the tests to make
them simpler to write, read and maintain.

Instead of the normal:

git rebase refs/subrepo/<subdir>/upstream subrepo/<subdir>

which can also be written:

git rebase subrepo/<subdir>{/upstream,}

people could write:

git subrepo rebase subrepo/<subdir>

I personally don't think it adds much and then we have to support it, so for
now I'm against the idea.

However, it might give us more control to fix simple conflicts, so we should
investigate it from that angle.

Many git command docs have example usage at the bottom.

I receive output of:

$ git subrepo push share
The 'git rebase' command failed during your push.
You will need to finish the push by hand. Follow these steps:

  # Resolve the rebase conflicts
  git checkout master
  git subrepo push share subrepo-push/share

To abort the push and reset back to where you started:

  git rebase --abort
  git checkout ORIG_HEAD
  git subrepo clean share

See 'git help subrepo' for more help.

Message above is misleading as push was aborted earlier in the code when it was detected there were no changes to push (in CALL subrepo:branch "$branch").
Pull request incoming

The pull and push commands can fail, especially during a rebase. Currently we
leave the user in the subrepo/<subdir> branch with the conflicts.

A reset command would just be an easy out, to undo the failed pull/push and get
back to a normal state. ie Not needed, but possibly comforting.

We need to know which branch to return to. We could either track it in a ref or
use ORIG_HEAD. We should do a sanity check on ORIG_HEAD, just to make sure it
doesn't point to a subrepo branch.

We should:

• git rebase --abort
• git checkout $orig_head
• git subrepo clean $subdir
• Print some useful message

git subrepo status however works.

And it seems intuitive that if that works, git subrepo pull should also.

A fetch command leaves behind a refs/subrepo/<subdir>/upstream ref.

Other commands could leave a ref:

• refs/subrepo/<subdir>/pull - The branch created and deleted to do a pull.
• refs/subrepo/<subdir>/push - The branch created and deleted to do a push.
• refs/subrepo/<subdir>/branch - The branch created.
• refs/subrepo/<subdir>/commit - The branch committed.
• refs/subrepo/<subdir>/fetch - A symbolic-ref to the branch pulled.

These refs could be useful to review what just happened, and also possibly
used internally for certain multi-command operations.

All refs are not deleted by the clean command, unless the --force option
is used, in which case they are deleted.

Hey @ingydotnet, I'm still around, just starting to use this day-to-day. :)

I'm using this to manage custom packages in a meteor app. When the app is running, if I attempt a git subrepo push, after the subrepo/prefix branch is created, meteor writes some temp files, which causes the git checkout master to fail. So then it becomes stranded on the subrepo/prefix branch.

Not data corruption or anything, but also could fail more gracefully.

Help can be displayed on command usage errors too.

In conjunction with bug #11, it seems somewhere in the logic is trying to create paths in refs/ with . in the path, which is invalid, and will never work.

git subrepo pull --all -v
>>> git rev-parse --short HEAD
>>> git rev-parse --abbrev-ref HEAD
>>> git rev-parse HEAD
>>> git rev-parse --is-inside-work-tree
>>> git rev-parse --verify HEAD
>>> git config --file=./.build/4aigJj5LnU/maint-travis-ci/.gitrepo subrepo.remote
>>> git config --file=./.build/4aigJj5LnU/maint-travis-ci/.gitrepo subrepo.branch
>>> git config --file=./.build/4aigJj5LnU/maint-travis-ci/.gitrepo subrepo.commit
>>> git config --file=./.build/4aigJj5LnU/maint-travis-ci/.gitrepo subrepo.parent
>>> git fetch --quiet https://github.com/kentfredric/travis-scripts.git master
>>> git rev-parse FETCH_HEAD
>>> git remote add subrepo/./.build/4aigJj5LnU/maint-travis-ci https://github.com/kentfredric/travis-scripts.git
fatal: 'subrepo/./.build/4aigJj5LnU/maint-travis-ci' is not a valid remote name
>>> git for-each-ref refs/subrepo/remote/./.build/4aigJj5LnU/maint-travis-ci
>>> git update-ref refs/subrepo/remote/./.build/4aigJj5LnU/maint-travis-ci 7c80dffae0d7e39b20582e368d85c290429219fa
fatal: Cannot lock the ref 'refs/subrepo/remote/./.build/4aigJj5LnU/maint-travis-ci'.
subrepo './.build/4aigJj5LnU/maint-travis-ci' is up to date
>>> git config --file=./.build/ZHF9Z1Gfsu/maint-travis-ci/.gitrepo subrepo.remote
>>> git config --file=./.build/ZHF9Z1Gfsu/maint-travis-ci/.gitrepo subrepo.branch
>>> git config --file=./.build/ZHF9Z1Gfsu/maint-travis-ci/.gitrepo subrepo.commit
>>> git config --file=./.build/ZHF9Z1Gfsu/maint-travis-ci/.gitrepo subrepo.parent
>>> git fetch --quiet https://github.com/kentfredric/travis-scripts.git master
>>> git rev-parse FETCH_HEAD
>>> git remote add subrepo/./.build/ZHF9Z1Gfsu/maint-travis-ci https://github.com/kentfredric/travis-scripts.git
fatal: 'subrepo/./.build/ZHF9Z1Gfsu/maint-travis-ci' is not a valid remote name
>>> git for-each-ref refs/subrepo/remote/./.build/ZHF9Z1Gfsu/maint-travis-ci
>>> git update-ref refs/subrepo/remote/./.build/ZHF9Z1Gfsu/maint-travis-ci 7c80dffae0d7e39b20582e368d85c290429219fa
fatal: Cannot lock the ref 'refs/subrepo/remote/./.build/ZHF9Z1Gfsu/maint-travis-ci'.
subrepo './.build/ZHF9Z1Gfsu/maint-travis-ci' is up to date
>>> git config --file=./Path-FindDev-0.4.3/maint-travis-ci/.gitrepo subrepo.remote
>>> git config --file=./Path-FindDev-0.4.3/maint-travis-ci/.gitrepo subrepo.branch
>>> git config --file=./Path-FindDev-0.4.3/maint-travis-ci/.gitrepo subrepo.commit
>>> git config --file=./Path-FindDev-0.4.3/maint-travis-ci/.gitrepo subrepo.parent
>>> git fetch --quiet https://github.com/kentfredric/travis-scripts.git master
>>> git rev-parse FETCH_HEAD
>>> git remote add subrepo/./Path-FindDev-0.4.3/maint-travis-ci https://github.com/kentfredric/travis-scripts.git
fatal: 'subrepo/./Path-FindDev-0.4.3/maint-travis-ci' is not a valid remote name
>>> git for-each-ref refs/subrepo/remote/./Path-FindDev-0.4.3/maint-travis-ci
>>> git update-ref refs/subrepo/remote/./Path-FindDev-0.4.3/maint-travis-ci 7c80dffae0d7e39b20582e368d85c290429219fa
fatal: Cannot lock the ref 'refs/subrepo/remote/./Path-FindDev-0.4.3/maint-travis-ci'.
subrepo './Path-FindDev-0.4.3/maint-travis-ci' is up to date
>>> git config --file=./Path-FindDev-0.5.0/maint-travis-ci/.gitrepo subrepo.remote
>>> git config --file=./Path-FindDev-0.5.0/maint-travis-ci/.gitrepo subrepo.branch
>>> git config --file=./Path-FindDev-0.5.0/maint-travis-ci/.gitrepo subrepo.commit
>>> git config --file=./Path-FindDev-0.5.0/maint-travis-ci/.gitrepo subrepo.parent
>>> git fetch --quiet https://github.com/kentfredric/travis-scripts.git master
>>> git rev-parse FETCH_HEAD
>>> git remote add subrepo/./Path-FindDev-0.5.0/maint-travis-ci https://github.com/kentfredric/travis-scripts.git
fatal: 'subrepo/./Path-FindDev-0.5.0/maint-travis-ci' is not a valid remote name
>>> git for-each-ref refs/subrepo/remote/./Path-FindDev-0.5.0/maint-travis-ci
>>> git update-ref refs/subrepo/remote/./Path-FindDev-0.5.0/maint-travis-ci 7c80dffae0d7e39b20582e368d85c290429219fa
fatal: Cannot lock the ref 'refs/subrepo/remote/./Path-FindDev-0.5.0/maint-travis-ci'.
subrepo './Path-FindDev-0.5.0/maint-travis-ci' is up to date
>>> git config --file=./maint-travis-ci/.gitrepo subrepo.remote
>>> git config --file=./maint-travis-ci/.gitrepo subrepo.branch
>>> git config --file=./maint-travis-ci/.gitrepo subrepo.commit
>>> git config --file=./maint-travis-ci/.gitrepo subrepo.parent
>>> git fetch --quiet https://github.com/kentfredric/travis-scripts.git master
>>> git rev-parse FETCH_HEAD
>>> git remote add subrepo/./maint-travis-ci https://github.com/kentfredric/travis-scripts.git
fatal: 'subrepo/./maint-travis-ci' is not a valid remote name
>>> git for-each-ref refs/subrepo/remote/./maint-travis-ci
>>> git update-ref refs/subrepo/remote/./maint-travis-ci 7c80dffae0d7e39b20582e368d85c290429219fa
fatal: Cannot lock the ref 'refs/subrepo/remote/./maint-travis-ci'.
subrepo './maint-travis-ci' is up to date

Currently all subrepo commands are disabled when you have a subrepo branch
checked out. This is mostly because we don't have access to the .gitrepo
files.

The help and version commands are just informational and should always work.

We also might want to make a special version of the status command for inside a
subrepo branch, that can tell the state of the rebase and such. That should be
written up as another issue.

I'm not sure what happens in the various commands yet. Off-hand it seems like
it should be fine as long as the function that detects all subrepos doesn't
detect subsubrepos.

A test would be good here.

When you try to rebase, especially with rebase --preserve, a whole lot of bad things happen due to the trees not being very similar.

For instance, if both sides have a README.mkdn, well, this happens:

git rebase --preserve repick
Auto-merging README.mkdn
CONFLICT (add/add): Merge conflict in README.mkdn
Automatic merge failed; fix conflicts and then commit the result.
Error redoing merge 13ad9ead2785efe6f2b6b00e39349749f8382bce

Even though their final paths are different, merge preservation means the magic merge commits are discarded and re-attempted, and they're re-attempted without the path transformation! Ugh! :/

Git subtree has the same defect and its infuriating to work around.

This was started in the Intro doc but left as a "to be finished". I yanked it
out.

We can make a separate comparison doc that shows a comparison of the actual
commands used in each of the 3 solutions. We should also talk about the good,
bad and ugly of each situation.

RUN branch -D "$checkout_branch"
should be
RUN git branch -D "$checkout_branch"
https://github.com/ingydotnet/git-subrepo/blob/master/lib/git-subrepo#L248

Just saw this and puzzled be a bit till I worked out what was happening:

$ git subrepo status
Subrepo: ./File-ShareDir-ProjectDistDir-0.5.3/maint-travis-ci/

Subrepo: ./maint-travis-ci/

kent@katipo2 ~/perl/File-ShareDir-ProjectDistDir $ 

One is a copy of the other, probably not a problem, just surprising is all =).

You could perhaps record the repo-relative path of the clone as some sort of QA check "Hey, this is not something we made".

And it makes me curious what happens when you sub-repo a repo with a subrepo in it =).

I figure you could possibly add something like:

make userenv

And have it print working values to stuff into bashrc to make it easier.

There abouts you could add glue to get manpages working, like:

mkdir $SUBREPODIR/man
ln -s $SUBREPODIR/doc $SUBREPODIR/man/man1

And then tell user to

export MANPATH="$SUBREPODIR/man;$MANPATH"

If it errors then:

git subrepo branch --all

Will fail. Just print the reason and return 0.

Add a test/all.t for this.

I made a change to a subrepo, and wanted to push it back to the source repo,
but onto a branch so I could test it and then merge to master.

git subrepo push <subdir> --branch=test-subrepo-branch

Fails because that wants to fetch/rebase with that upstream branch. If it did
exist, it would DTRT.

Maybe if the fetch fails (for the reason of non-existing upstream branch) we
can use the tracking branch instead.

Anyone got a better flow?

There's a lengthy example of replicating this here: https://gist.github.com/kentfredric/6e06db0933e27e414e93#file-gistfile1-txt

But the gist of it is this:

| - A
| - B 
| - C
| - Subrepo clone (D)
| - E

When rebased becomes

| - A
| - B+C (B')
| - Subrepo clone (D')
| - E'

On an unrebased branch you'd expect to see:

| - A
| - B 
| - C
| - Subrepo clone (D)
| - E
| - Subrepo pull  (F)

However, the rebased one is different, and for some reason the "Pull" sees the changes between D and F as collisions

I'm not sure what's going on, I don't seem to be able to rig any values to trick it into working :/

$ mkdir x
$ cd x
$ git init
$ git subrepo clone <url> -v
>>> git rev-parse --short HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Error: command failed: git rev-parse --abbrev-ref HEAD

If I touch a file and commit it, the clone works just fine. There shouldn't be a requirement of a first commit to use this plugin

So I have two projects, p1 and p2, which use a library lib. Here is a sample showing the problem:

mkdir lib p1 p2 && git init --bare lib && git init p1 && git init p2
cd p1 && touch p1 && git add p1 && git commit -m "Initial" && git subrepo clone ../lib lib -b master && cd ..
cd p2 && touch p2 && git add p2 && git commit -m "Initial" && git subrepo clone ../lib lib -b master && cd ../p1
echo "p1 initial add to subrepo" >> lib/readme && git add lib/readme && git commit -m "p1 initial add to subrepo" && git subrepo push --all
cd ../p2 && git subrepo pull --all
echo "p2 initial add to subrepo" >> lib/readme && git add lib/readme && git commit -m "p2 initial add to subrepo" && git subrepo push --all
cd ../p1 && git subrepo pull --all -v

That last command should pull the latest changes to the library into p1, but it fails:

$ git subrepo pull --all -v
>>> git rev-parse --verify HEAD
>>> git fetch --quiet ../lib master
>>> git rev-parse FETCH_HEAD
>>> git for-each-ref refs/subrepo/remote/lib
>>> git log --max-count=1 --format=%T 6cef7e0409fce373c483ce9d463c4097e9a79003
>>> git commit-tree 4c5947676b23bfe76de4800158946e215c25b56d -p 90fa69483974db8d6b4c025f07dac72a9d5e4e3e -m git subrepo pull lib ./lib/ {"local": {"subdir": "lib" ,"action": "pull"} ,"remote": {"url": "../lib" ,"branch": "master" ,"commit": "6cef7e0"} ,"git-subrepo": {"version": "0.1.0" ,"commit": "???" ,"origin": "???"}}
>>> git merge -s recursive -X subtree=lib 7a13f51898532b547fe90944fa391e8371ca03e8
>>> git filter-branch -f --parent-filter sed 's/ -p 90fa69483974db8d6b4c025f07dac72a9d5e4e3e//' -- 7a13f51898532b547fe90944fa391e8371ca03e8^..HEAD ^HEAD^
Cannot rewrite branches: You have unstaged changes.
Additionally, your index contains uncommitted changes.
>>> git rev-parse HEAD^2
fatal: ambiguous argument 'HEAD^2': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
>>> git config --file=lib/.gitrepo subrepo.commit 6cef7e0409fce373c483ce9d463c4097e9a79003
>>> git config --file=lib/.gitrepo subrepo.parent a515f37325a21c3ca0f4df947bed93006ef6096b
>>> git config --file=lib/.gitrepo subrepo.cmdver 0.1.0
>>> git add lib/.gitrepo
fatal: Needed a single revision
>>> git commit --quiet --amend --message=Merge subrepo commit ''
fatal: You are in the middle of a merge -- cannot amend.
git subrepo 'lib' pulled from '../lib' (master)

$ git status
On branch master
You have unmerged paths.
  (fix conflicts and run "git commit")

Changes to be committed:

        modified:   lib/.gitrepo

Unmerged paths:
  (use "git add <file>..." to mark resolution)

        both added:         lib/readme

subrepo:
  subdir:   "ext/test-tap-bash"
  merged:   "badc0fe"
upstream:
  origin:   "[email protected]:ingydotnet/test-tap-bash.git"
  branch:   "master"
  commit:   "8c660ae"

That way the owner can get at this ref even they clean. It will hang around
until gc.

This ref can be used later for a push, and it might represent quite a bit of
merge conflict resolution.

We need to test the output of commands when they succeed, and also fail.

We should also test --verbose and --debug output.

Make these instructions and Makefile match git-hub:

• Just modify PATH and MANPATH for easy install
• make install for /system/ install

I got messed up because my Cygwin didn't have wc available. git-hub does this
kind of check. git-subrepo should too.

I got a commit message like:

git subrepo commit ext/bashplus/ subrepo/ext/bashplus/push

should be:

git subrepo commit ext/bashplus subrepo/ext/bashplus/push

Since clones in git-subrepo are shallow by default, it seems like there is no way to specify any commit other than ones pointed to by branches.

Would you consider adding this feature or suggesting a workaround?

Mostly, because I doubt most people would expect it to be historically retroactive rebasing.

And lots of people are rebase-phobic.
Maybe you could have git subrepo remove simply return explanation and usage.

And use

git subrepo remove --rebase or git subrepo remove --deep

To indicate "Yes, I am happy with retroactively munging my history".

Or something.

And contrast with git subrepo remove --shallow which may be a proxy for git rm

And maybe git subrepo remove --depth=50 or git subrepo remove --since=sha1 to remove it only after a certain time frame. ( ie: if people want to remove it from a commit series that has not been distributed yet )

This stuff should all work on Cygwin.

There is a possibility it can run in cmd (DOS) shell as well, but need to
investigate. A friend told me that he can run any cygwin commands from cmd as
long as they are in the PATH.

See how git-hub does it.

This flag applies to these commands:

• fetch
• branch
• pull
• push
• clean

We should:

• Test that it works for each command
• Test what happens when one or more fail

commit shouldn't fetch by default. It should work offline, but it needs
upstream ref so, we can add 'Try --fetch' to error message and support it.

 git subrepo clone https://github.com/kentfredric/travis-scripts.git maint-travis-ci -b master
git subrepo 'maint-travis-ci' cloned from 'https://github.com/kentfredric/travis-scripts.git' (master)

Some time passess and the repo gets updated

git subrepo pull maint-travis-ci/ -v -b master
fatal: entry  not found in tree a91a135c1918c2c3dcd84469c32218c7887872a5

Duhhh....

fatal: entry  not found in tree a91a135c1918c2c3dcd84469c32218c7887872a5

There's literally nothing I can do to get around this, each command I try gives the same error, and no additional context.

I'm used to using subtree pull <github> and conversely subtree push <localrepo>,
because I always want to sync with the latest published history, but if I'm pushing history, I want to push locally and to a target branch so I can iron out any merge problems before the change gets pushed publically.

This is because each repository with a sub-repo is effectively a unique branch deviating from $origin, and a plain push will thus likely be akin to "git push $origin -f "

So instead, I push the hydra of heads to a single repo to resolve merge problems locally.