Error: panic: commit is not available about git-sizer

[mhagger]

After I running git filter-branch -f --prune-empty -- --all on my repo git sizer was happy.

@larsxschneider: presumably your repository had replace references or grafts, and the reason that git filter-branch fixed the problem is that it "baked them into" the actual repository structure. I doubt that the --prune-empty option was actually necessary.

[dscho]

Does GIT_GRAFT_FILE=NUL also work?

Yes, it does work, too!

[mhagger]

Are you using the newest version (v1.1) of git-sizer?

Is the repository publicly available so that I can try to reproduce the problem?

[jeremywrowe]

I've experienced the same with v1.1, this happens on rails/rails for me.

[itsbp]

Yes, I was using v1.1, sorry but repo I tried is not publicly available.

[mhagger]

Hmm, I can't reproduce this bug. I tried cloning rails/rails from github then ran git-sizer -v against it (version 1.1), but no error. I tried using Git 2.3.16, 2.14.2, and master. @jeremywrowe, which Git version do you have in your PATH?

[larsxschneider]

I experienced the same error too with git-sizer v1.1 and git 2.16.2 on Linux.

[jeremywrowe]

git-sizer v1.1 and git 2.16.2 on os x

[larsxschneider]

I think empty commits might be problematic in certain cases. After I running git filter-branch -f --prune-empty -- --all on my repo git sizer was happy.

[mhagger]

I tried in the simplest possible way to create an empty commit git commit --allow-empty [...] then run git-sizer on the repo. It didn't fail. Also, it seems that an empty commit would be an unlikely trigger for a git-sizer failure.

My guess is that git filter-branch -f --prune-empty -- --all rearranged a lot of things (e.g., including the commit topology) in such a way that the bug wasn't triggered.

If I could get my hands on a repo that triggers this bug reliably, it would be a big help. Until then, I'd just be guessing.

[mhagger]

I forgot to ask an important question: does git fsck --no-dangling run cleanly on the affected repositories?

Meanwhile, it occurred to me that I have about 80 Million Git repositories that I can run git-sizer against (output anonymized, of course) to look for problems. Maybe I can find one there that triggers the bug.

[larsxschneider]

git fsck --no-dangling did not fix the problem 😢

I can't share the repo but I am happy to run a git-sizer version with relevant debug output for you.

[mhagger]

git fsck --no-dangling did not fix the problem

@larsxschneider: Just to be sure we understand each other...

git fsck doesn't actually fix anything in a Git repository. If git fsck --no-dangling reports errors in a repository, then the repository is corrupt in one way or another and it would not be surprising for git-sizer to fail on that repo. If git fsck --no-dangling doesn't report any problems, then the repository is presumably not corrupt, so git-sizer should hopefully succeed.

I assume that you meant that "git fsck --no-dangling doesn't report any problems, so repository corruption doesn't seem to be the explanation for why git-sizer fails." But if you meant something else, please clarify.

I've run git-sizer on tens of thousands of repositories and haven't yet found an error that wasn't explained by some kind of mild repository corruption. But I'll keep trying.

If I come up with any theories for why the failure occurs, then I might take you up on your offer to run a debugging version of git-sizer against your repository.

[larsxschneider]

Oops. I am sorry, of course git fsck doesn't fix anything.

It doesn't report any errors on my repo either:

$ git fsck --no-dangling
Checking object directories: 100% (256/256), done.
Checking objects: 100% (2265686/2265686), done.
Checking connectivity: 1932408, done.

$ ~/bin/git-sizer
Processing blobs: 700857
Processing trees: 998950
panic: commit is not available

goroutine 1 [running]:
github.com/github/git-sizer/sizes.(*Graph).GetCommitSize(0xc4200c6000, 0x765aca5c538afed6, 0xf79c1b412029dceb, 0x5f4071f7, 0xc30000000f)
	/home/mhagger/github/proj/git-sizer/git/.gopath/src/github.com/github/git-sizer/sizes/graph.go:630 +0xc6
github.com/github/git-sizer/sizes.(*Graph).RegisterCommit(0xc4200c6000, 0x1efdd41421eb6764, 0x9c655f3f1847d6b, 0xc4cef55c9a, 0xc43b2ec880)
	/home/mhagger/github/proj/git-sizer/git/.gopath/src/github.com/github/git-sizer/sizes/graph.go:656 +0x1e1
github.com/github/git-sizer/sizes.ScanRepositoryUsingGraph(0xc4200c2000, 0x5687e8, 0x2, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/mhagger/github/proj/git-sizer/git/.gopath/src/github.com/github/git-sizer/sizes/graph.go:281 +0x11e8
main.mainImplementation(0x0, 0x0)
	/home/mhagger/github/proj/git-sizer/git/.gopath/src/github.com/github/git-sizer/git-sizer.go:150 +0x8ef
main.main()
	/home/mhagger/github/proj/git-sizer/git/.gopath/src/github.com/github/git-sizer/git-sizer.go:46 +0x26

[asheiduk]

I'm hit by that bug/message to. In my case .git/info/grafts is the culprit. When I remove that file it works, when I move it back, it breaks.

See git h repository-layout for a short description of that file.

[mhagger]

I'm hit by that bug/message to. In my case .git/info/grafts is the culprit.

Oh, that makes perfect sense. info/grafts or replace references could break invariants that git-sizer depends on. @larsxschneider and others, do your repos all have grafts and/or replace references?

I think there are env variables that can be used to disable both of those. I'll check the details when I'm at a computer again.

BTW, I ran git-sizer against over 100k repositories and none failed except for reasons of minor corruption. But none of those repos had grafts, and they all had replace refs turned off.

[asheiduk]

@mhagger Well, neither grafts nor replace references are cloned/pushed/fetched by default (grafts cannot, replacements can but not by default). So both things actually live only in local repositories where real users work.

[mhagger]

I looked into how to disable these things.

• Replace references can be disabled by running Git with git --no-replace-objects [...] or by setting the (undocumented) environment variable GIT_NO_REPLACE_OBJECTS=1.
• Grafts can be disabled by setting the (undocumented) environment variable GIT_GRAFT_FILE=<filename>, where <filename> is an empty or non-existent file.
• Shallow clones (detectable from the existence of $GIT_DIR/shallow, I believe) would also be a problem. (The location of the shallow file can be influenced by the environment variable GIT_SHALLOW_FILE, but I don't think that's useful.) We probably need to disallow running git-sizer in a shallow repository at all.

We should build these things into git-sizer. Meanwhile, those of you who experienced this bug should be able to get around it by running

GIT_GRAFT_FILE=/dev/null git --no-replace-objects sizer [...]

Please confirm whether that fixes your problems.

[larsxschneider]

@mhagger : 💯yes. That is exactly what happened. @asheiduk thanks for bringing up the grafts. I totally forgot about them in my repo.

[dscho]

GIT_GRAFT_FILE=/dev/null git --no-replace-objects sizer

@mhagger FWIW this works on Windows (I tested specifically the /dev/null part). It also works in CMD (of course you have to set the environment variable separately, then).

[mhagger]

@dscho: Thanks for checking. Does GIT_GRAFT_FILE=NUL also work? That's what #31 currently implements, and the code is IMO more straightforward that way (just rely on os.DevNull rather than having to worry that some other OS uses something different here).