Flaky tests: org.eclipse.jgit.errors.MissingObjectException about gemini

[smacker]

Currently, we have 4 fixtures as siva files.
I can reproduce the bug without running tests but just hash command.

First I tried to remove all fixtures and recover only one. Then run hash command multiple times.
To distinguish bugs of engine & fixtures. If I can reproduce for every fixture - it means the problem is in the engine and doesn't depend on fixtures. If not, the problem is engine + particular fixture.

#!/bin/bash

set -e

i="0"
while [ $i -lt 1000 ]
do
    ./hash $1
    i=$[$i+1]
done

Can reproduce with:
2636f3c62f1a407b2996da6e3fe6fdc5d1ccd764.siva
5fb38a5744b2496ff8484e57b46a754433ede457.siva
9279be3cf07fb3cca4fc964b27acea57e0af461b.siva

Error: org.eclipse.jgit.errors.MissingObjectException: Missing commit <sha1>

Can not reproduce with:
f281ab6f2e0e38dcc3af05360667d8f530c00103.siva

[smacker]

Next, I tried to update engine to v0.2 and see if I still can reproduce the bug.
I chose to use v0.2 instead of v0.3.1 because v0.3 requires repository format and it will cause larger changes in gemini code.

With 0.2 we actually don't need to join anymore, but I kept it to for testing purposes.
With 2636f3c62f1a407b2996da6e3fe6fdc5d1ccd764.siva I still can get Missing commit error.
Then I removed join (all necessary staff is already in getBlobs column), without join it works as expected.

[smacker]

Here is a PR with engine 0.2: #26

Here is spark-shell for different engine versions. Looks like bug has been fixed (or this time I was unlucky to catch it)

spark-shell for v0.2.0: (fails)

import tech.sourced.engine._
val engine = Engine(spark, "/Users/smacker/Work/gemini/src/test/resources/siva")
val headRefs = engine.getRepositories.getReferences.filter("name = 'refs/heads/HEAD' OR name = 'HEAD'").withColumnRenamed("hash", "commit_hash")
val files = headRefs.getCommits.getFirstReferenceCommit.getTreeEntries.getBlobs.select("blob_id", "commit_hash", "path")
val filesInRepos = files.join(headRefs, "commit_hash")
filesInRepos.select("blob_id", "path", "repository_id").collect()

spark-shell for v0.3.1: (works)

import tech.sourced.engine._
val engine = Engine(spark, "/Users/smacker/Work/gemini/src/test/resources/siva", "siva")
val headRefs = engine.getRepositories.getReferences.filter("name = 'refs/heads/HEAD' OR name = 'HEAD'").withColumnRenamed("hash", "commit_hash")
val files = headRefs.getCommits.getFirstReferenceCommit.getTreeEntries.getBlobs.select("blob_id", "commit_hash", "path")
val filesInRepos = files.join(headRefs, "commit_hash")
filesInRepos.select("blob_id", "path", "repository_id").collect()

[smacker]

Fixed.