Comments (4)
Currently, we have 4 fixtures as siva files.
I can reproduce the bug without running tests but just hash command.
First I tried to remove all fixtures and recover only one. Then run hash
command multiple times.
To distinguish bugs of engine & fixtures. If I can reproduce for every fixture - it means the problem is in the engine and doesn't depend on fixtures. If not, the problem is engine + particular fixture.
#!/bin/bash
set -e
i="0"
while [ $i -lt 1000 ]
do
./hash $1
i=$[$i+1]
done
Can reproduce with:
2636f3c62f1a407b2996da6e3fe6fdc5d1ccd764.siva
5fb38a5744b2496ff8484e57b46a754433ede457.siva
9279be3cf07fb3cca4fc964b27acea57e0af461b.siva
Error: org.eclipse.jgit.errors.MissingObjectException: Missing commit <sha1>
Can not reproduce with:
f281ab6f2e0e38dcc3af05360667d8f530c00103.siva
from gemini.
Next, I tried to update engine to v0.2 and see if I still can reproduce the bug.
I chose to use v0.2 instead of v0.3.1 because v0.3 requires repository format and it will cause larger changes in gemini code.
With 0.2 we actually don't need to join anymore, but I kept it to for testing purposes.
With 2636f3c62f1a407b2996da6e3fe6fdc5d1ccd764.siva
I still can get Missing commit
error.
Then I removed join (all necessary staff is already in getBlobs column), without join it works as expected.
from gemini.
Here is a PR with engine 0.2: #26
Here is spark-shell for different engine versions. Looks like bug has been fixed (or this time I was unlucky to catch it)
spark-shell for v0.2.0: (fails)
import tech.sourced.engine._
val engine = Engine(spark, "/Users/smacker/Work/gemini/src/test/resources/siva")
val headRefs = engine.getRepositories.getReferences.filter("name = 'refs/heads/HEAD' OR name = 'HEAD'").withColumnRenamed("hash", "commit_hash")
val files = headRefs.getCommits.getFirstReferenceCommit.getTreeEntries.getBlobs.select("blob_id", "commit_hash", "path")
val filesInRepos = files.join(headRefs, "commit_hash")
filesInRepos.select("blob_id", "path", "repository_id").collect()
spark-shell for v0.3.1: (works)
import tech.sourced.engine._
val engine = Engine(spark, "/Users/smacker/Work/gemini/src/test/resources/siva", "siva")
val headRefs = engine.getRepositories.getReferences.filter("name = 'refs/heads/HEAD' OR name = 'HEAD'").withColumnRenamed("hash", "commit_hash")
val files = headRefs.getCommits.getFirstReferenceCommit.getTreeEntries.getBlobs.select("blob_id", "commit_hash", "path")
val filesInRepos = files.join(headRefs, "commit_hash")
filesInRepos.select("blob_id", "path", "repository_id").collect()
from gemini.
Fixed.
from gemini.
Related Issues (20)
- Document ability to work with distrubited date sources
- Move docker build/upload in travis to separate stage
- Incorrect listing of repositories for bare/standard mode
- Make some constants configurable
- Remove feature-extractor service
- Update user documentation how to run gemini
- Update development documentation
- Investigate if it is possible to run igraph from scala
- Create a script for fixtures update
- Document how to run subsets of tests
- Setup travis to re-run tests weekly
- Update dependencies
- Migrate to gitbase-spark-connector
- Remove unused code
- Gemini can't detect similar C# files
- Create a doc with list of improvements HOT 2
- All log messages should go to stderr
- AbstractMethodError after start HOT 1
- How to generate a repo's siva file? HOT 1
- Running with docker compose fails connecting to Cassandra HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gemini.