yahooarchive / omid Goto Github PK

Whilst exploring the code I've realized that there's no concept of transaction timeout. Maybe it would be interesting to add it in order to automatically abort transactions that are never explicitly committed nor rolled-back by clients.

I think the combination of Omid plus the Phoenix (https://github.com/forcedotcom/phoenix) would be pretty powerful. Any interest?

We've got a related issue over on our github repo here:
forcedotcom/phoenix#209 (comment)

Or feel free to contract me off-list: [email protected]

Thanks.

I am referring to the 'replication' branch (the relevant data structure in the main branch is 'messageBuffersMap', I guess). The aforementioned set seems to be only growing and never deflates. There has to be a mechanism that removes buffers from this data structure once a client disconnects or when a client is idle for too long.

All uncommitted transactions that were running before a SO crash have to be aborted after its recovery. We can keep track of gaps on the transaction id space and mark all gaps as aborted.

Thanks to Maysam for noticing this problem.

Not sure if there is an issue with maven-antrun-plugin or not, but are you using Maven 2?
Have you tested with Maven 3?

Cheers,
Nick

Make the TSOServer's main method more flexible, allowing it to be ran either with BK enabled or disabled and let the user set several parameters.

The ' characters around the executable and arguments are
not part of the command.
[exec] mkdir -p ../../../target/main/native
[exec] g++ -fPIC -O3 -I/usr/local/java/jdk1.7.0_10/include -I/usr/local/java/jdk1.7.0_10/include/linux -I../../../target/main/native -c nativelib.cc -o ../../../target/main/native/nativelib.o
[exec] g++ -fPIC -lstdc++ -shared -Wl,-soname=libtso-commithashmap.so ../../../target/main/native/nativelib.o -o ../../../lib/libtso-co/usr/bin/ld: cannot open output file ../../../lib/libtso-commithashmap.so: No such file or directory
[exec] collect2: error: ld returned 1 exit status
[exec] make: *** [../../../lib/libtso-commithashmap.so] Error 1
[exec] mmithashmap.so
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.944s
[INFO] Finished at: Tue Dec 18 09:51:04 GST 2012
[INFO] Final Memory: 18M/239M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (build-native) on project omid: An Ant BuildException has occured: exec returned: 2
[ERROR] around Ant part ...... @ 13:97 in /home/nazakat/Downloads/omid/target/antrun/build-jni.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (build-native) on project omid: An Ant BuildException has occured: exec returned: 2
around Ant part ...... @ 13:97 in /home/nazakat/Downloads/omid/target/antrun/build-jni.xml
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: exec returned: 2
around Ant part ...... @ 13:97 in /home/nazakat/Downloads/omid/target/antrun/build-jni.xml
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
... 19 more
Caused by: /home/nazakat/Downloads/omid/target/antrun/build-jni.xml:13: exec returned: 2
at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:646)
at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:672)
at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:498)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:390)
at org.apache.tools.ant.Target.performTasks(Target.java:411)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:327)
... 21 more
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
nazakat@nazakat-HP:~/Downloads/omid$

if (msg.rows.length > 0 && msg.startTimestamp < sharedState.largestDeletedTimestamp) == true，then "for (RowKey r : msg.rows)" never execute。

if (msg.rows.length > 0 && msg.startTimestamp < sharedState.largestDeletedTimestamp) == false
=> "msg.rows.length > 0 is false" OR "msg.startTimestamp < sharedState.largestDeletedTimestamp is false"

if "msg.rows.length > 0 is false" then "for (RowKey r : msg.rows)" never execute。
if "msg.startTimestamp < sharedState.largestDeletedTimestamp is false"
=>(value == 0 && sharedState.largestDeletedTimestamp > msg.startTimestamp) == false

So, I think, the [else if ] branch "write-write conflicts check" is not necessary。

at tso/ClientHandler.java which is to test the status oracle, here is the way we generate the row ids:

byte writtenSize = readOnly ? 0 : (byte) rnd.nextInt(MAX_ROW);

if MAX_ROW is larger than max unsigned byte (i.e., 127). then the result could be negative. This is because the byte sign is the 8th bit.

We have inconsistent indentation thorough the project, both 3 and 4 spaces are used.

I propose we switch to 4 spaces as I think it is more standard (if some indentation can ever be standard...)

When the indentation policy is decided upon I don't know how should we change the offending files. One by one as they are modified or all of them at once losing the history?

Thoughts?

We should be rebuilding the state serially, because some operations should not be reordered. I think should read the log in parallel building a sequential list of operations until we reach a stop condition, and then replay these operations sequentially.

Currently OMID works with HBase 0.98. Do you have plans to upgrade to hbase 1.0/1.1? Thanks!

Currently, if two concurrent transactions share the same TSOClient, there could be a problem regarding aborted transactions.

The problem arises if a FullAbortedReport is received and the aborted transaction is cleaned up while other concurrent transactions still need this information.

One possible solution is to delay applying the FullAbortedReport until all previous transactions have completed.

Thanks to Maysam for finding the bug and providing a fix.