Document/provide example of external AdamPlugin (not part of build) about adam

[tdanford]

Michael, can you give a bit more detail on what would satisfy this issue (for you)? Is this a documentation-only issue? Or do we actually need to create a separate repo somewhere with an example of building a plugin who's code is not part of the core 'adam' repo?

[heuermh]

I can work on an example for this.

[tdanford]

That'd be great! We also have a very simple example in adam-cli/src/test/scala, I wasn't sure if that's the kind of example you were thinking of or if you wanted something more specific.

[heuermh]

Thanks, I've seen that example, but it doesn't really satisfy a true plugin scenario, since Take10Plugin is built as part of adam-core and as such is already on the classpath for adam-cli. The example for this issue is necessary to prove out the build can work.

[tdanford]

I'll leave it up to you Michael :-)

On Mon, Mar 3, 2014 at 1:31 PM, Michael L Heuer [email protected]:

Thanks, I've seen that example, but it doesn't really satisfy a true
plugin scenario, since Take10Plugin is built as part of adam-core and as
such is already on the classpath for adam-cli. The example for this issue
is necessary to prove out the build can work.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/131#issuecomment-36541672
.

[heuermh]

ok, this builds, passes tests, and creates adam-plugins-0.6.1-SNAPSHOT.jar, but I'm not sure how to run it with adam-cli.

https://github.com/heuermh/adam/tree/plugin-example

Would it just require a shell script that adds adam-plugins-0.6.1-SNAPSHOT.jar to the classpath and then calls Main in adam-0.6.1-SNAPSHOT.jar?

Here's the required documentation part. :)

[heuermh]

Something with command line parameters is not working right:

$ java -classpath "target/adam-plugins-0.6.1-SNAPSHOT.jar:../adam-cli/target/adam-0.6.1-SNAPSHOT.jar" edu.berkeley.cs.amplab.adam.cli.AdamMain plugin ../adam-core/src/test/resources/reads12.sam 
0   1   249250621
1   2   243199373

$ java -classpath "target/adam-plugins-0.6.1-SNAPSHOT.jar:../adam-cli/target/adam-0.6.1-SNAPSHOT.jar" edu.berkeley.cs.amplab.adam.cli.AdamMain plugin edu.berkeley.cs.amplab.plugins.external.ExternalTake10Plugin 
Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/Users/mheuer/working/adam/adam-plugins/edu.berkeley.cs.amplab.plugins.external.ExternalTake10Plugin
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:285)
    at parquet.hadoop.ParquetInputFormat.listStatus(ParquetInputFormat.java:291)
    at parquet.hadoop.ParquetInputFormat.getFooters(ParquetInputFormat.java:339)
    at parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:246)
    at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:66)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:201)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:199)
    at scala.Option.getOrElse(Option.scala:108)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:199)
    at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:26)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:201)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:199)
    at scala.Option.getOrElse(Option.scala:108)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:199)
    at org.apache.spark.rdd.RDD.distinct(RDD.scala:266)
    at edu.berkeley.cs.amplab.adam.rdd.AdamContext.adamDictionaryLoad(AdamContext.scala:240)
    at edu.berkeley.cs.amplab.adam.cli.ListDict.run(ListDict.scala:48)
    at edu.berkeley.cs.amplab.adam.cli.AdamSparkCommand$class.run(AdamCommand.scala:48)
    at edu.berkeley.cs.amplab.adam.cli.ListDict.run(ListDict.scala:41)
    at edu.berkeley.cs.amplab.adam.cli.AdamMain$.main(AdamMain.scala:65)
    at edu.berkeley.cs.amplab.adam.cli.AdamMain.main(AdamMain.scala)

$ java -classpath "target/adam-plugins-0.6.1-SNAPSHOT.jar:../adam-cli/target/adam-0.6.1-SNAPSHOT.jar" edu.berkeley.cs.amplab.adam.cli.AdamMain plugin edu.berkeley.cs.amplab.plugins.external.ExternalTake10Plugin ../adam-core/src/test/resources/reads12.sam 
Too many arguments: ../adam-core/src/test/resources/reads12.sam
 INPUT                                                           : The ADAM sequence dictionary to print
 -h (-help, --help, -?)                                          : Print help
 -parquet_block_size N                                           : Parquet block size (default = 128mb)
 -parquet_compression_codec [UNCOMPRESSED | SNAPPY | GZIP | LZO] : Parquet compression codec
 -parquet_disable_dictionary                                     : Disable dictionary encoding
 -parquet_page_size N                                            : Parquet page size (default = 1mb)
 -spark_env KEY=VALUE                                            : Add Spark environment variable
 -spark_home PATH                                                : Spark home
 -spark_jar JAR                                                  : Add Spark jar
 -spark_master VAL                                               : Spark Master (default = "local[#cores]")

[heuermh]

Created new issue #164 for the problem above

[heuermh]

I've also created an separate repo external adam-plugins module here

https://github.com/heuermh/adam-plugins

It does not use adam-parent as its parent pom.

[tdanford]

@heuermh #164, referenced above, is closed -- and your repo seems like a reasonable plugin example. Shall we close this issue too?

[heuermh]

Yes, thanks.

[fnothaft]

Thanks @heuermh! Closing.