BwaSpark parameter optimization about gatk

[lbergelson]

It won't be able to run any faster than BWA mem does with a similar number of cores, since it is essentially just running bwamem. It's potentially faster as part of a spark pipeline so you can load and process data once instead of saving the data to disk and reloading it repeatedly.

The complete list of spark configuration parameters is available on the spark docs. Many of them are not relevant in local mode. From what I understand the local mode is going to execute as a single executor with the number of cores specified in the local[#] block ( or the total number of system threads if it's set to *) It will use the available memory that java is configured with. I'm pretty sure it's ignoring the memory and configuration parameters you've set. Those will be relevant if you configure a stand alone spark cluster (potentially one running exclusively on your local machine).

Our spark tools are not being actively developed for the most part. We've moved away from them to use single threaded tools widely sharded and managed by cromwell. The additional complexity of the spark environment made it hard to see much benefit when most of the tools are embarassingly parallel and easily shardable.