java.lang.UnsatisfiedLinkError: org.rocksdb.LRUCache.newLRUCache(JIZD)J when trying to do NER using NerCrf about spark-nlp

[aleksei-ai]

@eoinlane It seems that RocksDB 5.2.1 is already used by DataBricks https://docs.databricks.com/release-notes/runtime/3.5.html

Probably we can fix it with shading or release special version for DataBricks

[eoinlane]

Thanks aleksei-ai, a release special version for DataBricks would be great.

[albertoandreottiATgmail]

hi @eoinlane, until a special release for Databricks comes along, you can use the following image to create your cluster in Databricks platform,
Spark 2.1 (Auto-updating, Scala2.11)

I tried that image and was able to run your code without problems.

[albertoandreottiATgmail]

I created a jar that uses older 5.2.1 rocksdb library, that will enable the use of the latest 3.5 Databricks images, take a look here,
https://drive.google.com/open?id=1WwNDDjPXBo3LDxYtbsADscwnDFLXKkNv

@saifjsl, @aleksei-ai, A PR for this can be found (here).

[eoinlane]

I can confirm that both of those options are working and look forward to a supported Databricks release going forward. Thanks for all your help.

[saif-ellafi]

Going with this downgrade fix for now #109
We should consider longer term solution though.