Concurrency issue with Atlas and Mongo persistence about spline

[vackosar]

Can you share your configuration?

[rjbarrington]

Something like this in spline.properties, run via spark-submit --class za.co.absa.spline.sample.batch.SampleJob2 spline-sample-0.3.1.jar

# Spline lineage config
spline.persistence.factory=za.co.absa.spline.persistence.api.composition.ParallelCompositeFactory
spline.persistence.composition.factories=za.co.absa.spline.persistence.atlas.AtlasPersistenceFactory,za.co.absa.spline.persistence.mongo.MongoPersistenceFactory

# MongoDB config for UI
spline.mongodb.url=mongodb://bar.domain:27017
spline.mongodb.name=spline

# Atlas Kafka config
atlas.kafka.bootstrap.servers=foo.domain:9027

# Combinations tested with Spline SampleJob2
# HDFS + Atlas + Mongo = fail
#spline.persistence.composition.factories=za.co.absa.spline.persistence.hdfs.HdfsPersistenceFactory,za.co.absa.spline.persistence.atlas.AtlasPersistenceFactory,za.co.absa.spline.persistence.mongo.MongoPersistenceFactory
# Atlas + Mongo = fail
#spline.persistence.composition.factories=za.co.absa.spline.persistence.atlas.AtlasPersistenceFactory,za.co.absa.spline.persistence.mongo.MongoPersistenceFactory
# HDFS + Atlas = works
#spline.persistence.composition.factories=za.co.absa.spline.persistence.hdfs.HdfsPersistenceFactory,za.co.absa.spline.persistence.atlas.AtlasPersistenceFactory
# HDFS + Mongo = works
#spline.persistence.composition.factories=za.co.absa.spline.persistence.hdfs.HdfsPersistenceFactory,za.co.absa.spline.persistence.mongo.MongoPersistenceFactory

[vackosar]

Does this occur also when using Atlas only?

[vackosar]

hm I can see mutable ArrayBuffer in the stack that could be the problem.

[vackosar]

I reproduced the exception. I will have a quick look now.

[vackosar]

This is caused by simultaneously linking to other lineages and saving to Atlas as well. I will implement fix storing non-linked lineages only to Atlas.

[rjbarrington]

Just confirming that it doesn't occur when using Atlas only. Thanks for checking it out.

[vackosar]

The fix commit is available on release branch 0.3 here: 179283e
I dont want to release new release just yet due to significant overhead of it.
Can you build it from the source using the snapshot version at the moment?

[vackosar]

Let me know if u have issues with above solution.

[rjbarrington]

Sorry, got busy. I'll have a look this weekend.

[rjbarrington]

Yep, that works for me.