Facing issues with json4s package, while saving model. Also not able to create a fat jar due to version conflict between liberaries. about isolation-forest

[sandeep1288]

Thanks a lot, I'm able to make it working with latest versions of spark.

[sandeep1288]

Below function threw error:
private def getMetadataToSave(
instance: Params,
spark: SparkSession,
extraMetadata: Option[JObject] = None): String = {

  val uid = instance.uid
  val cls = instance.getClass.getName
  val params = instance.extractParamMap.toSeq
  val jsonParams = render(params.map { case ParamPair(p, v) =>
    p.name -> parse(p.jsonEncode(v))
  }.toList)
  val basicMetadata = ("class" -> cls) ~
    ("timestamp" -> System.currentTimeMillis()) ~
    ("sparkVersion" -> spark.sparkContext.version) ~
    ("uid" -> uid) ~
    ("paramMap" -> jsonParams)
  val metadata = extraMetadata match {
    case Some(jObject) => basicMetadata ~ jObject
    case None => basicMetadata
  }
  val metadataJson = compact(render(metadata))

  metadataJson
}

}

And problem is happening while saving model.

[jverbus]

What version of Spark and Scala are you using?

The recommended versions for the project are:

Scala 2.11.8
Spark 2.3.0

These are the versions used to produce the jars on Bintray.

[sandeep1288]

I'm using databrick platform, and below is the scala and spark version:
spark-version : 2.4.3
scala-version : 2.11.12

In order to make current iForest working on latest spark version, what are the essential steps which I can follow. There should not be much difference between spark 2.3 and 2.4, apart from json4s.

[jverbus]

It is currently recommended to use Scala 2.11 and Spark 2.3.

You will need to build the isolation-forest library yourself using Spark 2.4 if you want to use it in a Spark 2.4 environment. You will need to change some dependencies in the module-level build.gradle file.

Please try the instructions in this ticket: #1

[sandeep1288]

Thanks a lot. I'll give it a try.

[jverbus]

@sandeep1288: Cool. Please let me know if it works. Thanks!

[sandeep1288]

It worked actually. Thanks a lot for your help. Sent from Yahoo Mail on Android On Wed, 18 Sep 2019 at 12:57, James Verbus<[email protected]> wrote: @sandeep1288: Cool. Please let me know if it works. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.