damavis / damavis-spark Goto Github PK

Issue Description:

In the FileReaderBuilder class under the com.damavis.spark.resource.file package, I've observed that the option and options functions behave differently, which may lead to varying interpretations by users.

option: Adds an option to the existing set of options.
options: Overwrites the existing options with the provided options map.

Expected Behavior

I propose that the options function should merge the provided options with the existing options using a strategy that allows users to add or update options without completely overwriting them. This would make the behavior of options more consistent with the behavior of option.

This change might also necessitate corresponding updates in the FileWriterBuilder class for consistency.

Example

For instance, consider the following code snippet:

FileReaderBuilder(format, path)
      .options(
        Map(
          "inferSchema" -> inferSchema.toString,
          "header" -> header.toString,
          "delimiter" -> delimiter
        ))
      .options(options)

In this example, it could be beneficial for users if the second options call merged the provided options with the ones set in the first options call, rather than entirely overwriting them.

Hi!

I've noticed the project's support for Spark 3.0.0, yet the build.sbt file employs Spark 2.4.5 for testing:

val sparkVersion = "3.0.0"
val sparkTestVersion = "2.4.5"
...
"com.holdenkarau" %% "spark-testing-base" % s"${sparkTestVersion}_0.14.0" % Test

I'm curious about the rationale behind this. Wouldn't it be more suitable to align the testing version with 3.0.0?

Attempting an upgrade, I found that there's no equivalent to com.holdenkarau.spark.testing.HDFSCluster in spark-testing-base version 1.X.X, a class heavily relied upon in our tests. (holdenk/spark-testing-base@09b031a)

Considering this, do you think it's worthwhile to invest effort in adapting the tests to accommodate Spark 3.0.0?

Hi, @piffall , @priera , I'd like to report a vulnerable dependency in com.damavis:damavis-spark-core_2.12:0.3.10.

Issue Description

I noticed that com.damavis:damavis-spark-core_2.12:0.3.10 directly depends on org.apache.spark:spark-core_2.12:3.0.0 in the pom. However, as shown in the following dependency graph, org.apache.spark:spark-core_2.12:3.0.0 sufferes from the vulnerability which the C library zstd(version:1.4.4) exposed: CVE-2021-24032.

Dependency Graph between Java and Shared Libraries

Suggested Vulnerability Patch Versions

org.apache.spark:spark-core_2.12:3.2.0 (>=3.2.0) has upgraded this vulnerable C library zstd to the patch version 1.5.0.

Java build tools cannot report vulnerable C libraries, which may induce potential security issues to many downstream Java projects. Could you please upgrade this vulnerable dependency?

Thanks for your help~
Best regards,
Helen Parr