damavis / damavis-spark Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
Issue Description:
In the FileReaderBuilder class under the com.damavis.spark.resource.file package, I've observed that the option and options functions behave differently, which may lead to varying interpretations by users.
option: Adds an option to the existing set of options.
options: Overwrites the existing options with the provided options map.
Expected Behavior
I propose that the options function should merge the provided options with the existing options using a strategy that allows users to add or update options without completely overwriting them. This would make the behavior of options more consistent with the behavior of option.
This change might also necessitate corresponding updates in the FileWriterBuilder class for consistency.
Example
For instance, consider the following code snippet:
FileReaderBuilder(format, path)
.options(
Map(
"inferSchema" -> inferSchema.toString,
"header" -> header.toString,
"delimiter" -> delimiter
))
.options(options)
In this example, it could be beneficial for users if the second options call merged the provided options with the ones set in the first options call, rather than entirely overwriting them.
Hi!
I've noticed the project's support for Spark 3.0.0, yet the build.sbt file employs Spark 2.4.5 for testing:
val sparkVersion = "3.0.0"
val sparkTestVersion = "2.4.5"
...
"com.holdenkarau" %% "spark-testing-base" % s"${sparkTestVersion}_0.14.0" % Test
I'm curious about the rationale behind this. Wouldn't it be more suitable to align the testing version with 3.0.0?
Attempting an upgrade, I found that there's no equivalent to com.holdenkarau.spark.testing.HDFSCluster
in spark-testing-base version 1.X.X, a class heavily relied upon in our tests. (holdenk/spark-testing-base@09b031a)
Considering this, do you think it's worthwhile to invest effort in adapting the tests to accommodate Spark 3.0.0?
Hi, @piffall , @priera , I'd like to report a vulnerable dependency in com.damavis:damavis-spark-core_2.12:0.3.10.
I noticed that com.damavis:damavis-spark-core_2.12:0.3.10 directly depends on org.apache.spark:spark-core_2.12:3.0.0 in the pom. However, as shown in the following dependency graph, org.apache.spark:spark-core_2.12:3.0.0 sufferes from the vulnerability which the C library zstd(version:1.4.4) exposed: CVE-2021-24032.
org.apache.spark:spark-core_2.12:3.2.0 (>=3.2.0) has upgraded this vulnerable C library zstd
to the patch version 1.5.0.
Java build tools cannot report vulnerable C libraries, which may induce potential security issues to many downstream Java projects. Could you please upgrade this vulnerable dependency?
Thanks for your help~
Best regards,
Helen Parr
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.