finraos / megasparkdiff Goto Github PK

ML Create Simple DT Model in Test CodeBase

ML Crearte a Visual Representation for the Tree

Are there any plans to release this library cross compiled to Scala 2.12?

Currently, I am getting this error:
NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;

The code runs on a databricks cluster running Scala 2.11, but does not run on Scala 2.12.

This issue is to add functionalities for data validation for DynamoDB.

How would you feel about some day moving the template out as a resource or something? It seems a little cluttered here. suggested by @mmlinford

have the compareAppleTable method return the results in a wrapper class that will further provide more APIs.

So the code should look something like this

result = compareAppleTables(x,y).getLeft().sortByCols(“a”,“b”)
OR
result = compareAppleTables(x,y).getJoinedByCols(“a”,“b”).sortBy(“a”,“b”)

Looks like there's old checkstyle plugin configuration in the POM and some configuration files in the config folder. We should either fix it or remove all of it.

ML Load Existing Model and Extract Tree Text

Why not make this an IllegalArgumentException (or scala equivalent)?
suggested by @mmlinford

  if (maxRecords > 0) {
    maxRecordsCopy = maxRecords;

Move the output formatting code outside of sparkcompare.scala to its own class. Suggesting DiffFormatter.scala to host the code.

Right now there's a scary "1/10 (high risk)" reported by BlackDuck for our project. We should really see what we can do to remedy this. It might not be possible for all dependencies, but in those cases we can at least document why we can't resolve it.

As part of a diff analysis on large datasets we would like to enable users to visualize the actual differences within the DataBricks workbook.

Refactor Java Code into Scala Code
classes under these packages should be refactored into scala

• CmdLine
• SourceVars
• SourceTypes
• VisualResultType
• Launcher
• FileUtil

Hi, I have cloned this project and wanted to learn its capabilities. So, I ran a test case inside the examples folder and got error:

java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x7f63425a) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x7f63425a

Besides that, I was going to try comparing 2 CSV flies in locally and see differences in the outcome report. Could someone, please assist me with the issue?
Thank you in advance!

Maybe sourceType should have a method like supportsSchemaCompare. That way we have the logic in one place, it won't be messy if we add more source types, and we could avoid the nested ifs with just having if (left.sourceType.supportsSchemaCompare() && right.sourceType.supportsSchemaCompare()

ML Visual Diff for a DT

Add a parameter to parallelizeJDBC to enable user to specify partition by column

ML Diff Two Version of the Model Textually

Since mega-spark-diff-parent isn't published in maven, importing as a library is not working.

APIs that utilize spark configuration options are a tad restrictive. We need to move the solution away from overloaded methods and into a new paradigm that allows more cleaner, more configurable API calls to the user.

This can possibly be done by providing a few basic APIs as we do now, but also allowing the user to directly interact with the SparkConf and appending options directly from there.

Currently only JavaDoc is published in MavenCentral, need to look into a solution to merge both scaladoc and java doc if possible