Comments (12)
Hi, share the error, what jar are you uploading?
from spline-spark-agent.
HI
i have attached the screen shot of the installation of the jar
the jar is the artifact of build the project
https://github.com/AbsaOSS/spline-spark-agent
the errors are
command-4378807525288898:2: error: not found: value za
import za.co.absa.spline.harvester.SparkLineageInitializer._
^
command-4378807525288898:3: error: not found: value za
import za.co.absa.spline.agent.AgentConfig
^
command-4378807525288898:4: error: not found: value za
import za.co.absa.spline.harvester.postprocessing.AbstractPostProcessingFilter
^
command-4378807525288898:5: error: not found: value za
import za.co.absa.spline.harvester.postprocessing.PostProcessingFilter
^
command-4378807525288898:6: error: object configuration is not a member of package org.apache.commons
import org.apache.commons.configuration.Configuration
^
command-4378807525288898:7: error: not found: value za
import za.co.absa.spline.harvester.conf.StandardSplineConfigurationStack
^
command-4378807525288898:8: error: not found: value za
import za.co.absa.spline.harvester.HarvestingContext
^
command-4378807525288898:9: error: not found: value za
import za.co.absa.spline.producer.model.ExecutionPlan
^
command-4378807525288898:10: error: not found: value za
import za.co.absa.spline.producer.model.ExecutionEvent
^
command-4378807525288898:11: error: not found: value za
import za.co.absa.spline.producer.model.ReadOperation
^
command-4378807525288898:12: error: not found: value za
import za.co.absa.spline.producer.model.WriteOperation
^
command-4378807525288898:13: error: not found: value za
import za.co.absa.spline.producer.model.DataOperation
^
command-4378807525288898:14: error: not found: value za
import za.co.absa.spline.harvester.ExtraMetadataImplicits._
^
command-4378807525288898:15: error: not found: value za
import za.co.absa.spline.harvester.SparkLineageInitializer._
^
from spline-spark-agent.
If you are modifying the code, you must build and use your own spark-3.3-spline-agent-bundle.jar
not the one from maven.
from spline-spark-agent.
error: not found: value za
It looks like the Jar is either not seen by the Spark, or is empty/does not contain any Spline related classes. Please check your Jar content and the way you attach it to Spark.
from spline-spark-agent.
the jar is 5K - it was created when i made a build on the Intelij
the file is called spline-spark-agent_2.12.jar
i took the content of the POM.xml from Bundle 3.3
I then uploaded the jar as "Install New" on the Databricks Cluster
Do you have a working solution of spline for Databricks which is not from Maven Central ?
from spline-spark-agent.
Please follow the building procedure from this section in the README - https://github.com/AbsaOSS/spline-spark-agent#building
Then take the bundle jar from the bundle-3.3/target
folder.
from spline-spark-agent.
from spline-spark-agent.
I've never seen this error. Something might be wrong with your POM files. Try it on a clean project. call the Maven command in the project root directory.
Here's how it should look like:
~/Projects/spline-spark-agent │ develop
❯ mvn scala-cross-build:change-version -Pscala-2.11
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Spline Spark Agent [pom]
[INFO] agent-commons_2.12 [jar]
[INFO] agent-core_2.12 [jar]
[INFO] examples_2.12 [jar]
[INFO] integration-tests_2.12 [jar]
[INFO] spark-2.2-spline-agent-bundle_2.11 [jar]
[INFO] spark-2.3-spline-agent-bundle_2.11 [jar]
[INFO] spark-2.4-spline-agent-bundle_2.12 [jar]
[INFO]
[INFO] -------< za.co.absa.spline.agent.spark:spline-spark-agent_2.12 >--------
[INFO] Building Spline Spark Agent 2.1.0-SNAPSHOT [1/8]
[INFO] from pom.xml
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ spline-spark-agent_2.12 ---
[INFO]
[INFO] ----------< za.co.absa.spline.agent.spark:agent-commons_2.12 >----------
[INFO] Building agent-commons_2.12 2.1.0-SNAPSHOT [2/8]
[INFO] from commons/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ agent-commons_2.12 ---
[INFO]
[INFO] -----------< za.co.absa.spline.agent.spark:agent-core_2.12 >------------
[INFO] Building agent-core_2.12 2.1.0-SNAPSHOT [3/8]
[INFO] from core/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ agent-core_2.12 ---
[INFO]
[INFO] ------------< za.co.absa.spline.agent.spark:examples_2.12 >-------------
[INFO] Building examples_2.12 2.1.0-SNAPSHOT [4/8]
[INFO] from examples/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ examples_2.12 ---
[INFO]
[INFO] --------< za.co.absa.spline.agent.spark:integration-tests_2.12 >--------
[INFO] Building integration-tests_2.12 2.1.0-SNAPSHOT [5/8]
[INFO] from integration-tests/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ integration-tests_2.12 ---
[INFO]
[INFO] --< za.co.absa.spline.agent.spark:spark-2.2-spline-agent-bundle_2.11 >--
[INFO] Building spark-2.2-spline-agent-bundle_2.11 2.1.0-SNAPSHOT [6/8]
[INFO] from bundle-2.2/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ spark-2.2-spline-agent-bundle_2.11 ---
[INFO]
[INFO] --< za.co.absa.spline.agent.spark:spark-2.3-spline-agent-bundle_2.11 >--
[INFO] Building spark-2.3-spline-agent-bundle_2.11 2.1.0-SNAPSHOT [7/8]
[INFO] from bundle-2.3/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ spark-2.3-spline-agent-bundle_2.11 ---
[INFO]
[INFO] --< za.co.absa.spline.agent.spark:spark-2.4-spline-agent-bundle_2.12 >--
[INFO] Building spark-2.4-spline-agent-bundle_2.12 2.1.0-SNAPSHOT [8/8]
[INFO] from bundle-2.4/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- scala-cross-build:0.2.2:change-version (default-cli) @ spark-2.4-spline-agent-bundle_2.12 ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spline Spark Agent 2.1.0-SNAPSHOT:
[INFO]
[INFO] Spline Spark Agent ................................. SUCCESS [ 0.078 s]
[INFO] agent-commons_2.12 ................................. SUCCESS [ 0.009 s]
[INFO] agent-core_2.12 .................................... SUCCESS [ 0.009 s]
[INFO] examples_2.12 ...................................... SUCCESS [ 0.009 s]
[INFO] integration-tests_2.12 ............................. SUCCESS [ 0.009 s]
[INFO] spark-2.2-spline-agent-bundle_2.11 ................. SUCCESS [ 0.007 s]
[INFO] spark-2.3-spline-agent-bundle_2.11 ................. SUCCESS [ 0.008 s]
[INFO] spark-2.4-spline-agent-bundle_2.12 ................. SUCCESS [ 0.009 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.279 s
[INFO] Finished at: 2023-11-06T10:58:25+01:00
[INFO] ------------------------------------------------------------------------
from spline-spark-agent.
from spline-spark-agent.
There seems to be a compilation error. Are you compiling the original source code, or you changed something in it?
from spline-spark-agent.
the origanl code
from spline-spark-agent.
Sorry I cannot reproduce it. It builds fine on all our environments including CI and personal PCs. Please try to build it on another env. It must work.
from spline-spark-agent.
Related Issues (20)
- Datahub integration reference link HOT 1
- Getting error while connecting tryng to use Kafka dispatcher HOT 1
- Initialization failure handling control HOT 2
- Cant track org.apache.hadoop.fs.rename
- Spline Agent required for spark 3.4 version and Scala 2.12 HOT 2
- Facing issues while sending Databricks Lineage (Spline) to kafka HOT 1
- Support for spark 3.5 HOT 4
- Does spline solution support snowpark
- Support of setting the arangoDB name on the configuration HOT 7
- Need to deploy on Synapse environment in order to track lineage for Synapse notebooks HOT 1
- Unity Catalog Support to get the Notebook details
- Support for AWS Glue DynamicFrame HOT 2
- Error when running spark application with agent-core as queryExecutionListeners in java 17
- Spline Support of Statment Create table as select HOT 11
- Using Spline in Spark Scala project HOT 1
- Can Spline support lineage for AWS glue Spark dynamic frames? HOT 1
- Spline doesn't track some in memory operations in pyspark HOT 5
- Customized Listener HOT 1
- Databricks - Support of Delta live tables
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spline-spark-agent.