Comments (4)
Correct. Spline 0.4 has a different data model. Now the agent generates two documents for every Spark persistent action - the execution plan and the execution event. The former represents the static part of the lineage (logical plan and the data sources) while the later represents the fact that something has happened. Since Spark appId
is different from run to run it (along with the timestamp) logically belongs to the execution event, while the appName
(being a user defined property that is usually a constant for a given app) belongs to the exec plan.
from spline-spark-agent.
It's in the ExecutionEvent.extra
property
https://github.com/AbsaOSS/spline-spark-agent/blob/develop/core/src/main/scala/za/co/absa/spline/harvester/ModelConstants.scala
from spline-spark-agent.
Thanks for the quick answer, @wajda.
I'll take a further look on what happened here. I recreated the HDFS dispatcher from Spline 0.3, and noticed on my tests that the AppID was not present on the output file.
You can see here the sample file I created.
{
"id": "1dcad612-ee27-49f2-be0a-5c151589e06b",
"operations": {
"write": {
"outputSource": "file:/df_select_two.parquet",
"append": false,
"id": 0,
"childIds": [
1
],
"params": {
"path": "/df_select_two.parquet"
},
"extra": {
"name": "InsertIntoHadoopFsRelationCommand",
"destinationType": "Parquet"
}
},
"reads": [
{
"inputSources": [
"file:/address.csv"
],
"id": 9,
"schema": [
"0695065c-55e3-4ea9-9fb3-e7e1516384a2",
"919ac273-a4c6-400c-b0a7-d952316ca18d",
"33541cb8-ecb0-4c26-a80a-c7c3739f283d",
"455d1de0-1bbb-44b6-a1f2-57dcb370365c"
],
"params": {
"inferschema": "true",
"header": "true"
},
"extra": {
"name": "LogicalRelation",
"sourceType": "CSV"
}
},
{
"inputSources": [
"file:/user.csv"
],
"id": 7,
"schema": [
"a00923a7-0794-4783-937f-17e4b71f0b8f",
"d6a16779-ea38-4326-829f-69e35c884020",
"c72d4aef-c872-408f-a948-2bb4785d4721",
"e370c1b8-2346-404c-a51a-39c3c2c6c7f1"
],
"params": {
"inferschema": "true",
"header": "true"
},
"extra": {
"name": "LogicalRelation",
"sourceType": "CSV"
}
}
],
"other": [
{
"id": 8,
"childIds": [
9
],
"params": {
"keys": [
{
"_typeHint": "expr.AttrRef",
"refId": "0695065c-55e3-4ea9-9fb3-e7e1516384a2"
},
{
"_typeHint": "expr.AttrRef",
"refId": "919ac273-a4c6-400c-b0a7-d952316ca18d"
},
{
"_typeHint": "expr.AttrRef",
"refId": "33541cb8-ecb0-4c26-a80a-c7c3739f283d"
},
{
"_typeHint": "expr.AttrRef",
"refId": "455d1de0-1bbb-44b6-a1f2-57dcb370365c"
}
]
},
"extra": {
"name": "Deduplicate"
}
},
{
"id": 6,
"childIds": [
7,
8
],
"schema": [
"a00923a7-0794-4783-937f-17e4b71f0b8f",
"d6a16779-ea38-4326-829f-69e35c884020",
"c72d4aef-c872-408f-a948-2bb4785d4721",
"e370c1b8-2346-404c-a51a-39c3c2c6c7f1",
"0695065c-55e3-4ea9-9fb3-e7e1516384a2",
"919ac273-a4c6-400c-b0a7-d952316ca18d",
"33541cb8-ecb0-4c26-a80a-c7c3739f283d",
"455d1de0-1bbb-44b6-a1f2-57dcb370365c"
],
"params": {
"joinType": "INNER",
"condition": {
"_typeHint": "expr.Binary",
"symbol": "=",
"dataTypeId": "03a56209-c6e8-44ba-b2ff-dc58bb997961",
"children": [
{
"_typeHint": "expr.AttrRef",
"refId": "a00923a7-0794-4783-937f-17e4b71f0b8f"
},
{
"_typeHint": "expr.AttrRef",
"refId": "0695065c-55e3-4ea9-9fb3-e7e1516384a2"
}
]
}
},
"extra": {
"name": "Join"
}
},
{
"id": 5,
"childIds": [
6
],
"schema": [
"a00923a7-0794-4783-937f-17e4b71f0b8f",
"d6a16779-ea38-4326-829f-69e35c884020",
"c72d4aef-c872-408f-a948-2bb4785d4721",
"e370c1b8-2346-404c-a51a-39c3c2c6c7f1",
"919ac273-a4c6-400c-b0a7-d952316ca18d",
"33541cb8-ecb0-4c26-a80a-c7c3739f283d",
"455d1de0-1bbb-44b6-a1f2-57dcb370365c"
],
"params": {
"projectList": [
{
"_typeHint": "expr.AttrRef",
"refId": "a00923a7-0794-4783-937f-17e4b71f0b8f"
},
{
"_typeHint": "expr.AttrRef",
"refId": "d6a16779-ea38-4326-829f-69e35c884020"
},
{
"_typeHint": "expr.AttrRef",
"refId": "c72d4aef-c872-408f-a948-2bb4785d4721"
},
{
"_typeHint": "expr.AttrRef",
"refId": "e370c1b8-2346-404c-a51a-39c3c2c6c7f1"
},
{
"_typeHint": "expr.AttrRef",
"refId": "919ac273-a4c6-400c-b0a7-d952316ca18d"
},
{
"_typeHint": "expr.AttrRef",
"refId": "33541cb8-ecb0-4c26-a80a-c7c3739f283d"
},
{
"_typeHint": "expr.AttrRef",
"refId": "455d1de0-1bbb-44b6-a1f2-57dcb370365c"
}
]
},
"extra": {
"name": "Project"
}
},
{
"id": 4,
"childIds": [
5
],
"params": {
"condition": {
"_typeHint": "expr.Binary",
"symbol": "=",
"dataTypeId": "03a56209-c6e8-44ba-b2ff-dc58bb997961",
"children": [
{
"_typeHint": "expr.AttrRef",
"refId": "455d1de0-1bbb-44b6-a1f2-57dcb370365c"
},
{
"_typeHint": "expr.Literal",
"value": "Madrid",
"dataTypeId": "2b145c88-9691-4104-afb9-63a54c98ff28"
}
]
}
},
"extra": {
"name": "Filter"
}
},
{
"id": 3,
"childIds": [
4
],
"params": {
"condition": {
"_typeHint": "expr.Binary",
"symbol": "=",
"dataTypeId": "03a56209-c6e8-44ba-b2ff-dc58bb997961",
"children": [
{
"_typeHint": "expr.AttrRef",
"refId": "e370c1b8-2346-404c-a51a-39c3c2c6c7f1"
},
{
"_typeHint": "expr.Literal",
"value": 5,
"dataTypeId": "766b28ff-e3ab-4ec1-a543-57e7dd473f30"
}
]
}
},
"extra": {
"name": "Filter"
}
},
{
"id": 2,
"childIds": [
3
],
"schema": [
"a00923a7-0794-4783-937f-17e4b71f0b8f",
"19a3777c-8899-47d2-8029-b07720cea2ba",
"22ffd530-d149-4eb3-b9d7-31e63d0fa110",
"919ac273-a4c6-400c-b0a7-d952316ca18d",
"455d1de0-1bbb-44b6-a1f2-57dcb370365c"
],
"params": {
"projectList": [
{
"_typeHint": "expr.AttrRef",
"refId": "a00923a7-0794-4783-937f-17e4b71f0b8f"
},
{
"_typeHint": "expr.Alias",
"alias": "user_name",
"child": {
"_typeHint": "expr.AttrRef",
"refId": "d6a16779-ea38-4326-829f-69e35c884020"
}
},
{
"_typeHint": "expr.Alias",
"alias": "user_age",
"child": {
"_typeHint": "expr.AttrRef",
"refId": "c72d4aef-c872-408f-a948-2bb4785d4721"
}
},
{
"_typeHint": "expr.AttrRef",
"refId": "919ac273-a4c6-400c-b0a7-d952316ca18d"
},
{
"_typeHint": "expr.AttrRef",
"refId": "455d1de0-1bbb-44b6-a1f2-57dcb370365c"
}
]
},
"extra": {
"name": "Project"
}
},
{
"id": 1,
"childIds": [
2
],
"schema": [
"a00923a7-0794-4783-937f-17e4b71f0b8f",
"19a3777c-8899-47d2-8029-b07720cea2ba",
"22ffd530-d149-4eb3-b9d7-31e63d0fa110",
"919ac273-a4c6-400c-b0a7-d952316ca18d",
"455d1de0-1bbb-44b6-a1f2-57dcb370365c",
"3754c05e-0265-469e-9d24-f440759f3ef1"
],
"params": {
"projectList": [
{
"_typeHint": "expr.AttrRef",
"refId": "a00923a7-0794-4783-937f-17e4b71f0b8f"
},
{
"_typeHint": "expr.AttrRef",
"refId": "19a3777c-8899-47d2-8029-b07720cea2ba"
},
{
"_typeHint": "expr.AttrRef",
"refId": "22ffd530-d149-4eb3-b9d7-31e63d0fa110"
},
{
"_typeHint": "expr.AttrRef",
"refId": "919ac273-a4c6-400c-b0a7-d952316ca18d"
},
{
"_typeHint": "expr.AttrRef",
"refId": "455d1de0-1bbb-44b6-a1f2-57dcb370365c"
},
{
"_typeHint": "expr.Alias",
"alias": "two_times_age",
"child": {
"_typeHint": "expr.Binary",
"symbol": "*",
"dataTypeId": "bc4a4a4b-a2c3-4615-956e-e5d39f8819e1",
"children": [
{
"_typeHint": "expr.AttrRef",
"refId": "22ffd530-d149-4eb3-b9d7-31e63d0fa110"
},
{
"_typeHint": "expr.Literal",
"value": 2,
"dataTypeId": "766b28ff-e3ab-4ec1-a543-57e7dd473f30"
}
]
}
}
]
},
"extra": {
"name": "Project"
}
}
]
},
"systemInfo": {
"name": "spark",
"version": "2.3.2"
},
"agentInfo": {
"name": "spline",
"version": "0.5.0-SNAPSHOT"
},
"extraInfo": {
"appName": "lineageFileAssessment",
"dataTypes": [
{
"_typeHint": "dt.Simple",
"id": "bc4a4a4b-a2c3-4615-956e-e5d39f8819e1",
"name": "integer",
"nullable": true
},
{
"_typeHint": "dt.Simple",
"id": "68883ddc-1f4b-42a6-a3ac-078e752fd82e",
"name": "string",
"nullable": true
},
{
"_typeHint": "dt.Simple",
"id": "03a56209-c6e8-44ba-b2ff-dc58bb997961",
"name": "boolean",
"nullable": true
},
{
"_typeHint": "dt.Simple",
"id": "2b145c88-9691-4104-afb9-63a54c98ff28",
"name": "string",
"nullable": false
},
{
"_typeHint": "dt.Simple",
"id": "766b28ff-e3ab-4ec1-a543-57e7dd473f30",
"name": "integer",
"nullable": false
}
],
"attributes": [
{
"id": "0695065c-55e3-4ea9-9fb3-e7e1516384a2",
"name": "id",
"dataTypeId": "bc4a4a4b-a2c3-4615-956e-e5d39f8819e1"
},
{
"id": "919ac273-a4c6-400c-b0a7-d952316ca18d",
"name": "street",
"dataTypeId": "68883ddc-1f4b-42a6-a3ac-078e752fd82e"
},
{
"id": "33541cb8-ecb0-4c26-a80a-c7c3739f283d",
"name": "number",
"dataTypeId": "bc4a4a4b-a2c3-4615-956e-e5d39f8819e1"
},
{
"id": "455d1de0-1bbb-44b6-a1f2-57dcb370365c",
"name": "city",
"dataTypeId": "68883ddc-1f4b-42a6-a3ac-078e752fd82e"
},
{
"id": "a00923a7-0794-4783-937f-17e4b71f0b8f",
"name": "id",
"dataTypeId": "bc4a4a4b-a2c3-4615-956e-e5d39f8819e1"
},
{
"id": "d6a16779-ea38-4326-829f-69e35c884020",
"name": "name",
"dataTypeId": "68883ddc-1f4b-42a6-a3ac-078e752fd82e"
},
{
"id": "c72d4aef-c872-408f-a948-2bb4785d4721",
"name": "age",
"dataTypeId": "bc4a4a4b-a2c3-4615-956e-e5d39f8819e1"
},
{
"id": "e370c1b8-2346-404c-a51a-39c3c2c6c7f1",
"name": "qtd_kids",
"dataTypeId": "bc4a4a4b-a2c3-4615-956e-e5d39f8819e1"
},
{
"id": "19a3777c-8899-47d2-8029-b07720cea2ba",
"name": "user_name",
"dataTypeId": "68883ddc-1f4b-42a6-a3ac-078e752fd82e"
},
{
"id": "22ffd530-d149-4eb3-b9d7-31e63d0fa110",
"name": "user_age",
"dataTypeId": "bc4a4a4b-a2c3-4615-956e-e5d39f8819e1"
},
{
"id": "3754c05e-0265-469e-9d24-f440759f3ef1",
"name": "two_times_age",
"dataTypeId": "bc4a4a4b-a2c3-4615-956e-e5d39f8819e1"
}
]
}
}
from spline-spark-agent.
Thanks for the explanation @wajda, I will adjust the HDFS dispatcher accordingly.
from spline-spark-agent.
Related Issues (20)
- Spline agent affecting databricks driver performance HOT 7
- Datahub integration reference link HOT 1
- Getting error while connecting tryng to use Kafka dispatcher HOT 1
- Initialization failure handling control HOT 2
- Cant track org.apache.hadoop.fs.rename
- use splineAgent as a Jar file HOT 12
- Spline Agent required for spark 3.4 version and Scala 2.12 HOT 2
- Facing issues while sending Databricks Lineage (Spline) to kafka HOT 1
- Support for spark 3.5 HOT 4
- Does spline solution support snowpark
- Support of setting the arangoDB name on the configuration HOT 7
- Need to deploy on Synapse environment in order to track lineage for Synapse notebooks HOT 1
- Unity Catalog Support to get the Notebook details
- Support for AWS Glue DynamicFrame HOT 2
- Error when running spark application with agent-core as queryExecutionListeners in java 17
- Spline Support of Statment Create table as select HOT 11
- Using Spline in Spark Scala project HOT 1
- Can Spline support lineage for AWS glue Spark dynamic frames? HOT 1
- Spline doesn't track some in memory operations in pyspark HOT 5
- Customized Listener HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spline-spark-agent.