Comments (3)
follow this doc, https://datahubproject.io/docs/next/generated/ingestion/sources/sql-queries/#cli-based-ingestion. datahub ingest successed, but no table lineage or column lineage in the web ui.
eg:
![image](https://private-user-images.githubusercontent.com/20497246/307773381-ca7393e1-b957-41f3-80ad-c2e8fe6f23b6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTEzMjY5MzcsIm5iZiI6MTcxMTMyNjYzNywicGF0aCI6Ii8yMDQ5NzI0Ni8zMDc3NzMzODEtY2E3MzkzZTEtYjk1Ny00MWYzLTgwYWQtYzJlOGZlNmYyM2I2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzI1VDAwMzAzN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTA4MmMyZWM1NmYxMWU3ZjUzNzIwODdiZTI2YTVlYTFjMjczNWZiOTE4Zjg2YTk5YWEyN2JhNmU3MTc4ZTdiMDQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ynERTsqltS-XzHCbU-gJdU7BZk4ohrDWryZo669I7Rk)
![image](https://private-user-images.githubusercontent.com/20497246/307773522-45910f50-9d40-4cc9-a4f9-f7795124b0b0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTEzMjY5MzcsIm5iZiI6MTcxMTMyNjYzNywicGF0aCI6Ii8yMDQ5NzI0Ni8zMDc3NzM1MjItNDU5MTBmNTAtOWQ0MC00Y2M5LWE0ZjktZjc3OTUxMjRiMGIwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzI1VDAwMzAzN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTA4MjdlZTUyNDBiMzM4YmYwODhlM2Y1N2EwNzkyMGEyNWEwZjE3NzhlMjA5NDA3ZTQxNzI4MzI1M2RkYWYwNmMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.0GbHL60SOly5QPdYuaUVwXAHV28QtYxIsgtJjXlZrDQ)
and sql file like this
{"query": "insert into datahub.t_customer_country_city_rank ( ca_country ,ca_city ,cnt ,rid ) select ca_country ,ca_city ,count(1) as cnt ,1 as rid from datahub.t_customer_address group by ca_country,ca_city", "user": "test", "upstream_ tables": ["datahub.t_customer_address"]}
config file like this
datahub_api: # Only necessary if using a non-DataHub sink, e.g. the file sink
server: http://localhost:8080
timeout_sec: 60
source:
type: sql-queries
config:
platform: "mysql"
default_db: "datahub"
query_file: "/tmp/sql-ingest-sample.json"
reporting:
- type: datahub
config:
report_recipe: false
after success run with cmd: datahub ingest -c sql-ingest-recipe.yaml. in the webui do not have the table lineage or column lineage
from datahub.
So you need running web app to use this feature?
Is it possible to generate sql column level lineage with just importing SqlQueriesSource from the module and do it offilne?
from datahub.
The docs are more up to date if you select "next" in the version selector
This source needs access to datahub's backend in order to fetch schema information for tables - otherwise the accuracy will be lower.
There was a bug with our processing of certain insert into
statements, so that query should generate lineage with the latest rc (0.12.1.6rc5) release of acryl-datahub
I'm going to close this issue for now, but feel free to ask any additional questions.
from datahub.
Related Issues (20)
- Add simple_add_dataset_ownership for Tableau charts HOT 2
- [Snowflake ingestion] Handling Variant Column Fields HOT 2
- SiblingAssociationHook causes NullPointerException in ESSearchDAO when using standalone MAE consumer HOT 2
- Ingestion for S3 "Got ValueError: Trailing data" HOT 2
- MSSQL view lineage not shown HOT 2
- Glossary term junk links in related entities for Tableau workbook HOT 2
- Admin user "datahub" has no admin rights in demo HOT 2
- Pip packages are not installed for Iceberg source plugin with Hive type HOT 4
- GlobalTags Aspect API returns HTTP Code 204 instead of 200 HOT 2
- Using up and down arrows while editing description also changes selected field and overwrites it HOT 2
- A short description of the bug HOT 5
- MongoDB source unable to parse datetimes with years > 9999
- GraphQL mutation createDomain doesn't overwrite existing domains HOT 2
- "View in Looker" button for Explore does not pick the `external_base_url` from config
- Custom Properties are deleted when a dataset is re-ingested HOT 2
- Getting Failed to log in! An unexpected error occurred when logging in - where do I go to view these error logs? HOT 1
- DomainEntitiesResolver class does not support `values` FacetFilterInput parameter HOT 3
- Cannot assign Admin role for user authen via Keycloak HOT 2
- Hive + databricks ingestion HOT 1
- Keep tags per dataset schema version but not remove them in the previous versions HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datahub.