eth-post / connectors

Various connectors for Metaphor app

Metaphor Connectors

This repository contains a collection of Python-based "connectors" that extract metadata from various sources to ingest into the Metaphor platform.

Installation

This package requires Python 3.7+ installed. You can verify the version on your system by running the following command,

python -V  # or python3 on some systems

Once verified, you can install the package using pip,

pip install "metaphor-connectors[all]"  # or pip3 on some systems

This will install all the connectors and required dependencies. You can also choose to install only a subset of the dependencies by installing the specific extra, e.g.

pip install "metaphor-connectors[snowflake]"

Similarly, you can also install the package using requirements.txt or pyproject.toml.

Connectors

Each connector is placed under its own directory under metaphor and extends the metaphor.common.BaseExtractor class.

Connector Name	Metadata
airflow_plugin	Lineage
bigquery	Schema, description, statistics, queries
bigquery.lineage	Lineage
bigquery.profile	Data profile
dbt	dbt model, test, lineage
dbt.cloud	dbt model, test, lineage
glue	Schema, description
looker	Looker view, explore, dashboard, lineage
manual.data_quality	Data quality
manual.governance	Ownership, tags, description
manual.lineage	Lineage
manual.metadata	Custom metadata
metabase	Dashboard, lineage
postgresql	Schema, description, statistics
postgresql.profile	Data profile
postgresql.usage	Usage
power_bi	Dashboard, lineage
redshift	Schema, description, statistics, queries
redshift.lineage	Lineage
redshift.profile	Data profile
snowflake	Schema, description, statistics, queries
snowflake.lineage	Lineage
snowflake.profile	Data profile
synapse	Schema, queries
tableau	Dashboard, lineage
thought_spot	Dashboard, lineage
unity_catalog	Schema, description

Development

See Development Environment for more instructions on how to setup your local development environment.

Custom Connectors

See Adding a Custom Connector for instructions and a full example on how to create your custom connectors.