dagster-io / devrel-project-demos

Developer-Relations Project Demos

This repository is home to the example projects shared by the Developer Relations team at Dagster. If you have any questions, then feel free to reach out to us on Slack, or through a GitHub Issue / Discussion.

Dagster Deep Dives

Date	Title	Description	Code
2024.04.18	Motherduck, Dagster, Evidence Demo Stack	Explore the developer productivity gains of using tools like MotherDuck, Dagster, and Evidence.	[code]
2024.04.02	Enable Data Mesh	Learn how the Data Mesh paradigm maps to Dagster and enables organizations.	[code]
2024.03.05	Thinking in Partitions	See how pipelines can be parallelized and scale using Dagster partitions.	[code]
2024.02.20	Resources & Configurations	Learn how to a build a platform that promotes reusability and extensibility with configurations and resources.	[code]
2024.02.13	Automations	Explore the concepts of automation on the Dagster platform through cron-based schedules, sensors, and auto-materialization policies.	[code]

Slides

---

See dagster-deep-dives/README.md for more information.

devrel-project-demos's Issues

[motherduck-demo] it's not clear how a user could configure this project to run locally

Description

There are two configuration files that have to be modified to allow this project to run locally (./.env and ./reports/sources/dagster_hybrid_demo/connection.yaml), and this isn't very clear to the user.

Firstly, setting DUCKDB_DATABASE="local.duckdb" as is explained in the .env.example file will result in the dbt assets failing to materialize as I believe dbt assumes this path is relative to its runtime directory, meaning it can't locate the database. Instead, it's mandatory to use the absolute path in the .env file like so: DUCKDB_DATABASE="/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/local.duckdb".

Secondly, the EVIDENCE_SOURCE__DAGSTER_HYBRID_DEMO__FILENAME environment variable isn't referenced in the evidence project files isn't respected by evidence meaning it will fail to find the local database and dagster will fail to materialize the evidence assets. Because of this it's necessary to modify the connection.yaml file in the evidence sources directory, where you unintuitively have to use a relative path, using an absolute paths will result in evidence modifying the path provided at run time which causes the same error.

Proposed Solution

Firstly we can change the instructions in .env.example to properly indicate how to configure the project for local use, using absolute paths in the .env file.

Secondly we can either find a way to configure evidence such that it reads from the EVIDENCE_SOURCE__DAGSTER_HYBRID_DEMO__FILENAME environment variable properly, or give users instructions to modify the connection.yaml file to point to the local duckdb file using a relative path.

Error running motherduck-dagster-hybrid-compute pipeline

I've cloned the repository to wsl2 on my windows machine, I created a .env file containing the following:

# MotherDuck / DuckDb Connection String
#
# Can either be a simple <filename>.duckdb, or a connection string to MotherDuck.
#
# References:
# - https://motherduck.com/docs/authenticating-to-motherduck/#authentication-using-a-service-token
#
DUCKDB_DATABASE="local.duckdb"

# Evidence Connection Variables
#
# References
#  - https://motherduck.com/docs/integrations/evidence/
#  - https://docs.evidence.dev/cli#environment-variables
#
EVIDENCE_SOURCE__DAGSTER_HYBRID_DEMO__TYPE="duckdb"
EVIDENCE_SOURCE__DAGSTER_HYBRID_DEMO__FILENAME="local.duckdb"

I ran make install and then make dev, after which I clicked materialize all in the dagster webpage and get the following error:

Invalid Input Error: Python Object "birds" of type "AssetsDefinition" found on line "/home/maxisq/dev/dagster-project-demos/motherduck-dagster-hybrid-compute/motherduck_dagster_hybrid/assets.py:145" not suitable for replacement scans.
Make sure that "birds" is either a pandas.DataFrame, duckdb.DuckDBPyRelation, pyarrow Table, Dataset, RecordBatchReader, Scanner, or NumPy ndarrays with supported format

Line 145 in assets.py doesn't appear to mention a replacement scan at all and googling "dagster AssetDefinition type not suitable for replacement scan error" seems to show not relevant results.

Can anyone advise on how I can fix this error?

[motherduck-demo] Error when evaluating SSR module ReferenceError: window is not defined

Appearance

"500 | Window is not defined" when starting evidence web server.

Reproduction Steps

Clone repository, cd into motherduck demo, run make evidence-install and make evidence as described in the Makefile to start the evidence web server in developer mode.

Logs

(venv) maxisq@MS-V-Desktop:~/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute$ make evidence
npm --prefix ./reports run dev -- --port 4000

> [email protected] dev
> evidence dev --open / --port 4000

Starting Evidence development workspace - this can take up to 90 seconds

[!] Unable to load source manifest
        This likely means you have no source data, and need to generate it.
        Running npm run sources will generate the needed data. See npm run sources --help for more usage information
        Documentation: https://docs.evidence.dev/core-concepts/data-sources/
Cleared query cache

Forced re-optimization of dependencies

  VITE v5.1.6  ready in 1050 ms

  ➜  Local:   http://localhost:4000/
  ➜  Network: use --host to expose
  ➜  press h + enter to show help
Unable to load manifest, do you need to generate sources?
3:22:06 PM [vite] Error when evaluating SSR module /@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/ui/QueryViewerSupport/QueryDataTable.svelte: failed to import "echarts"
|- ReferenceError: window is not defined
    at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
    at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
    at Module._compile (node:internal/modules/cjs/loader:1368:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
    at Module.load (node:internal/modules/cjs/loader:1205:32)
    at Module._load (node:internal/modules/cjs/loader:1021:12)
    at cjsLoader (node:internal/modules/esm/translators:366:17)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)

3:22:06 PM [vite] Error when evaluating SSR module /@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/ui/QueryViewer.svelte: failed to import "/@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/ui/QueryViewerSupport/QueryDataTable.svelte"
|- ReferenceError: window is not defined
    at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
    at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
    at Module._compile (node:internal/modules/cjs/loader:1368:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
    at Module.load (node:internal/modules/cjs/loader:1205:32)
    at Module._load (node:internal/modules/cjs/loader:1021:12)
    at cjsLoader (node:internal/modules/esm/translators:366:17)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)

3:22:06 PM [vite] Error when evaluating SSR module /@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/ui/index.js: failed to import "/@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/ui/QueryViewer.svelte"
|- ReferenceError: window is not defined
    at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
    at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
    at Module._compile (node:internal/modules/cjs/loader:1368:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
    at Module.load (node:internal/modules/cjs/loader:1205:32)
    at Module._load (node:internal/modules/cjs/loader:1021:12)
    at cjsLoader (node:internal/modules/esm/translators:366:17)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)

3:22:06 PM [vite] Error when evaluating SSR module /@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/index.js: failed to import "/@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/ui/index.js"
|- ReferenceError: window is not defined
    at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
    at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
    at Module._compile (node:internal/modules/cjs/loader:1368:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
    at Module.load (node:internal/modules/cjs/loader:1205:32)
    at Module._load (node:internal/modules/cjs/loader:1021:12)
    at cjsLoader (node:internal/modules/esm/translators:366:17)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)

3:22:06 PM [vite] Error when evaluating SSR module /@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/index.js: failed to import "/@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/unsorted/index.js"
|- ReferenceError: window is not defined
    at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
    at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
    at Module._compile (node:internal/modules/cjs/loader:1368:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
    at Module.load (node:internal/modules/cjs/loader:1205:32)
    at Module._load (node:internal/modules/cjs/loader:1021:12)
    at cjsLoader (node:internal/modules/esm/translators:366:17)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)

3:22:06 PM [vite] Error when evaluating SSR module /src/pages/+layout.svelte: failed to import "/@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/index.js"
|- ReferenceError: window is not defined
    at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
    at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
    at Module._compile (node:internal/modules/cjs/loader:1368:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
    at Module.load (node:internal/modules/cjs/loader:1205:32)
    at Module._load (node:internal/modules/cjs/loader:1021:12)
    at cjsLoader (node:internal/modules/esm/translators:366:17)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)

3:22:06 PM [vite] Error when evaluating SSR module /src/pages/+error.svelte: failed to import "/@fs/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/@evidence-dev/core-components/dist/index.js"
|- ReferenceError: window is not defined
    at detect (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:123:54)
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:97:9
    at /home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:22:68
    at Object.<anonymous> (/home/maxisq/dev/dagster-devrel-fork/motherduck-dagster-hybrid-compute/reports/node_modules/echarts/dist/echarts.js:25:2)
    at Module._compile (node:internal/modules/cjs/loader:1368:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
    at Module.load (node:internal/modules/cjs/loader:1205:32)
    at Module._load (node:internal/modules/cjs/loader:1021:12)
    at cjsLoader (node:internal/modules/esm/translators:366:17)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:315:7)

^Cmake: *** [Makefile:30: evidence] Interrupt