asg017 / dataflow

An experimental self-hosted Observable notebook editor, with support for FileAttachments, Secrets, custom standard libraries, and more!

Dataflow

A self-hosted Observable notebook editor, with support for FileAttachments, Secrets, custom standard libraries, and more!

test.mp4

Examples

Here are some examples of Observable notebooked created and compiled with Dataflow, along with their original source:

• Wikpedia Pageview (source code)
• GitHub API Notebook (source code)
• Census API (source code)

Documentation

Check out https://alexgarcia.xyz/dataflow for documentation! Fun fact, this site is entirely build with Dataflow 👀

Install

npm i -g @alex.garcia/dataflow

dataflow --help

dataflow run ./my-notebook.ojs

Background

Observable notebooks are reactive, JavaScript-based computational notebooks that run inside your browser. Dataflow is one of the first fully open-sourced and fully featured Observable notebook editors, with key differences that make it easier to integrate with other developer tools!

Edit Notebooks in "Observable JavaScript" .ojs files

Dataflow notebooks are files on your computer, in the form of .ojs files. A single .ojs file is analagous to a single Observable notebook, and .ojs files can import from other .ojs files.

dataflow run my-notebook.ojs will start a dev server at localhost:8080 that shows a live rendered look at a notebook defined in my-notebook.ojs. Any update you make to the my-notebook.ojs file from any text editor will instantly update.

Since notebooks are file-based, they can be easily version controlled (ie git) and appear alongside your other frontend code. Dataflow can also compile these notebooks with dataflow compile to plain JavaScript ES modules, and generate HTML files that run a notebook locally.

Takes advantage of local development

Since Dataflow notebooks aren't ran in a sandboxed iframe, that means you can control every aspect to how a notebook looks. Include CSS and stylesheets to change the look of it, change favicons or document.title, or access browser features not available for iframe like acessing Bluetooth devices or printing notebooks.

Dataflow also offers easy acess to your filesystem with file attachments. Instead manually uploading files, you can simple include a configuration comment in a notebook to the path of your FileAttachment, and Dataflow will be instantly available with the same FileAttachment API as observablehq.com. You can also have "live" file attachments with LiveFileAttachment, which updates everytime a file attachment's file updates, making it even faster to rapidily test different data sources. See documentation for Dataflow File Attachments for more.

Finally, since Dataflow is just another service that runs on localhost:8080, you can build your own APIs and webservices that run local to your computer that Dataflow can access. You can build a proxy to your own database or query from local data apps with a little elbow grease!

Customizable

Dataflow aims to be extensible and customizable. Custom Standard Library make it easier to define new builtin cells for your notebooks, Secrets make it easier to pass in sensitive configuration, and working with "files as notebooks" mean you can bring in whatever text editor you want.

That being said, There's still a lot of room to make Dataflow more customizable! Custom styling, more importing options, and more compiling options are planned, so watch this repo for updates!

License

Dataflow is MIT licensed, and heavily relies on these ISC licensed libraries:

• https://github.com/observablehq/parser
• https://github.com/observablehq/runtime
• https://github.com/observablehq/stdlib

dataflow's Issues

Improper cells margin collapsing

Hello,

overflow: auto CSS declaration for .observablehq selector in dataflow index.html prevents proper cells margin collapsing. So, for example, if you create multiple consecutive md cells, each containing a couple of paragraphs, the gap between these cells will be much bigger than the one between internal paragraphs, and the resulting formatting will feel unnatural.

Removing the above declaration seems to solve the problem. at least for the notebooks I'm experimenting with.

"Visual Dataflow" for a notebook's cell depedencies

Visual Dataflow is a observablehq.com feature that displays a minimap of a notebook's cell dependencies, and show information like a cell's status (pending/fulfilled/rejected), dependencies, and editor status (selected/not selected, pinned/not pinned).

It would be nice if Dataflow offered something similar. Maybe a pull-up option from the bottom bar, that shows a full-width minimap with multiple levels of dependencies (different than observablehq.com's 3 levels maximum).

It would also be cool if this was integrated in the editor sdk (#18 ), so text editor devs could add visual indicators of a cell's status, like a green dot for fulfilled, blue for pending, and red for rejected.

Shebang support

#!/usr/bin/env -S dataflow run -p 3495 --allow-file-attachments
a = 1

b = 2

c = a + b

similar to deno which is

#!/usr/bin/env -S deno run --allow-net --allow-read
let x = "..."

• Ignore shebang on:
  • dataflow run file updates
  • dataflow format
  • importing .ojs files
  • compiling
• Where should the shebang point to?
  • looks like /usr/bin/env -S does magic here

Live-reload displays blank page on Windows 10

Attempted to run dataflow on a Windows 10 computer. Initial load of a notebook works, however changing the notebook using notepad++ causes the page to become and remain blank.

The issue is that the file is empty for the brief moment when dataflow receives a file change notification. I'm not entirely sure why Windows behaves this way but I've seen similar issues in Python modules that watch for file changes. Perhaps opening the file for writing changes the length to 0 and triggers file change notification, then it finally updates after writing completes. It's also odd that dataflow doesn't receive notification when the write completes.

On my computer Notepad++ triggers this issue every time. Atom only trigger this issue when the file becomes large enough. Notepad continued to work, but maybe my test file wasn't large enough to trigger the issue.

I did the simplest possible fix and did added a setTimeout for 100mS in run.js before attempting to read the file. In general some sort of debounce seems necessary, whether that's a fixed delay or more intelligent checks to determine when the file is no longer changing.

Allow auth token for importing private notebooks

After #10, let people define a DATAFLOW_AUTH_TOKENS env var, similar to DENO_AUTH_TOKENS, so people can fetch from private github/npm/custom packages.

custom `.ojs` is not standardized, why not use markdown with extensions instead

• at first, i love your great work.

• but writing .ojs is not so happy.

• the drawbacks of ojs:

  • current ide won't highlight the syntax. writing experience is not happy 🙃️
  • a custom format. put js first. the syntax needs guessing.
  • look at your examples in .ojs. most contents are already in md. so why not put markdown first, instead of js first?

• the benefits of markdown

  • yes, most ide already support syntax highlighting. 😋️
  • markdown extensions are so common that many users are already familiar with.
    • especially the popular mdx.
    • mdx is much easier to both read and write than js.
  • markdown has popular front matter convention for defining meta data like FileAttachment that can provide extra info but won't be rendered

• with the large ecosystem of mdx, we can push this editor even better.

• for example, to solve one problem of observablehq.com, that we cannot hide the cell name.

  • start = new Date("2020-10-01");
    • the start = is often unnecessay.
  • with mdx, u just need to use export const start = new Date("2020-10-01");
    • in mdx, all import and export won't be rendered by default.
    • if u want to use the variable, just use it <div>{myVariable}</div>
  • this is just an example, i haven't considered the implementation details.

• another example, for ui views, may be we can write the popular jsx

// instead of 

viewof pagesSource = Inputs.textarea({
  rows: 6,
  width: 220,
  label: md`<b>Pages"`,
  value: `Donald_Trump
Mike_Pence
Joe_Biden
Kamala_Harris`,
});

// maybe we can write the popular jsx

<Inputs.textarea rows='6' width='220' {...restProps} />

• one more thing, there are so many ui components and other plugins wo can use directly in mdx.

@asg017 do u feel excited 🤗️

• another option is to write a converter to tranfrom mdx ast to ojs, seems doable 🤔️

More import options

Only observablehq and local imports are supported now. Here are other places we should support importing from:

• Any .js file on the internet that has a export default function define() definition
• Any local .js file that has a export default function define() definition
• Github (proxy to raw.githubusercontent.com)
• gists
• NPM (maybe should just document how to use https://www.jsdelivr.com instead of doing npm resolution ourselves

Invalid binding when importing from observable

This is my notebook

import {select} from "https://observablehq.com/@jashkenas/inputs"

select(['a', 'b'])

This is what appears:

Error: invalid binding

`dataflow library`, point to a directory of notebooks

Right now dataflow only runs a single .ojs file, like dataflow run notebook.ojs. this is fairly limiting, if you're working on multiple notebooks at a time, you'd either have to keep killing/restarting dataflow to point to a different file, or run them separately on different ports (which isn't fun at all).

dataflow library (different name?) could instead point to a directory of .ojs files. The dev server would then 1) allow you to view any of the .ojs files in that directory, and 2) offer a UI for navigating through all the notebooks.

This would be similar to jupyterlab, which has a navigation UI for a given directory (and even displays non-notebook files).

• Most of the dev server logic assumes the same notebookPath, so will need to probably rewrite most of it
• Detect when notebooks are added/removed
• What should the navigation UI look like, and how to distinguish it from notebook contents

RuntimeError: Inputs.form is not a function

Thanks for the great work..
I managed to write a simple golang gist code that downloads an observablehq notebook and create "ojs" file. I noted that the current dataflow does not support "Inputs.form"

Other viewof Inputs works well. Is there anything else to be updated/included for the "forms" to work ?

Dataflow-specific builtin cell

A builtin cell named dataflow that have some Dataflow-specific utility functions and constants.

dataflow.theme

A string that updates whenever a user updates the "theme" of a notebook, ex dark -> light or light -> custom theme. #9

dataflow.grid

Object with utilities for the dataflow grid system, #22 .

dataflow.grid.get("myGrid").observeWidth("sidebar");
dataflow.grid.observeWidth("myGrid", "sidebar");

^^ how will this work for compiled notebooks?

Dataflow + Starboard?

Hi Alex,

I'm building Starboard Notebook (some demo noteboks on the frontpage here), a notebook system for the browser. Much like dataflow it also features notebooks which are defined as plaintext files that are easy-to-git (click view source on the top right on the Starboard homepage).

Different than this project however is that notebooks are entirely static: you don't need any server running at all. Your system would embed the notebook in an iframe (and talk to it to for instance inject any secrets).

Starboard is built around dynamic importing of functionality: you can arbitrarily add new "cell types". For instance it supports Python in the browser, and recently I managed to hack together talking to a remote Jupyter kernel.

It would be really cool to support a cell type that is "Observable", I would love to work together on this. What it will provide Dataflow is a runtime that is sandboxed by default: a Starboard notebook runs entirely on a different origin in an iframe (editor included!), as well as a live notebook environment with code editor (as opposed to editing the source text files) - so pretty similar to Observable!

Now imagine interop with Python in the browser.. A workflow could be: "Input CSV file" -> "run some Pandas code in Python through Pyodide" -> "Visualize using Observable cell"... Super powerful stuff!

I can ramble for a long time about why this would be great, but I'd love to hear your thoughts on perhaps adopting Starboard as the notebook "runtime" here. Happy to answer any questions as well of course! (happy to chat somewhere if its easier)

Roadmap to beta version

I intend to "release" a v0.1.0 sometime soon! dataflow already does most of what I want to do, but there still some cleaning up/error handling I want to include before I make an announcement notebook about it. At the same time, I don't wanna spend a long time building everything, v1 will be looong down the road.

Keeping TODOs here for ease/publicity, but please file bugs/problem in a separate issue and I'll merge here!

Notebook editing

• Error notification on parsing error
• "Status bar" at bottom, show websocket connection status
• Reconnect when websocket closes

Compiling

• Support compiling notebooks with local .ojs imports
• Support compiling notebooks with FileAttachments
• Throw error when compiling a notebook that use Secrets
• feature: compiler to directory

For later

• Friendly messaging for syntax errors: e.g. const x =4 , window\n\n width
• Support compiling notebooks with custom stdlibs
• feature: inject styling, other JS into <head> of a index.html
• feature: customizable observable, i.e. "only render this one cell"
• dataflow.startServer(port)
• dataflow.compile(source)

Bundler plugins for webpack, esbuild, rollup, etc.

I think it would be cool to make plugins for common JavaScript build/bundler tools in order to make it easier to embed Dataflow notebooks into other applications. These plugins would compile .ojs files into ES modules (as well as handle FileAttachments the dataflow-way), in ways like:

import {Runtime} from "@observablehq/runtime"
import {Library} from "@alex.garcia/dataflow/core" // not possible right now
import define from "./notebook.ojs"

// could also be react, vue, svelte, etc.
function main() {
  const runtime = new Runtime(new Library());
  const main = runtime.module(define, Inspector.into(document.body));
  // etc.
}

The bundler tool would compile .ojs imports into ES modules, similar to dataflow compile.

• Webpack
• esbuild (have started)
• rollup

Custom stylesheets, fonts, themes

dataflow run notebook.ojs \
  --stylesheet ./path/to/app.css
  --font ./path/to/font.woff

^^ also work with compiling.

Dataflow specific styling, like markdown and the status-footer, should be overwriteable for theming, kinda like streamlit themes

Truly self-host all dependencies

These js/css deps are loaded on every pageload, from a CDN. Should be bundled in and version controlled someway.

• index.html
  • inspector.css
  • bundle in custom css, why not
  • github-markdown-css
  • highlight.js ally-light.css

Dataflow Grid System

Currently, Observable notebooks in Dataflow are rendered top->bottom. Any custom side-by-side cells have to be done in the same cell (kinda like the table-of-content sidebar in the Dataflow docs, but it's tedious and requires re-writing cells.

Option 1: in-line comments

/* @df-grid start myGrid
  template: 
    - sidebar title
    - sidebar content 
    - sidebar content
*/

// @df-grid area sidebar
viewof num = html`<input type=range >`

// @df-grid area sidebar
viewof color = html`<input type=color >`

// @df-grid area sidebar
viewof text = Inputs.text("Chicago")

// @df-grid area title
md`## ${title}`

// @df-grid area content
map = svg`<svg> ${/* ... */}`

// @df-grid end myGrid

• pros
  • comments can easily be ignored, and is picked up by parser
  • easy opt-in/opt-out, especially with unnamed cells
• cons
  • a lot of logic will be needed to parse comments
  • what if the df-grid end comment is missing at the end?
  • dataflow compiler will also need to parse these comments

https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Grid_Layout/Grid_Template_Areas

Option 2: Top-level Configuration Comment

/*
FileAttachments:
  example: ./example.txt
grids:
  grid1:
    template-area:
      - head head
      - nav  main 
      - nav  foot
    cells:
      head: [a, b, c]
      main: [chart]
      sidebar: [viewof color]
*/

• Pros:
  • easier to implement
  • template-area at the top makes it clear, multiple grids is possible
• Cons:
  • logic when editing (e.g. rm'ing 'chart' from 'main') might be tricky to do on the fly
  • not possible with unnamed cells (maybe not too big of a deal)

Overall issues

width

width is the width of the browser (or the container of the notebook), and not the width of a sidebar/"main" section. So would need to offer a reactive variable for width/height for a given grid area. Maybe something like:

sidebarWidth = dataflow.grids.observeWidth("myGrid", "sidebar")

Node API

Currently dataflow is a CLI that you install and use by itself, but I could see how a Node API would be useful. Here's some ideas of what could be exposed (and how users could use it):

• Compiling .ojs code to .js
  • Dataflow specific compiling, not compiling ojs in general (use the compiler for that)
  • For plugins like webpack, esbuild, rollup, etc.
• Start the dataflow run dev server with options
• Export the StandardLibrary to make it install-able (basically with observablehq/htl

More features on the compiler

• Compile to a single .js file
  • Probably can bundle with esbuild api/plugin
  • ignore file attachments? resolve to null? option to bundle as url/base64? need research
• Compile to a .tar.gz file
  • This should be able stream to stdout + filter out files that you want.... right?
• Include ObservableHQ bundles, ex https://api.observablehq.com/@d3/bar-chart.tgz?v=3
• --no-file-attachments option
• --no-html option
• --no-tree-shake option

Better way to define custom builtin cells (stdlib)

window.DATAFLOW_STDLIB works well with run/compile, but it's awkward, constant/dependency is confusing, and you can only be dependent on 1 stdlib cell. This should be done in a newer way.

Maybe:

export function d3() {
  return {
    depend: ["require"],
    define: require => require("d3@6")
  }
}

• I like using ES modules instead of window stuff here
• depend is awkward
• define is probably as best as we will get

Bug in html template literal

Strings interpolated inside an html template literal are not treated as html:

html`<span>${`<b>hello</b>`}</span>`

expected result: hello
actual result: <b>hello</b>

You can verify the expected result in an Observable cell.

My actual use case was to build an html table with rows like:

html`
<table>
${rows.map(({a,b}) => `<tr><td>${a}</td><td>${b}</td></tr>`).join('\n')}
</table>
`

Conflicting GitHub and Observable CSS

First of all, I find dataflow fantastic!

Using It I've noticed that rules from GitHub markdown CSS (being used for all cells) sometimes clashes with the ones from Observable. For example if you create a cell with Inputs.table(), the generated table will have neither the horizontal scroll bar nor the sticky header.

A possible solution to this problem could be to create a specific CSS for dataflow, eventually using GitHub one as a starting point.

Live-Reload imported notebooks

Say you have top.ojs:

import {a} from "./a.ojs"
import {b} from "./b.ojs"

content = md`a=${a}, b=${b}`

a.ojs:

a = 100

b.ojs:

b = 200

If you're running dataflow run top.ojs, then the rendered output will show a=100, b=200 as expected. If top.ojs gets updated, then those changes will be reflected as well.

But, if a.ojs or b.ojs gets updated, then the changes do not immediately get reflected. You would have to refresh the entire page in order to fetch the latest version of the compiled a.ojs or b.ojs file.

This is currently the same case for editor and imports at observablehq.com, but since dataflow works with the filesystem, it should be easier to implement this.

So, when running dataflow run top.ojs, and a.ojs is edited to a = 400, then the live rendered notebook should reload the import and the new 400 value should appear.

Design for Notebooks in Dataflow

Currently, the design for notebooks in Dataflow isn't that great. It uses github-markdown-css for ok-ish styling for the markdown and a lot of weird custom CSS on top to solve random issues.

I want to spend the time to create a better design system for dataflow, one that's great for computational and data heavy notebooks. I don't want it to be too similar to what observablehq.com has, since I feel like that design is a little more lenient for blogging/prose-heavy notebooks (and I'd imagine dataflow would be used more for software engineering/data analyst/development heavy notebooks).

• Markdown styling that doesn't use github-markdown-css
• Table that looks nice for large datasets
• Sensible padding/responsive design for entire container
• Clearer distinction btwn
• fonts!
• CSS variables
• dark mode!

Editor-agnostic auto-completion

Say you have:

element = html`<div>
<table>
...`

update = {
  element;
}

When updating the update cell and accessing element, I want to be able to use autocompletion (ie, <Tab>) to see all th enumerable fields of the element cell's value. In this case, it's an HTML element, so I should see a dropdown list of querySelector, querySelectorAll, getAttribute, etc.

But I use VS Code, some people like vim, others emacs. Autocompletion in each editor would be different, but the same underlying list of enumerable fields should be the same for all potential editor clients.

• In the client (index.html websocket client), send messages back of resolved cells and their enumerable fields
  • Which client? there could be multiple ones, would need the editor to "choose" which client to use (maybe a keyboard shortcut on the index.html to mark "I wanna see this clients enumerable fields"?)
  • Would this work for nested values? like d = {a: {b: {c: 7}}} could have a autosuggested path d.a.b.c, may need to request valid fields from the editor client -> dev server -> index.html client on every cell (seems like fun)
  • Doesn't have to be cells either, window/global values like document or window.location should be supported as well
  • Seems like dealing with VS code will be the hardest part of this

Remove display of result on `viewof`

Cells that have viewof are meant to be view and it would be great to remove display of their result (it makes the notebook much cleaner)

Reload FileAttachments when FA is added/removed

/*
FileAttachments:
  a: ./a.txt
  b: ./b.txt
  c: ./c.txt
*/

files = [FileAttachment("a"), FileAttachment("b"), FileAttachment("c")]

When a.txt gets deleted, the rendered notebook should reflect that (so authors know if they accidentally broke their notebook). When b.txt gets added, the rendered notebook should reflect that (so authors know if they "fixed' their notebook).

But if c.txt updates (ie file metadata change or contents change), then FileAttachment shouldn't refresh, that's the job for LiveFileAttachment.

Add Cell Menus for downloading cell values

For every cell, include an dropdown option in the left margin of a cell that allows readers to download the values of a given cell.

For example, if you have a cell that has a value of some JSON array, then the menu option will let you 1) download that JSON as a file or 2) copy that JSON value to your clipboard.

I'd imagine downloading and copying are the only operations you would care about. Maybe there could be some "caching" operation that lets you transform some data then include it in a notebook "pre-compiled" or "pre-rendered", but that would be out of scope for now.

• Not for compiled notebooks
• Should work for unnamed cells as well
• Download type: Text, JSON, CSV, Blobs

Could also have some builtin cells that create "Download as X" buttons for a given data variable. That was the original workflow for observablehq.com notebook for download cells, before the cell menu was updated.

A better markdown builtin cell

Currently, Dataflow uses htl for the html and svg builtin cells, which offers much more functionality than the current builtin html and svg cells in the standard library.

However, it still uses the default markdown builtin md, which I find a little bit awkward. I'm not too sure how to improve it, but here's a few things I've always found annoying with the md builtin, which would be nice to fix:

// #1 tables

md`|These|columns|are|annoying|to|type
|-|-|-|-|-|-
${data.map(d=>{
  returning a string that gets concatenating into markdown which compiles to html, how weird
  return `|${d.i}|${d.hate}|${d.this}`
} )}`

// #2 no plugins, like checkboxes or anything

md`todo list

- [ ] this don't work
${customCheckbox("you have to build your own custom html component")}
`

Then again, markdown parsing doesn't sound too fun, it would probably be backwards incompatible, and doesnt feel too high of a priority

Secure notebook, import, and fileattachment access

Right now, anything can request from localhost:8080 and get data back, like source code, file attachments, etc.

We should tighten that up, so only the developer can access info.

dataflow run should open a webpage like http://localhost:8080?auth_token=65c87a64c6e5, where that auth token will authorize that user to view notebooks/get file attachments and all.

• Don't connect to websocket unless valid auth token.
• /api/import should also have some form of auth
• /api/file-attachments should also have some auth
• What about multiple clients? Not sure how jupyter handles this...
• Toggle-able, let people expose a dev server if they really want to

Add option for displaying (non-editable) source code below rendered output

dataflow run just executes the notebook, it doesn't display the source code like observablehq does.

A new option, --display-source, can be added to signal to dataflow that source code should be showed alongside the rendered output, so people can learn more/inspect the code if they want to learn more.

For example, this is example.ojs

// example.ojs

// @dataflow show
chart = {
  const canvas = DOM.canvas(width, 640);
  const ctx = canvas.getContext("2d");
  ctx.rect(0, 0, 100, 100);
  return canvas;
}

dataflow run example.ojs --display-source would then have show this in its rendered output:

+----------------------+
|    the canvas element  |
+----------------------+
+----------------------+
|       the source code     |
+----------------------+

instead of:

+----------------------+
|    the canvas element  |
+----------------------+
+----------------------+
|       the source code     |
+----------------------+

Something like a // @dataflow show could be used to "pin" cells to enable this on a cell-by-cell basis.

This would not be available in dataflow compile.