ekzhang / percival

📝 Web-based, reactive Datalog notebooks for data analysis and visualization

Percival

Percival is a declarative data query and visualization language. It provides a reactive, web-based notebook environment for exploring complex datasets, producing interactive graphics, and sharing results.

percival.ink

Percival combines the flexibility of Datalog as a query language for relational data with the beauty of exploratory visualization grammars. These declarative components interact through a reactive dataflow system. Because Percival uses web technologies (including Web Workers for multithreaded, sandboxed execution), fully-interactive notebooks can be shared with anyone on the Internet, making data analyses more tangible to others.

At the core of Percival is a custom Datalog compiler, built with Rust and WebAssembly, which integrates with its notebook runtime. This compiles the query language to JavaScript through a staged evaluation process that also allows users to embed their own JavaScript code. The interface aims to be lightweight, friendly, and accessible, and there is no hidden workspace state.

This is an early-stage research project, and we welcome your feedback, so please feel free to say hello at our discussions page!

Getting Started

If you've gotten to this point in the README, please first try out the web application and demo notebook at percival.ink! The information below is technical documentation intended for contributors.

Building Percival from scratch requires Node v16+, NPM v8+, Rust 1.56+, Cargo, and Wasm-Pack installed on your machine. To build the Rust/WebAssembly portion of the project, use the command:

wasm-pack build --target web crates/percival-wasm

Next, run npm install to install JavaScript dependencies, then run the following command to start the development server:

npm run dev

This should open a Percival notebook in your browser, with live reloading.

Architecture

This section outlines the high-level technical design of Percival.

User Interface

Percival is a client-side web application running fully in the user's browser. The notebook interface is built with Svelte and styled with Tailwind CSS. It relies on numerous other open source libraries, including CodeMirror 6 for live code editing and syntax highlighting, Remark and KaTeX for Markdown rendering, and Vite for frontend bundling.

The code for the web frontend is located in src/, which contains a mix of Svelte (in src/components/) and TypeScript (in src/lib/). These modules are bundled into a static website at build time, and there is no dynamic server-side rendering.

JIT Compiler

Users write code cells in a custom dialect of Datalog, and they are translated to JavaScript by a Rust compiler, which itself is compiled to WebAssembly using wasm-bindgen. The Percival compiler's code is located in the crates/ folder. For ergonomic parsing with human-readable error messages, the compiler relies on chumsky, a parser combinator library.

After the percival-wasm crate is compiled to WebAssembly, it can be used by client-side code. The compiler processes code cells, then sends the resulting JavaScript to separate web workers that sandbox the code and execute it just-in-time. As the user writes queries, their notebook automatically tracks inter-cell dependencies and evaluates cells in topological order, spawning / terminating worker threads on demand.

Data Visualization

Plotting is done using a specialized web worker that runs JavaScript code with access to the Observable Plot library. In order for this library (and D3) to run in a worker context, we patch the global document with a lightweight virtual DOM implementation ported from Domino.

Deployment

In production, the main branch of this repository is continuously deployed to percival.ink via Vercel, which hosts the static website. It also runs a serverless function (see api/index.go) that allows users to share notebooks through the GitHub Gist API.

Development

To build, lint, and format the Svelte project, use the corresponding scripts:

npm run build
npm run check
npm run format

For the Rust crates, you can run unit tests for the core functionality with:

cargo test

You can also run tests for the WebAssembly component using a headless Chrome or Firefox browser:

wasm-pack test --chrome --headless crates/percival-wasm

Since Percival uses a Rust-based compiler but outputs JavaScript, the easiest way to test code generation functionality is within the browser. We use Mocha and Puppeteer for this, and tests can be run with:

npm test

Acknowledgements

Created by Eric Zhang (@ekzhang1). Licensed under the MIT license.

percival's Issues

Better error messages with a lexical grammar

Right now, the Percival compiler simply parses inputs as a character stream with parser combinators. Although this is simple and was sufficient for the purposes of the basic language, some error messages are a little bit suspect, especially ones that mention "expected ... in input." For example, the error below is misleading. It should suggest either an import, rule, or end-of-file.

By switching to a two-phase lexical and syntactic analysis step on a stream of tokens, rather than characters, the parser will be much more flexible, while also having clearer error messages.

`dev` cells throws module import error

I suspect this is something rather obvious to someone more familiar with the JavaScript build tools:

when running npm run dev.

npm run serve allows cell execution (as expected) -- I'm not exactly sure why this throws during dev.

Adjustments for mobile

I would love to run Datalog on my phone. This requires some small tweaks to the current UI:

• Make all elements visible without hovering for mobile devices (or for all devices?). This includes the gutter and buttons.
• Add a "run code" button. Shift + enter doesn't work on mobile; even if it did, a "run" button would be more convenient.

Website https://percival.ink/ doesnt load on Firefox

I see a white page if I open https://percival.ink/ in Firefox 78.15 (From Debian repo). But it works fine on Chrome

Load and save percival files in git repos

I think it would be neat to be able to load, save, edit, and commit percival files directly against a git repo.

Something like isomorphic-git could be useful here:

https://github.com/isomorphic-git/isomorphic-git

It seems there are some potential complications to supporting a git workflow, especially related to CORS.

Thoughts?

JS in Percival / Percival in JS

Percival makes Datalog so much more accessible! Now instead of needing to pick an implementation, install it, read the manual, figure out syntax highlighting in your editor, etc etc etc - we can just visit https://percival.ink. There's a bunch of examples, syntax highlighting, and a quick syntax to fetch data. That's great.

But though you've broken the barrier to access a Datalog implementation, there's more we can do to make Datalog useable.

1. Bring JS into the Percival notebook: We should add JS code blocks that can consume relations and emit new data to the DAG. This will allow learners to use Percival for analysis even if they're Datalog beginners. The user can attempt analysis in Datalog, and bail out to JS if they hit a roadblock. As they gain fluency, users gradually transition to Datalog.

   If we allow cells to return Promise, Percival becomes end-user extendable via ES6 modules. We can now import anything from NPM - not just CSV & JSON data.

   The simplest implementation of this idea would be to parse plot cells of the form output_name = (more, than, one, arg) => ....

2. Bring Datalog rules out of Percival into the wider JS world: publish percival-wasm and/or src/lib/** as experimental packages on NPM. Advanced users can build on top of the work here incentivizes further contribution. The risk of publishing to NPM a backwards compatibility burden. But, given the niche nature of Datalog today, I don't expect this to be a substantial issue if we publish using alpha tags like 0.0.0-alpha001

For both (1) and (2), I'm personally excited to explore integrating Percival <--> ObservableHQ.

• We use Plot already. We could also use @observablehq/inspector and @observablehq/stdlib in plot/JS code blocks.
• Observable's runtime is open source. We could investigate replacing the notebook toposort engine with @observablehq/runtime. This could allow importing Observable notebooks into Percival.
• If we publish runtime.build to NPM, we can import Percival and run datalog within ObservableHQ notebooks. But, we won't have syntax highlighting :(

code cell with syntax error gets erased on toggle

Steps to reproduce:

1. Create a code cell.
2. Write invalid code, such as: 1, 2, 3. The code block is highlighted in orange.
3. Click the down arrow to toggle the visibility of the code block, rendering it not visible.
4. Click the right arrow to toggle visibility of the code block, rendering it visible.
5. The code block is empty, while expected to contain 1, 2, 3.