tcr / edit-text

Collaborative rich text editor for the web. Written in Rust + WebAssembly.

edit-text

edit-text is a collaborative text editor with Markdown import and export. Both the server and client are written in Rust, and its frontend is cross-compiled WebAssembly and TypeScript.

✏️ Use edit-text online in the edit.io sandbox.

📚 Read "Getting Started" instructions in the edit-text documentation.

Or clone this repo and run: ./tools frontend-build && ./tools server --open

This is alpha-quality software.

License

Apache-2.0

edit-text's Issues

Have mercutio-sync hot-reload from the dist folder unless otherwise specified

Right now, mercutio-sync compiles all support files into its binary when cargo build is run. Another way to do this would be only in release mode are the support files compiled in, and outside release mode it loads on-demand from the frontend/dist/ directory.

This allows x.rs js-watch to be run in parallel without having to restart the server.

Todo

• Backspace in lists to join bullet groups to match splitting them
• The monkey test that working
• Then fixing the phantom empty skip group thing on multi when you start stop monkey
• Then monkey test that thing
• Then adding random click back in and debugging it, including using mercutio-replay to do so
• Then having random click fix more bugs around multiple people using Mercutio
• Then you have it resilient for launch
• Add pages?
• Add warning about this being demo software
• Write a talk.
• Test more usability fixes when testing out how to give a demo.
• Add export
• Add custom page names
• Add bottom of page warning
• Add import(?)
• Remove carets from terminated clients(?)

Have server generate client IDs

Todo

_{Sent with GitHawk}

Audit use of supports_text()

This is used as a hack to support carets, but shouldn't be used that way (instead, use the actual parents() operators)

Support a flatfile backing store

This would allow you to run edit-text locally (for example) and store .ron files inside of Dropbox or a git repository. Not perfect (will Dropbox automatically try to do conflict resolution on utf-8 text?) but gives an alternative to storing an Sqlite database.

This would require abstracting out the storage component and making sure it works when shared across sync and server threads.

Some notes about fs flushing though: https://news.ycombinator.com/item?id=16573487

Need a new favicon

Probably a monkey?

Support export

Support H4-H6 as well

Since the goal of the editor is to be compatible with Markdown, this means we'll need to support H1-H6 (the editor currently supports H1, H2, and H3).

This should be straightforward, as in just duplicating the code for H1 H2 and H3 that lives in mercutio-client.

Support custom tasks when cargo gets them

https://www.reddit.com/r/rust/comments/8a6lxp/custom_tasks_in_cargo/

http://aturon.github.io/2018/04/05/workflows/

TODO: What is the list of tasks we'd need to install?

Implement transform for outdated operations on client

(TODO)

Enforce consistent naming: Sync, User, and Frontend

(TODO expand this)

Right now "client" is used to mean many different things, and this becomes confusing for commands etc. It would be good to redo this naming so we have the following:

enum SyncToUserCommand { ... }
enum UserToSyncCommand { ... }
enum UserToFrontendCommand { ... }
enum FrontendToUserCommand { ... }

Also the book should be updated with info on this.

Button header should be sticky

Otherwise when you're scrolling a long document, you have to scroll all the way up in order to change any of the settings.

Block buttons should highlight depending on which block type you are currently in

When you are in a Heading 1 block, that box should be highlighted (in bold) and the rest should not be. etc.

Start an editor book.

High level concepts (sync server / client server / wasm proxy) about the system and details of how the operational transform mechanism works are solidified enough by now it's worth writing out a guide to them.

Can the db automatically be set up when starting?

idk if this involves running diesel-cli from command code, or something else.

Add GraphQL endpoint to Sync

Right now the Sync server (mercutio-server/src/sync.rs) and the file server (mercutio-server/src/bin/mercutio-server.rs) both share access to the database, which can result in spurious "database is locked" errors from SQLite. One solution to this problem is to make Sync have a GraphQL endpoint, and have the file server make modifications through this API.

As a result, we can also support a number of other features that fit an out-of-band request/response model:

• Downloading the raw Markdown or raw HTML for a file
• Overwriting a page with new Markdown contents
• Downloading an archive of all data
• Searching all documents for a string (full text search)
• Github or Dropbox login/logout

I played around with Juniper and think it should be a great fit here.

Add SSL to the sandbox

https://github.com/dokku/dokku-letsencrypt

Convert x.rs commands to use shlex + custom macro

TODO

Support selecting spans of text

Initial design: similar to carets, we can add an "anchor" caret at the target of what you're selecting to match the "end" caret that usually appears where you are writing. So both are tracked inline with the document:

This is some text <caret client-id=... />selected from here<anchor-caret client-id=... /> until here.

The selected text would lie between <caret> and <anchor-caret>, and be "selected from here".

• Server would need to be able to evict abandoned anchor carets wherever it evicts regular carets: when a client disconnects, at the end of each operation checking for a missing client.
• The client would need to be able to render a span from an anchor to its selection point.
• The client would need to be able to author operations that operate on the span (delete selection, copy, etc.)

Test Markdown sanitizer thoroughly

Relevant: a development diary about documents and Markdown.

If the Markdown sanitizer can be thoroughly proven to make validatible documents when inputted into edit-text, then we can enable 1) importing Markdown files and 2) possible flatfile storage with Markdown support (like storing directly to Dropbox).

1. Is there an independent Markdown test suite of documents?
2. Can this be fed into a test suite for making sure the sanitizer can always produce a valid Mercutio document as an output?

Support option backspace to delete whole words (macOS)

Similar to how option left and right to move the cursor was implemented in 482a678, it'd be awesome to support option backspace to delete whole words quickly (a holdover until actual caret selections are supported and deleting text is made even simpler.)

Support multiple “pages” on server

Support an -r shortcut for --release in ./x.rs

It's a lot of work to type ./x.rs server --client-proxy --release in one window and ./x.rs client-proxy --release in another!

-r would allow this to shorten to ./x.rs server -cr and ./x.rs client-proxy -r.

Because this has to be handled at the x.rs (not the shelled out binary) level, this needs special logic to detect the short argument.

Use stdweb to manage timers in wasm mode?

This would be a good usecase for using native web APIs directly instead of the editor.ts shim.

Replace tiny_http with a static file server

The way tiny_http is setup right now doesn't do any caching, which makes the .wasm files take a long time to load on client.

Remove jQuery requirement from frontend

jQuery was great for prototyping, but there is so little frontend TypeScript code now that jQuery can be swapped out for lower level DOM commands.

Mostly visible here: https://github.com/tcr/edit-text/blob/master/mercutio-frontend/src/editor.ts

Navigating via arrow keys should update the caret like native code

See https://developer.mozilla.org/en-US/docs/Web/API/Document/caretPositionFromPoint and the polyfill opportunities at the bottom.

Remove carets from terminated clients

After 30s of inactivity (including a lack of a heartbeat ping), clients should be considered "terminated" and then their caret removed from the server state.

caret-51 and the DelGroupAll conversion

TODO, but should this be the appropriate way to solve this test?

Move oatie/src/schema.rs to mercutio/

One of the oldest sins of the codebase is although it abstracted out the logic for how HTML should be transformed, it can't test itself effectively without a schema. There should be some pragmatic way to move oatie/src/schema.rs to mercutio/src/ so that oatie can be an actually independent library for operational transform logic.

This might require refining the Schema/Track traits and making them more internally consistent.

Use wasm-bindgen to generate JS bindings?

We currently do this is in a sloppy way in mercutio-wasm. https://github.com/alexcrichton/wasm-bindgen looks like a much better approach.

Cache results of setup_controls so we don't send redundant controls setup to the client

The ClientImpl::setup_controls method in mercutio-client/src/client.rs will send a new "controls" payload to the client with the following info:

1. Which key combinations the Frontend should listen to
2. Which buttons the Frontend should display
3. Which buttons are highlighted (i.e. representing the result of the latest action).

Because we already know the previous ClientCommand::Controls we sent to the client, it would be helpful if we could

1. Cache the most recently sent ClientCommand::Controls data in pub struct Client, and
2. Not send a new payload if this structure hasn't changed.

There's a few ways to do it, but the trivial way would be to just track the previously sent information and then compare it with the new packet.

One option here is to extract the "keys" and "buttons" fields of ClientCommand::Controls into a "ControlState" struct, so we don't have to match against the ClientCommand enum to compare these values. Then we can change ClientCommand::Controls to ClientCommand::Controls(ControlState) rather than imbedding the data directly.

Combine frontend/ and dist/ folders

Files are spread across both needlessly

Use nom or lalrpop to replace oatie/src/parse.rs

This was a hand-rolled parser but could be better.

No end-to-end test for Mercutio

There's no end-to-end test showing that edit-text can launch successfully and perform live edits in HTML. There should be one that we can gate issues or PRs on, or test locally for radical changes. This is part of a much longer trajectory for how frequently the sandbox should be updated.

The only tests that exist at the moment are for Oatie's transform logic and some Operation APIs.

Refactor the X-Ray feature to be more maintainable

The X-Ray feature is old, from when the frontend was a generic "tree editor" and you could interact with any element directly. Now the text editor is the default, and most of the actions you could perform directly on the X-Ray aren't enabled. But there's a lot of vestigal code here, and even more vestigal Sass.

• Replace in the SASS file the terms "style-mock" => "edit-text" and "style-block" => "xray", since the "mock" and "block" nomenclature is super old
• Remove unused xray styles from Sass code
• Replace mentions of the term "mote" in CSS and JS with "mercutio" (mote being the much earlier prototype for the frontend)

Document the walkers, steppers, placeables, writers, and reversecaretsteppers of the codebase

All of these seem to have some overlap in functionality, and it's confusing when to use which and what crate they live in.

Move WASM off main thread

This solves issues with Chrome, but also might speed up programs.

Support text spans (bold, inline, underline, links, etc.)

What if we styled Characters, in addition to having operations on tree structures made from Characters + Groups? Individual characters would be styled, allowing spans to be implemented easily, but also there would be an easy integration point for having text selections work. This also gets us closer to CRDT algorithms which operate on spans and which might play a future role in the editor.

The other option is to do full-fledged span elements which have specialty merging logic. This is similar to styling characters and possibly more generic, but at the cost of preserving more complexity than maybe it's worth. This is how spans were implemented in earlier attempts at Oatie.

Render characters using text nodes, not individually wrapped spans

Rendering each character wrapped in its own span is an optimization to make the caret click-to-reposition behavior work. Using a feature like https://developer.mozilla.org/en-US/docs/Web/API/DocumentOrShadowRoot/elementFromPoint would enable the frontend to move to text nodes, which is a large optimization over rendering distinct DOM nodes. It's also less data over the wire when running proxy mode.

Support images

At a minimum, edit-text should support block-level images.

It could work similar to how <hr> was implemented. 9cd29f4 Only caveat being, it needs to support inputting a URL at creation time, which stretches the limits of what the client controlling the button array can support.

This might just require the button_handlers input on the TypeScript side to accept a "prompt: bool" argument.

Make sure the monkey runner can run again

Move out schema methods from transform.rs

Monkey fails when editing a document with a unicode character in it.

The 👼 emoji was added to the front page, but a multimonkey client soon failed with an AddWithGroup is malformed error. We should be supporting unicode input and transform out of the box. It's possible the DocString abstraction isn't comprehensive enough, or that the JavaScript frontend has issues counting characters (UCS-2 vs UTF-8).

Rename ./x.rs mercutio-sync to `./x.rs server`

Why isn't mercutio.js not being served with etags?

Something is up with the deployment configuration with the nginx Docker that prevents mercutio.js (the single JS bundle) from being served with e-tags, thus only requiring a 304 response instead of a 2mb bundle download with each page load.

Fix support for Chrome, only currently works on Firefox

See https://github.com/rustwasm/wasm-bindgen/tree/master/examples/hello_world "Caveat for Chrome users" for what the issue is.

#40 is probably the route we'll go with.

Better method definitions for OT and Schema

At a recent talk I presented a simpler API (for purposes of explanation) and noticed that RtfSchema and TrackType could be unified. With some normalizing, this makes the trait API oatie into the following:

/// impl on the Operation
pub trait OT where Self: Sized {
    type Doc;

    fn apply(&Self::Doc, &Self) -> Self::Doc; // should reverse args?
    fn compose(&Self, &Self) -> Self;
    fn transform<S: Schema>(&Self, &Self) -> (Self, Self);
    fn transform_advance<S: Schema>(a: &Self, b: &Self) -> Self;

    fn empty() -> Self;
    fn compose_iter<'a, I>(iter: I) -> Self where I: Iterator<Item=&'a Self>, Self: 'a;
}

/// impl on an enum of types
pub trait Schema: Copy {
    type Attrs = HashMap<String, String>; // overloadable?

    fn allowed_in_root(&self) -> bool;
    fn contains_text(&self) -> bool;
    fn contains_children(&self) -> Vec<Self>;

    fn validate(&Doc) -> Result<(), Error>;
    fn identify(&Self::Attrs) -> Option<Schema>;
}

enum RtfSchema {
    ListItems,      // bullet
    BlockQuotes,    // blockquote
    Blocks,         // h1, h2, h3, h4, h5, h6, p, pre
    BlockObjects,   // hr
    Inlines,        // span
    InlineObjects,  // caret
}

Support ordered lists

This is slightly different than just duplicating the "bullet" tag to be something new, because it actually should change all adjacent tags to be a series of "enumerated" tags. But that would conflict with the transform logic, which would want to preserve intermediary insertions or joining of bullets into being bullet tags.

Changing a single parent tag at once (like a <ul> or <ol>) would be a more correct way to do this. I forget why that wasn't chosen except to avoid the problem of disjoint transforms not supporting being merged. But actually... that's the reason.

Also there has to be successive bullet information preserved anyway, such as when a list starts with a numeral other than 1. So it's either first bullet + all successive bullets being rendered as if they were joined on the client, or 2. supporting <ul> and <ol> and figuring out a disjoint transform fix.

Can trigger infinite loop in WASM mode when clicking around <hr>

I have no consistent repro, but I have run into this a handful of times. When the user triggers a certain sequence, Firefox will spin continuously and it will become difficult to close the tab.