y-crdt / ypy Goto Github PK

View Code? Open in Web Editor NEW

163.0 163.0 21.0 332 KB

Python bindings to y-crdt

Home Page: https://ypy.readthedocs.io/en/latest/

License: Other

Rust 64.23% Python 35.77%

crdt

ypy's Introduction

Y CRDT

A collection of Rust libraries oriented around implementing Yjs algorithm and protocol with cross-language and cross-platform support in mind. It aims to maintain behavior and binary protocol compatibility with Yjs, therefore projects using Yjs/Yrs should be able to interoperate with each other.

Project organization:

lib0 is a serialization library used for efficient (and fairly fast) data exchange.
yrs (read: wires) is a core Rust library, a foundation stone for other projects.
yffi (read: wifi) is a wrapper around yrs used to provide a native C foreign function interface. See also: C header file.
ywasm is a wrapper around yrs that targets WebAssembly and JavaScript API.

Other projects using yrs:

ypy - Python bindings.
yrb - Ruby bindings.

Feature parity among projects

	yjs (13.6)	yrs (0.18)	ywasm (0.18)	yffi (0.18)	y-rb (0.5)	y-py (0.6)	ydotnet (0.4)	yswift (0.2)
YText: insert/delete	✅	✅	✅	✅	✅	✅	✅	✅
YText: formatting attributes and deltas	✅	✅	✅	✅	✅	✅	✅	✅
YText: embeded elements	✅	✅	✅	✅	✅	✅	✅	✅
YMap: update/delete	✅	✅	✅	✅	✅	✅	✅	✅
YMap: weak links	✅ (weak-links branch)	✅	✅	✅	❌	❌	❌	❌
YArray: insert/delete	✅	✅	✅	✅	✅	✅	✅	✅
YArray & YText quotations	✅ (weak links branch)	✅	✅	✅	❌	❌	❌	❌
YArray: move	✅ (move branch)	✅	✅	✅	❌	✅	✅	❌
XML Element, Fragment and Text	✅	✅	✅	✅	✅	✅	✅	❌
Sub-documents	✅	✅	✅	✅	❌	❌	✅	❌
Shared collections: observers	✅	✅	✅ (incompatible with yjs)	✅	✅	✅	✅	✅
Shared collections: recursive nesting	✅	✅	✅	✅	❌	❌	❌	❌
Document observers	✅	✅	✅ (incompatible with yjs)	✅	✅	✅	✅
Transaction: origins	✅	✅	✅	✅	❌	❌	✅	✅
Snapshots	✅	✅	✅	✅	❌	❌	❌	❌
Sticky indexes	✅	✅	✅	✅	❌	❌	✅	❌
Undo Manager	✅	✅	✅	✅	❌	❌	✅	✅
Awareness	✅	✅	❌	❌	✅	❌	✅	❌
Network provider: WebSockets	✅ (y-websocket)	✅ (yrs-warp)	❌	❌	✅ (y-rb_actioncable)	✅ (ypy-websocket)	✅	❌
Network provider: WebRTC	✅ (y-webrtc)	✅ (yrs-webrtc)	❌	❌	❌	❌	❌	❌

Maintainers

ypy's People

Contributors

Stargazers

Watchers

ypy's Issues

Throw more expressive error when already integrated type is inserted

The issue described here is definitely in the top 10 of common mistakes when working with Y*.

It would make sense to throw an error like "YType has already been integrated and cannot be inserted at a different place" instead of "Type cannot be integrated".

We should do that whenever we try to insert a type that is not prelim.

No wheels for python 3.11

Python 3.11 is out and already available on conda-forge, could we release y-py with wheels for it?

Add CHANGELOG.md

When updating to newer versions of ypy it's often useful to know about the recent changes.

Some GitHub Releases have a "What's Changed" section which is very useful, but some don't: https://github.com/y-crdt/ypy/releases

Having a CHANGELOG.md file in the repo would make it convenient to have all the changes logged in one place.

Double borrow in `observe_after_transaction`

Key YDoc methods are inaccessible inside observe_after_transaction callbacks due to a double borrow. For example

def test_borrow_issue():
    d = Y.YDoc()
    m = d.get_map("foo")
    update: bytes = None

    def put_updates(ydoc: Y.YDoc, event) -> None:
        nonlocal update
        update = Y.encode_state_as_update(ydoc, event.before_state) #borrowed again here FAILS

    d.observe_after_transaction(lambda e: put_updates(d, e)) # Borrowed here

    with d.begin_transaction() as txn:
        m.set(txn, "hi", "there")

YBytes?

What would be the best way to handle a binary document?
A YText would not work, and a YArray seems overkill, as it can handle elements of arbitrary type.
Would it make sense to create a YBytes type?

cc @Horusiath @Waidhoferj @dmonad

Making a (pre)release with the newly added move feature

Opening an issue to track this. Would it be possible to make a release of ypy (or at least a pre-release) including the move feature that was recently added.

That would allow @hbcarlos @davidbrochart to update the relevant PRs in JupyterLab and other downstream projects to test that it works as expected.

cc @dmonad @Waidhoferj @Horusiath.

Support ymap.update(d) ?

It would be nice to support ymap.update(d), where d is a dictionary. It would be equivalent to:

with ydoc.begin_transaction() as t:
    for k, v in d.items():
        ymap.set(t, k, v)

Turn rust errors into exceptions

There are bugs we cannot report because the underlying error is not reported. E.G:

import y_py as Y

d1 = Y.YDoc()
root = d1.get_map('stuff')
sub = d1.get_map('sub')

with d1.begin_transaction() as txn:
    root.set(txn, "foo", "bar")
    sub.set(txn, "a", "b")
    root.set(txn, "sub", sub)

Will raise "SystemError: <method 'set' of 'builtins.YMap' objects> returned a result with an error set", so I have no way to know what happens and create a minimal working example out of it.

Wheel for macOS and Python 3.10 is not on Pypi

@agoose77 noticed over in jupyterlab/jupyterlab#12436 that there is no wheel for MacOS Python 3.10 up on pypi: https://pypi.org/project/y-py/#files

Indeed, trying to install in a new Python 3.10 environment on macOS 12.3 (on an m1 chip) leads to errors about not having Cargo (i.e., it picked up the sdist instead of a wheel):

Command-line output

$ pip install y-py

Collecting y-py

Using cached y_py-0.4.3.tar.gz (30 kB)

Installing build dependencies ... done

Getting requirements to build wheel ... done

Preparing metadata (pyproject.toml) ... error

error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.

│ exit code: 1

╰─> [6 lines of output]
  Cargo, the Rust package manager, is not installed or is not on PATH.
  This package requires Rust and Cargo to compile extensions. Install it through
  the system's package manager or via https://rustup.rs/
  
  Checking for Rust toolchain....
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

error: metadata-generation-failed
× Encountered error while generating package metadata.

╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.

hint: See above for details.

Ypy int passed as BigInt to Yjs

It looks like an int in y-py is received as a BigInt by Yjs.

Mappings are missing keys() and values() methods

The mapping should mirror Python mapping API, and provide not only items(), but also keys() and values().

Cannot use non string mapping key

Doing this:

import y_py as Y

d1 = Y.YDoc()
root = d1.get_map('stuff')

with d1.begin_transaction() as txn:
    root.set(txn, 1, 1)

Will result in "TypeError: argument 'key': 'int' object cannot be converted to 'PyString'".

However, using any hashable object as a mapping key is a common pattern. ypy should either support it, or raise a more explicit exception and document this behavior.

Upgrade to yrs 0.13+

I have looked a bit into how an upgrade to yrs 0.13+ could look like (mainly because I want XMLFragement support).

Apparently transactions are now required in more places and the previous quite Pythonic Python API is no longer easily achievable.

E.g. previously:

d1 = Y.YDoc()
text = d1.get_text('test')
print(len(text))

would no longer work as text.__len__ needs access to a read transaction now.

I wanted to gather some thoughts about how the API design should go forward. Here are two alternatives that came to mind:

1. Trying to keep it Pythonic

d1 = Y.YDoc()

with d1.begin_transaction() as txn:
    text = txn.get_text('test')
    print(len(text))
    text.extend("hello world!")

This would store a reference to the transaction in YTextand would then pass it internally to yrs::types::text::Text.len etc.
Not sure if that's a good idea, as the YText objects would need to be invalidated when outside the transaction block which may come as a surprise.

2. Embrace the Rust interface

d1 = Y.YDoc()
text = d1.get_text('test')
with d1.begin_transaction() as txn:
    print(text.len(txn))

This will not be Pythonic, removing e.g. __getitem__ and __iter__ implementations but doesn't hide the complexity and may be an easy forward.

Any other ideas or thoughts?

Observe is not recursive?

Consider the following code, where ydoc has a YMap, which has a YArray in "array":

import y_py as Y

ydoc = Y.YDoc()
ymap = ydoc.get_map("map")
yarray = Y.YArray()

with ydoc.begin_transaction() as t:
    ymap.set(t, "array", yarray)

def callback(event):
    print(event.target)

ymap.observe(callback)
print("Changes:")
with ydoc.begin_transaction() as t:
    ymap["array"].push(t, [0])

ymap["array"].observe(callback)
print("Changes:")
with ydoc.begin_transaction() as t:
    ymap["array"].push(t, [1])

The output is:

Changes:
Changes:
[0.0, 1.0]

Meaning that observing only the top-level YMap doesn't trigger the callback if the YArray is changed.
I think we should have an option to enable that, e.g. ymap.observe(callback, recursive=True). What do you think?

Updates seem to not apply

In this example, the following characters are supposed to be appended to ysource: a, b, c, d. But if you run it you can see that only a and b are appended.

import y_py as Y

ydoc = Y.YDoc()
ystate = ydoc.get_map("state")
ysource = ydoc.get_text("source")

updates = [
    [1, 2, 242, 196, 218, 129, 3, 0, 40, 1, 5, 115, 116, 97, 116, 101, 5, 100, 105, 114, 116, 121, 1, 121, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 4, 112, 97, 116, 104, 1, 119, 13, 117, 110, 116, 105, 116, 108, 101, 100, 52, 46, 116, 120, 116, 0],
    [1, 1, 242, 196, 218, 129, 3, 2, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 13, 108, 97, 115, 116, 95, 109, 111, 100, 105, 102, 105, 101, 100, 1, 119, 27, 50, 48, 50, 50, 45, 48, 52, 45, 49, 51, 84, 49, 48, 58, 49, 48, 58, 53, 55, 46, 48, 55, 51, 54, 50, 51, 90, 0],
    [1, 2, 242, 196, 218, 129, 3, 3, 4, 1, 6, 115, 111, 117, 114, 99, 101, 1, 97, 168, 242, 196, 218, 129, 3, 0, 1, 120, 0],
    [1, 1, 242, 196, 218, 129, 3, 4, 168, 242, 196, 218, 129, 3, 0, 1, 120, 1, 242, 196, 218, 129, 3, 1, 0, 1],
    [1, 1, 152, 182, 129, 244, 193, 193, 227, 4, 0, 168, 242, 196, 218, 129, 3, 4, 1, 121, 1, 242, 196, 218, 129, 3, 2, 0, 1, 4, 1],
    [1, 2, 242, 196, 218, 129, 3, 5, 132, 242, 196, 218, 129, 3, 3, 1, 98, 168, 152, 190, 167, 244, 1, 0, 1, 120, 0],
    [1, 1, 242, 196, 218, 129, 3, 6, 168, 152, 190, 167, 244, 1, 0, 1, 120, 1, 152, 190, 167, 244, 1, 1, 0, 1],
    [1, 1, 242, 196, 218, 129, 3, 7, 132, 242, 196, 218, 129, 3, 5, 1, 99, 0],
    [1, 1, 242, 196, 218, 129, 3, 8, 132, 242, 196, 218, 129, 3, 7, 1, 100, 0],
]

for update in updates:
    Y.apply_update(ydoc, update)
    print(ystate.to_json())
    print(ysource.to_json())

Type checking fails

Running mypy on this file:

import y_py as Y

ydoc = Y.YDoc()
update = Y.encode_state_as_update(ydoc)

Leads to the following errors:

foo.py:3: error: Missing positional arguments "client_id", "offset_kind", "skip_gc" in call to "YDoc"
foo.py:4: error: Missing positional argument "vector" in call to "encode_state_as_update"
Found 2 errors in 1 file (checked 1 source file)

Parent points to a block which is not a shared type

We are using ypy in a project and in some cases we are getting following error:

thread '<unnamed>' panicked at 'Defect: parent points to a block which is not a shared type', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/yrs-0.12.2/src/block.rs:1128:30

and once this error comes, the document gets corrupted and starts throwing following error:

pyo3_runtime.PanicException: Couldn't get item's parent

Here is how I am processing the message from clients (this is copied from ypy-websocket and modified for our use case):

async def process_message(self, message: bytes, ydoc: Y.YDoc):
        if message[0] != YMessageType.SYNC:
            return
        message_type = message[1]
        msg = message[2:]
        if message_type == YSyncMessageType.SYNC_STEP1:
            state = read_message(msg)
            update = Y.encode_state_as_update(ydoc, state)      **# this is where the error occurs.**
            reply = create_sync_step2_message(update)
            await self.send_message(reply)
        elif message_type in (YSyncMessageType.SYNC_STEP2, YSyncMessageType.SYNC_UPDATE):
            update = read_message(msg)
            # Ignore empty updates (see https://github.com/y-crdt/ypy/issues/98)
            if update == b"\x00\x00":
                return
            try:
                Y.apply_update(ydoc, update)
            except:  # noqa: E722
                # here i am ignoring errors from Rust.
                logger.exception("Bad message - not applying and storing in redis")
            else:
                return update

Before the ypy==0.5.5 the error used to happen only on Y.apply_update(ydoc, update) but now it has started happening inside Y.encode_state_as_update(ydoc, state) also.

I tried to debug it lot but couldn't identify the exact problem. If it helps I am using Lexicaljs as my frontend which generates these updates and I am storing them in redis (also merging updates every 5 minutes when there is no client connected).

This was the last update which might have caused the shared type error:

b'\x01\x01\xea\xe9\x89!:\x84\xea\xe9\x89!9\x01n\x00'

Following is the state and message which caused bad parent afterwards:
state: b'\x00'
message: b'\x00\x00\x01\x00'

PS: https://github.com/y-crdt/ypy-websocket/blob/main/ypy_websocket/yutils.py#L46 - read_message in above code.

Add Rich Text Formatting

Update YText to handle rich text attributes and formatting.

Attempt to divide by zero

Just posting a ypy error I got today, I don't know what caused it except that it was while removing the only character in a YText and then probably pushing an empty string.

thread '' panicked at 'attempt to divide by zero', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/yrs-0.7.0/src/block_store.rs:257:28 stack backtrace: 0: 0x7fd6bb6cc6dd - std::backtrace_rs::backtrace::libunwind::trace::hee598835bc88d35b at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5 1: 0x7fd6bb6cc6dd - std::backtrace_rs::backtrace::trace_unsynchronized::h9cdc730ba5cf5d72 at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5 2: 0x7fd6bb6cc6dd - std::sys_common::backtrace::_print_fmt::h75aeaf7ed30e43fa at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys_common/backtrace.rs:66:5 3: 0x7fd6bb6cc6dd - ::fmt::h606862f787600875 at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys_common/backtrace.rs:45:22 4: 0x7fd6bb6eebdc - core::fmt::write::he803f0f418caf762 at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/fmt/mod.rs:1190:17 5: 0x7fd6bb6ca2e8 - std::io::Write::write_fmt::h70bc45872f37e7bb at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/io/mod.rs:1657:15 6: 0x7fd6bb6ce187 - std::sys_common::backtrace::_print::h64d038cf8ac3e13e at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys_common/backtrace.rs:48:5 7: 0x7fd6bb6ce187 - std::sys_common::backtrace::print::h359300b4a7fccf65 at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys_common/backtrace.rs:35:9 8: 0x7fd6bb6ce187 - std::panicking::default_hook::{{closure}}::hf51be35e2f510149 at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:295:22 9: 0x7fd6bb6cde50 - std::panicking::default_hook::h03ca0f22e1d2d25e at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:314:9 10: 0x7fd6bb6ce8d9 - std::panicking::rust_panic_with_hook::h3b7380e99b825b63 at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:698:17 11: 0x7fd6bb6ce589 - std::panicking::begin_panic_handler::{{closure}}::h8e849d0710154ce0 at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:586:13 12: 0x7fd6bb6ccba4 - std::sys_common::backtrace::__rust_end_short_backtrace::hedcdaddbd4c46cc5 at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/sys_common/backtrace.rs:138:18 13: 0x7fd6bb6ce2d9 - rust_begin_unwind at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5 14: 0x7fd6bb621613 - core::panicking::panic_fmt::he1bbc7336d49a357 at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14 15: 0x7fd6bb6214dd - core::panicking::panic::h4241c5ccea17faca at /rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:48:5 16: 0x7fd6bb678a19 - yrs::transaction::Transaction::commit::h12039726d879d785 17: 0x7fd6bb62caa2 - std::panicking::try::h647896984698a0a9 18: 0x7fd6bb64b6b2 - y_py::y_transaction::_:: for pyo3::impl_::pyclass::PyClassImplCollector>::py_methods::ITEMS::__wrap::hc8edfdcc5059ed2b 19: 0x564104b44248 - method_vectorcall_FASTCALL_KEYWORDS at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/descrobject.c:405 20: 0x564104b8bd1b - _PyObject_VectorcallTstate at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:114:11 21: 0x564104b8bd1b - PyObject_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:123 22: 0x564104b8bd1b - call_function at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5867 23: 0x564104b8bd1b - _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:4198 24: 0x564104b6efd5 - _PyEval_EvalFrame at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/internal/pycore_ceval.h:46 25: 0x564104b6efd5 - _PyEval_Vector at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5065 26: 0x564104b6efd5 - _PyFunction_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:342 27: 0x564104b6efd5 - _PyObject_VectorcallTstate at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:114:11 28: 0x564104b6efd5 - method_vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/classobject.c:53 29: 0x564104b8ba14 - _PyObject_VectorcallTstate at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:114:11 30: 0x564104b8ba14 - PyObject_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:123 31: 0x564104b8ba14 - call_function at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5867 32: 0x564104b8ba14 - _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:4213 33: 0x564104b6dddd - _PyEval_EvalFrame at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/internal/pycore_ceval.h:46 34: 0x564104b6dddd - _PyEval_Vector at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5065 35: 0x564104b6dddd - _PyFunction_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:342 36: 0x564104b57366 - _PyObject_VectorcallTstate at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:114:11 37: 0x564104b57366 - object_vacall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:734 38: 0x564104c189c8 - PyObject_CallFunctionObjArgs at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:841 39: 0x564104c18b55 - property_descr_set at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/descrobject.c:1637 40: 0x564104b78246 - _PyObject_GenericSetAttrWithDict at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/object.c:1353 41: 0x564104b78246 - PyObject_GenericSetAttr at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/object.c:1403 42: 0x564104b78246 - PyObject_SetAttr at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/object.c:1034 43: 0x564104b8c467 - _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:2850 44: 0x564104b61266 - _PyEval_EvalFrame at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/internal/pycore_ceval.h:46 45: 0x564104b61266 - gen_send_ex2 at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/genobject.c:213 46: 0x564104b61266 - PyGen_am_send at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/genobject.c:272 47: 0x564104b61266 - PyIter_Send at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/abstract.c:2893 48: 0x564104b8cabf - _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:2586 49: 0x564104b61266 - _PyEval_EvalFrame at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/internal/pycore_ceval.h:46 50: 0x564104b61266 - gen_send_ex2 at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/genobject.c:213 51: 0x564104b61266 - PyGen_am_send at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/genobject.c:272 52: 0x564104b61266 - PyIter_Send at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/abstract.c:2893 53: 0x7fd6bf75c81b - task_step_impl at /usr/local/src/conda/python-3.10.2/Modules/_asynciomodule.c:2655:22 54: 0x7fd6bf75c81b - task_step at /usr/local/src/conda/python-3.10.2/Modules/_asynciomodule.c:2952:11 55: 0x7fd6bf75d09d - task_wakeup at /usr/local/src/conda/python-3.10.2/Modules/_asynciomodule.c:3001:20 56: 0x564104b43be2 - cfunction_vectorcall_O at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/methodobject.c:516 57: 0x564104bdb841 - _PyObject_VectorcallTstate at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:114:11 58: 0x564104a3e676 - context_run at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/context.c:665 59: 0x564104af197f - cfunction_vectorcall_FASTCALL_KEYWORDS at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/methodobject.c:446 60: 0x564104aef21a - PyVectorcall_Call at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:255 61: 0x564104aef21a - _PyObject_Call at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:290 62: 0x564104aef21a - PyObject_Call at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:317:12 63: 0x564104b91367 - do_call_core at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5891 64: 0x564104b91367 - _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:4277 65: 0x564104b6dddd - _PyEval_EvalFrame at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/internal/pycore_ceval.h:46 66: 0x564104b6dddd - _PyEval_Vector at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5065 67: 0x564104b6dddd - _PyFunction_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:342 68: 0x564104b8bd1b - _PyObject_VectorcallTstate at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:114:11 69: 0x564104b8bd1b - PyObject_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:123 70: 0x564104b8bd1b - call_function at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5867 71: 0x564104b8bd1b - _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:4198 72: 0x564104b6dddd - _PyEval_EvalFrame at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/internal/pycore_ceval.h:46 73: 0x564104b6dddd - _PyEval_Vector at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5065 74: 0x564104b6dddd - _PyFunction_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:342 75: 0x564104b8bd1b - _PyObject_VectorcallTstate at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:114:11 76: 0x564104b8bd1b - PyObject_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:123 77: 0x564104b8bd1b - call_function at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5867 78: 0x564104b8bd1b - _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:4198 79: 0x564104b6dddd - _PyEval_EvalFrame at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/internal/pycore_ceval.h:46 80: 0x564104b6dddd - _PyEval_Vector at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5065 81: 0x564104b6dddd - _PyFunction_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:342 82: 0x564104b8bd1b - _PyObject_VectorcallTstate at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:114:11 83: 0x564104b8bd1b - PyObject_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:123 84: 0x564104b8bd1b - call_function at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5867 85: 0x564104b8bd1b - _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:4198 86: 0x564104b6dddd - _PyEval_EvalFrame at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/internal/pycore_ceval.h:46 87: 0x564104b6dddd - _PyEval_Vector at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5065 88: 0x564104b6dddd - _PyFunction_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:342 89: 0x564104b8bd1b - _PyObject_VectorcallTstate at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:114:11 90: 0x564104b8bd1b - PyObject_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:123 91: 0x564104b8bd1b - call_function at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5867 92: 0x564104b8bd1b - _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:4198 93: 0x564104b6dddd - _PyEval_EvalFrame at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/internal/pycore_ceval.h:46 94: 0x564104b6dddd - _PyEval_Vector at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5065 95: 0x564104b6dddd - _PyFunction_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:342 96: 0x564104b8bd1b - _PyObject_VectorcallTstate at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:114:11 97: 0x564104b8bd1b - PyObject_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:123 98: 0x564104b8bd1b - call_function at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5867 99: 0x564104b8bd1b - _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:4198 100: 0x564104b6dddd - _PyEval_EvalFrame at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/internal/pycore_ceval.h:46 101: 0x564104b6dddd - _PyEval_Vector at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5065 102: 0x564104b6dddd - _PyFunction_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:342 103: 0x564104b8bd1b - _PyObject_VectorcallTstate at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:114:11 104: 0x564104b8bd1b - PyObject_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:123 105: 0x564104b8bd1b - call_function at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5867 106: 0x564104b8bd1b - _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:4198 107: 0x564104b6efd5 - _PyEval_EvalFrame at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/internal/pycore_ceval.h:46 108: 0x564104b6efd5 - _PyEval_Vector at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5065 109: 0x564104b6efd5 - _PyFunction_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/call.c:342 110: 0x564104b6efd5 - _PyObject_VectorcallTstate at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:114:11 111: 0x564104b6efd5 - method_vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Objects/classobject.c:53 112: 0x564104b8ba14 - _PyObject_VectorcallTstate at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:114:11 113: 0x564104b8ba14 - PyObject_Vectorcall at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/cpython/abstract.h:123 114: 0x564104b8ba14 - call_function at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5867 115: 0x564104b8ba14 - _PyEval_EvalFrameDefault at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:4213 116: 0x564104b6cdd9 - _PyEval_EvalFrame at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Include/internal/pycore_ceval.h:46 117: 0x564104b6cdd9 - _PyEval_Vector at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:5065 118: 0x564104c246d7 - PyEval_EvalCode at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/ceval.c:1134 119: 0x564104c24799 - run_eval_code_obj at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/pythonrun.c:1291 120: 0x564104c4a7a4 - run_mod at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/pythonrun.c:1312 121: 0x564104c51e69 - pyrun_file at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/pythonrun.c:1208 122: 0x564104c5201f - _PyRun_SimpleFileObject at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/pythonrun.c:456 123: 0x564104c52123 - _PyRun_AnyFileObject at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Python/pythonrun.c:90 124: 0x564104c53018 - pymain_run_file_obj at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Modules/main.c:357 125: 0x564104c53018 - pymain_run_file at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Modules/main.c:376 126: 0x564104c53018 - pymain_run_python at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Modules/main.c:591:21 127: 0x564104c53018 - Py_RunMain at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Modules/main.c:670 128: 0x564104c53169 - Py_BytesMain at /home/conda/feedstock_root/build_artifacts/python-split_1646754077016/work/Modules/main.c:1090 129: 0x7fd6c0914d90 - 130: 0x7fd6c0914e40 - __libc_start_main 131: 0x564104bbcce1 -

Y.encode_state_vector(ydoc) returns a list

It looks like Y.encode_state_vector(ydoc) returns a list. I think it should return bytes now that 4d13308 is in.

Uploading emscripten wheels to pypi fails

Nooo! https://github.com/y-crdt/ypy/actions/runs/3566940726/jobs/5994217644

I did not see this coming. Apologies, I should have tried to upload an emscripten wheel to test pypi on another project. Looks like pypi/warehouse#10416 is relevant.

I'll revert the wasm build in ci/cd later today if someone else doesn't get to it first.

ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         Binary wheel 'y_py-0.5.5-cp310-cp310-emscripten_3_1_14_wasm32.whl' has 
         an unsupported platform tag 'emscripten_3_1_14_wasm32'.                
Error: Process completed with exit code 1.

Defect: item has no parent

I have a pyo3_runtime.PanicException: Defect: item has no parent error while applying the last update in this code:

import y_py as Y

d = Y.YDoc()

x = d.get_text("test")

updates = [
    [1, 2, 208, 216, 199, 216, 11, 0, 40, 1, 5, 115, 116, 97, 116, 101, 5, 100, 105, 114, 116, 121, 1, 121, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 4, 112
, 97, 116, 104, 1, 119, 13, 117, 110, 116, 105, 116, 108, 101, 100, 50, 46, 116, 120, 116, 0],
    [1, 1, 208, 216, 199, 216, 11, 2, 4, 1, 6, 115, 111, 117, 114, 99, 101, 1, 32, 0],
    [1, 1, 208, 216, 199, 216, 11, 3, 168, 208, 216, 199, 216, 11, 0, 1, 120, 1, 208, 216, 199, 216, 11, 1, 0, 1],
    [1, 1, 208, 216, 199, 216, 11, 4, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 13, 108, 97, 115, 116, 95, 109, 111, 100, 105, 102, 105, 101, 100, 1, 119, 2
7, 50, 48, 50, 50, 45, 48, 52, 45, 48, 54, 84, 48, 55, 58, 53, 49, 58, 52, 57, 46, 51, 50, 49, 53, 54, 52, 90, 0],
    [1, 1, 208, 216, 199, 216, 11, 5, 168, 208, 216, 199, 216, 11, 3, 1, 121, 1, 208, 216, 199, 216, 11, 1, 3, 1],
    [1, 1, 208, 216, 199, 216, 11, 6, 168, 208, 216, 199, 216, 11, 5, 1, 120, 1, 208, 216, 199, 216, 11, 1, 2, 1],
    [1, 1, 208, 216, 199, 216, 11, 6, 168, 208, 216, 199, 216, 11, 5, 1, 120, 1, 208, 216, 199, 216, 11, 1, 5, 1],
    [1, 2, 208, 216, 199, 216, 11, 7, 132, 208, 216, 199, 216, 11, 2, 1, 32, 168, 242, 241, 203, 145, 14, 0, 1, 120, 0]
]

for update in updates:
    Y.apply_update(d, bytes(update))

I'm not sure what's going on, maybe it is not related to y-py, but it's hard to debug.
Any idea?

Error applying update

I have this error:

"thread '' panicked at 'called Option::unwrap() on a None value', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/yrs-0.7.1/src/transaction.rs:571:59"

when applying this update to a document:

b"\x01\x0e\xde\xd6\xe0\xbe\x0c\t\xc1\xaa\xb7\xc3\xa8\x04s\xaa\xb7\xc3\xa8\x04z\x01\x00\x06\xc1\xde\xd6\xe0\xbe\x0c\t\xaa\xb7\xc3\xa8\x04z\x01\x00\x06\xc7\xc0\xcb\xf4\xe6\r\x96\x01\xab\xf3\xab\xff\r\n\x01(\x00\xde\xd6\xe0\xbe\x0c\x17\x02id\x01w$a584a3d2-2490-4926-b156-0e9fa9b5d718(\x00\xde\xd6\xe0\xbe\x0c\x17\tcell_type\x01w\x04code'\x00\xde\xd6\xe0\xbe\x0c\x17\x06source\x02\x04\x00\xde\xd6\xe0\xbe\x0c\x1a\x052 + 2'\x00\xde\xd6\xe0\xbe\x0c\x17\x08metadata\x01(\x00\xde\xd6\xe0\xbe\x0c \x07trusted\x01x(\x00\xde\xd6\xe0\xbe\x0c\x17\x0fexecution_count\x01}\x0c'\x00\xde\xd6\xe0\xbe\x0c\x17\x07outputs\x00\x08\x00\xde\xd6\xe0\xbe\x0c#\x01v\x04\x0fexecution_count}\x0c\x0boutput_typew\x0eexecute_result\x04datav\x01\ntext/plainw\x014\x08metadatav\x00\x1c\xf6\xe7\x84\x95\x0f\x01\x00\x05\xd4\x80\x96\xbb\x02\x01\x00\r\xa0\x84\x82\xd9\t\x01\x00?\xf9\xa3\xae\xba\x06\x01\x00\x1f\xce\xeb\x90\xaf\x0b\x01\x00\x03\xab\xf3\xab\xff\r\x02\x00\xce\x01\xd5\x01B\xff\xc6\xe3\xe8\x0b\x01\x00\x03\x90\x8b\x8d\xbe\x07\x01\x00\x03\x82\xc0\xd5\xa1\n\x01\x00^\xd0\xa5\xa2\xa7\x02\x01\x00\x03\xfc\xbc\x9a\xa9\x03\x01\x00\x03\xed\xbf\xb4\xd5\x0b\x02\x00\x07\tm\xaa\xb7\xc3\xa8\x04\x02\x00elF\x8c\xe0\xcc\xcc\x03\x01\x00\x95\x01\xc1\x80\xa5\xa4\x07\x02\x00\x03\x17\x92\x02\x9f\xdc\xc6\xa8\x01\x01\x00\x03\xc0\xcb\xf4\xe6\r\x01\x00\xc5\x01\xdf\xdc\xe4\xe2\x0e\x01\x00\x10\xed\xec\x9d\xfd\x08\x01\x00\x03\xa6\xc5\xfd\xa0\x0e\x01\x009\xe0\x93\xc3\xf6\x04\x03\x009J\x01LH\xde\xd6\xe0\xbe\x0c\x02\x00\x04\x06\x11\xe7\xce\xda\xc1\x0f\x01\x00\x03\xb6\xab\xb3c\x01\x00\xd4\x01\x89\xb1\x84\xcd\x0e\x01\x00\x05\xb6\xdb\xec\x87\x05\x01\x00\x03\xff\xe6\xd5\xf2\x0b\x01\x00\x03\xbc\xc5\x89\xcf\x0b\x03\x00\xa4\x04\xbf\x05\x14\xf5\x05\x14"

I'm not sure if you can figure out something wrong with the update itself. Let me know if you need mode details.

Updates don't apply

These updates are supposed to push "b" to the initial content "a" of a YText, but it doesn't work:

import y_py as Y

ydoc1 = Y.YDoc()
ysource1 = ydoc1.get_text("source")

with ydoc1.begin_transaction() as t:
    ysource1.push(t, "a")

updates = [
    [1, 2, 201, 210, 153, 56, 0, 40, 1, 5, 115, 116, 97, 116, 101, 5, 100, 105, 114, 116, 121, 1, 121, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 4, 112, 97, 116, 104, 1, 119, 13, 117, 110, 116, 105, 116, 108, 101, 100, 52, 46, 116, 120, 116, 0],
    [1, 1, 201, 210, 153, 56, 2, 168, 201, 210, 153, 56, 0, 1, 120, 1, 201, 210, 153, 56, 1, 0, 1],
    [1, 1, 201, 210, 153, 56, 3, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 13, 108, 97, 115, 116, 95, 109, 111, 100, 105, 102, 105, 101, 100, 1, 119, 27, 50, 48, 50, 50, 45, 48, 52, 45, 49, 54, 84, 49, 52, 58, 48, 51, 58, 53, 51, 46, 57, 51, 48, 52, 54, 56, 90, 0],
    [1, 1, 201, 210, 153, 56, 4, 168, 201, 210, 153, 56, 2, 1, 121, 1, 201, 210, 153, 56, 1, 2, 1],
    [1, 2, 201, 210, 153, 56, 5, 132, 228, 254, 237, 171, 7, 0, 1, 98, 168, 201, 210, 153, 56, 4, 1, 120, 0],
    [1, 1, 201, 210, 153, 56, 6, 168, 201, 210, 153, 56, 4, 1, 120, 1, 201, 210, 153, 56, 1, 4, 1],
]

for update in updates:
    Y.apply_update(ydoc1, update)

print(ysource1.to_json())

`to_json()` generates updates

The following example shows that calling to_json() actually generates an update:

import y_py as Y

ydoc = Y.YDoc()
a = ydoc.get_array("a")

def cb(event):
    print("update:", event.get_update())

ydoc.observe_after_transaction(cb)
print("a =", a.to_json())

# update: b'\x00\x00'
# a = []

For a notebook document, it can generate quite a lot of them.
I am surprised that it happens for an action that doesn't modify the document. Is it expected? What do these updates correspond to?

cc @ellisonbg

Unable to install with Rust compiliation error

Description

I'm trying to install ypy as a transitive dependency and am getting the following error. I tried with 0.5.0 and 0.5.2 with the same result.

> rustc -V
rustc 1.57.0 (f1edd0429 2021-11-29)

> cargo -V
cargo 1.57.0 (b2e52d7ca 2021-10-21)

> maturin --help
maturin 0.12.20

> pipenv --version
pipenv, version 2022.6.7

> python --version
Python 3.8.9

> pipenv install y-py==0.5.0
Installing y-py==0.5.0...
Error:  An error occurred while installing y-py==0.5.0!
Error text: Collecting y-py==0.5.0
  Downloading y_py-0.5.0.tar.gz (36 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: y-py
  Building wheel for y-py (pyproject.toml): started
  Building wheel for y-py (pyproject.toml): finished with status 'error'
Failed to build y-py

  error: subprocess-exited-with-error

  × Building wheel for y-py (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [85 lines of output]
      Running `maturin pep517 build-wheel -i /Users/ethan/.local/share/virtualenvs/3d-models-8Ws8ixiA/bin/python --compatibility off`
      ⚠️  Warning: Please use maturin in pyproject.toml with a version constraint, e.g. `requires = ["maturin>=0.12,<0.13"]`. This will become an error.
         Compiling proc-macro2 v1.0.40
         Compiling libc v0.2.126
         Compiling unicode-ident v1.0.1
         Compiling quote v1.0.20
         Compiling target-lexicon v0.12.4
         Compiling syn v1.0.98
         Compiling cfg-if v1.0.0
         Compiling wasm-bindgen-shared v0.2.81
         Compiling once_cell v1.12.1
         Compiling log v0.4.17
         Compiling getrandom v0.1.16
         Compiling autocfg v1.1.0
         Compiling smallvec v1.9.0
         Compiling lazy_static v1.4.0
         Compiling bumpalo v3.10.0
         Compiling parking_lot_core v0.9.3
         Compiling wasm-bindgen v0.2.81
         Compiling ppv-lite86 v0.2.16
         Compiling scopeguard v1.1.0
         Compiling unindent v0.1.9
         Compiling indoc v1.0.6
         Compiling lib0 v0.7.1
         Compiling smallstr v0.2.0
         Compiling lock_api v0.4.7
         Compiling pyo3-build-config v0.16.5
         Compiling rand_core v0.5.1
         Compiling parking_lot v0.12.1
         Compiling rand_chacha v0.2.2
         Compiling rand v0.7.3
         Compiling pyo3-ffi v0.16.5
         Compiling pyo3 v0.16.5
         Compiling wasm-bindgen-backend v0.2.81
         Compiling pyo3-macros-backend v0.16.5
         Compiling wasm-bindgen-macro-support v0.2.81
         Compiling wasm-bindgen-macro v0.2.81
         Compiling pyo3-macros v0.16.5
         Compiling yrs v0.7.1
         Compiling y-py v0.5.0 (/private/var/folders/gq/cpjxq2196f3dmyj7hjs8pv_h0000gn/T/pip-install-90z9pgas/y-py_d793641b962c41ff8c6ae81ddc8ee697)
      error: could not compile `y-py` due to 6 previous errors
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `cargo rustc --manifest-path Cargo.toml --message-format json --release --lib -- -C link-arg=-undefined -C link-arg=dynamic_lookup -C link-args=-Wl,-install_name,@rpath/y_py.cpython-38-darwin.so`
      🔗 Found pyo3 bindings
      🐍 Found CPython 3.8 at /Users/ethan/.local/share/virtualenvs/3d-models-8Ws8ixiA/bin/python
      error: there is no argument named `shared`
         --> src/type_conversions.rs:252:102
          |
      252 |                     "Cannot integrate a nested Ypy object because is already integrated into a YDoc: {shared}"
          |                                                                                                      ^^^^^^^^


      error: there is no argument named `v`
         --> src/type_conversions.rs:267:98
          |
      267 |                 "Cannot integrate a nested Ypy object because is already integrated into a YDoc: {v}"
          |                                                                                                  ^^^


      error: there is no argument named `v`
         --> src/type_conversions.rs:271:58
          |
      271 |                 "Cannot integrate this type into a YDoc: {v}"
          |                                                          ^^^


      error: there is no argument named `key`
         --> src/y_map.rs:161:46
          |
      161 |             Err(PyKeyError::new_err(format!("{key}")))
          |                                              ^^^^^


      error: there is no argument named `key`
         --> src/y_map.rs:182:58
          |
      182 |         entry.ok_or_else(|| PyKeyError::new_err(format!("{key}")))
          |                                                          ^^^^^


      error: aborting due to 5 previous errors


      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/Users/ethan/.local/share/virtualenvs/3d-models-8Ws8ixiA/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for y-py
ERROR: Could not build wheels for y-py, which is required to install pyproject.toml-based projects

Building on an M1 Mac

Build `emscripten` wheels

I want to micropip.install('y-py') in a Pyodide script and use the same syntax in Pyodide as I do with server-side Python. I understand I could just plug into the yjs library itself through Pyodide's js interface, but I would prefer the consistent syntax and ability to prototype/test in Python server side.

I've put together https://github.com/kafonek/py-rust-fib as a reference while learning how y-crdt and ypy work together, starting at zero with Rust. It's been a fun experience, changes some of my perspectives on parts of the Python ecosystem.

Following this section in py-rust-fib, I tried using emsdk version 3.1.14 (matching emscripten version for Pyodide 0.21.3) to build an emscripten wheel. Loading it into Pyodide failed on the first try because maturin does not have a pure Python wheel, but when I moved that into an optional dependency in the hatch branch, I was also able to import y_py into a Pyodide worker!

(js.console.log(y_py.YDoc()))

I'll work on a PR if building emscripten wheels sounds good to you all.

Add `YArray.__set_item__`

It looks like a YArray has a __get_item__ but no __set_item__, which would be useful.

YMapEvent issues

The documentation says:

"The target references the YText element that receives the update". It is not a YText but a YMap.
keys :List[YMapEventKeyChange]: it is not a List but a Dict.
delta :List[Dict]: there is not delta attribute.
path(self) → List[Union[int, str]]: it seems to always return an empty list. Also I don't understand the description: "Array of keys and indexes creating a path from root type down to current instance of shared type (accessible via target getter)".

Also, how can we access the YMapEvent object? y_py.YMapEvent doesn't exist.

Building ypy with setuptools_rust instead of maturin

Hi.

I have one question which might be silly. I'm trying to package JupyterLab and Jupyter Notebook into Fedora Linux and this package is in the dependency graph. To be able to package it, I have to build it and maturin is not available in Fedora yet, unfortunately.

So, the question is: Is there any chance to build ypy with setuptools_rust instead of maturin if I prepare a configuration for it? I'd like to know whether it makes sense to start investigating it or if it's completely impossible. I have a lot of experience with Python packaging but almost no experience with Rust.

Thanks a lot and have a nice day.

Release flow not triggered

Hi @Waidhoferj,

I just tried to publish a new release (with your changes to the observer API). However, it seems that it is not published to PyPI. Do you have an idea why Release doesn't run?

Expose transaction's before_state

It looks useful to expose the transaction's before_state property.

Implement YDoc's event handlers

It looks like YDoc's event handlers are not implemented yet in ypy. I can work around that in https://github.com/davidbrochart/ypy-websocket, but I will need them soon.

Error when moving an element from yjs

When using the new move feature from a Yjs client, If there is a Ypy client connected, the first client (the Yjs client that moved an element) receives an event undoing the move.

Everything works fine if there is not a Ypy client connected or if the client moving an element is a python client (Ypy) .
I suspect the problem comes from my PR for the move feature #83, which I guess is missing handling a moving event coming from another client.

I have been looking into the code to see how Yrs or Ypy handles the events coming from other clients but I couldn't find anything.
@Waidhoferj, @Horusiath or @dmonad do you know where should I look in the code to debug it?

Context

To reproduce the bug you can use a Yjs client like:

const doc = new Doc();
const test = doc.getArray('test');
const provider = new WebsocketProvider('ws://localhost:8888', 'rtc_yjs_test', doc);

const observe = (event) => console.log("OBSERVER:", event.changes);
test.observe(observe);

test.push([0,1,2,3,4]);
test.move(0, 2);

and a Ypy client like:

def callbackArray(event):
    print("OBSERVER:", event.delta)

doc = Y.YDoc()
test = doc.get_array('test')
idTest = test.observe(callbackArray)

ws = await connect("ws://localhost:8888/rtc_yjs_test")
WebsocketProvider(doc, ws)

and you can start a WebSocket server with:
HOST=localhost PORT=8888 npx y-websocket

When the Yjs client moves an element it receives two events, the first event is the correct move of the element:

{
  "added": {},
  "deleted": {},
  "delta": [
    { "delete": 1 },
    { "retain": 1 },
    { "insert": [0] }
  ]
}

But the second event is reverting the move:

{
  "added": { Item([0]) },
  "deleted": {},
  "delta": [
    { "insert": [0]  },
    { "retain": 1 },
    { "delete": 1}
  ]
}

Access YTransaction

I seems that YTransaction cannot be accessed:

import y_py as Y

Y.YTransaction
# AttributeError: module 'y_py' has no attribute 'YTransaction'

YText API

After #47 and #49 were merged, it seems weird to:

have a YText observe_deep method, since it doesn't have nested attributes.
have a YText delete method which looks like a YArray delete_range method, and not have a delete method that deletes a single character.
have a YText push method, which looks like a YArray extend method.

Thread error

I've noticed that since I've been using y-py in the back-end in Jupyter, I have threading issues while exiting with Ctrl-C:

Exception ignored in: <module 'threading' from '/home/david/mambaforge/envs/jupyterlab-dev/lib/python3.10/threading.py'>
Traceback (most recent call last):
  File "/home/david/mambaforge/envs/jupyterlab-dev/lib/python3.10/threading.py", line 1560, in _shutdown
    lock.acquire()
KeyboardInterrupt: 
FATAL: exception not rethrown
Aborted (core dumped)

I know that Yrs uses threads, so it might be related.

y_py has no attribute `Y.Event`

y_py.pyi defines Event

ypy/y_py.pyi

Line 21 in d3387a5

    
           Event = Union[YTextEvent, YArrayEvent, YMapEvent, YXmlTextEvent, YXmlElementEvent]

so it shows up when tab completing in Jupyter or an editor. However trying to use it will raise AttributeError: module 'y_py' has no attribute 'Event'

Applying updates gives inconsistent result

Using y-py==0.4.3, running the following script sometimes prints "aba", sometimes "aab":

import y_py as Y

ydoc1 = Y.YDoc()
ysource1 = ydoc1.get_text("source")

with ydoc1.begin_transaction() as t:
    ysource1.push(t, "a")

updates = [
    [1, 1, 157, 204, 146, 147, 3, 0, 4, 1, 6, 115, 111, 117, 114, 99, 101, 1, 97, 0],
    [1, 2, 234, 178, 234, 232, 13, 0, 40, 1, 5, 115, 116, 97, 116, 101, 5, 100, 105, 114, 116, 121, 1, 121, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 4, 112, 97, 116, 104, 1, 119, 13, 117, 110, 116, 105, 116, 108, 101, 100, 52, 46, 116, 120, 116, 0],
    [1, 1, 234, 178, 234, 232, 13, 2, 168, 234, 178, 234, 232, 13, 0, 1, 120, 1, 234, 178, 234, 232, 13, 1, 0, 1],
    [1, 1, 234, 178, 234, 232, 13, 3, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 13, 108, 97, 115, 116, 95, 109, 111, 100, 105, 102, 105, 101, 100, 1, 119, 27, 50, 48, 50, 50, 45, 48, 52, 45, 49, 54, 84, 49, 51, 58, 51, 55, 58, 51, 56, 46, 50, 49, 54, 56, 55, 49, 90, 0],
    [1, 1, 234, 178, 234, 232, 13, 4, 168, 234, 178, 234, 232, 13, 2, 1, 121, 1, 234, 178, 234, 232, 13, 1, 2, 1],
    [1, 2, 234, 178, 234, 232, 13, 5, 132, 157, 204, 146, 147, 3, 0, 1, 98, 168, 234, 178, 234, 232, 13, 4, 1, 120, 0],
    [1, 1, 234, 178, 234, 232, 13, 6, 168, 234, 178, 234, 232, 13, 4, 1, 120, 1, 234, 178, 234, 232, 13, 1, 4, 1],
]

for update in updates:
    Y.apply_update(ydoc1, update)

print(ysource1.to_json())

TODOs

add a better description to the github test runner
Try out without rust-nightly
Wrap observer in result to throw error in python, and not from the rust y-py.
What is the idiomatic approach to unsubscribe observers? Try to use that in the observer API.

Expose diff

I'm trying to reconstruct a document composed in slate-yjs with this logic (uses embedded xmltexts instead of xmlelements). I don't seem to be able to access these underlying elements via the current python interface, and I think I need a similar way of accessing the delta representation of the YXmlText, finding the embeds, and re-constructing the nested structure that way.

items() should return an itemview

Currently items returns a generator, but the current dict python API returns a set-like object called a itemview, which ypy should mirror.

Observer API

Discussing approaches to the Observer API

I went a few rounds on subscription management patterns for the Observer API:

Current Pattern:

def callback(event):
   ...

subscription = datatype.observe(callback)
del subscription # Drops the callback from the event handler

This works very well and will guard against leaking callbacks, but requires users to store the subscription reference before exiting the caller's scope. This seems a bit unintuitive and fails silently (the callback just won't receive updates if the user doesn't stash the subscription somewhere).

Observer Pattern

class SomeObserver:
   ...
   def update(event):
       # do callback stuff here

observer = SomeObserver()
datatype.observe(observer)
datatype.unobserve(observer)

While this is the most powerful of the three, since the class has self contained state and functionality, creating a class for every callback seems overkill.

Callback with explicit unobserve

def callback(event):
   ...
datatype.observe(callback)
datatype.unobserve(callback)

This solution is basically the same as 1. but the function pointer serves as the identifier. The advantage of this is that users don't have to store references to the callbacks that they don't wish to cancel. Overall I feel this is more intuitive since users only have to think about ending the callbacks lifecycle if they intend to cancel it, and let it be otherwise. It mirrors the Yjs Observer API which is a nice plus.

To do this, it would be nice to get access to the SubscriptionIds in the EventHandler via some sort of getter method. Optionally, I could create my own manager that holds the Subscriptions.

Deleted key instead of updated

I have a strange behavior where Yjs sets state[dirty] = true, but when I observe the state YMap in y-py I receive an event saying that the dirty key was deleted:

{'dirty': {'action': 'delete', 'oldValue': False}}

The received update was:

[26, 1, 1, 144, 236, 139, 150, 12, 6, 168, 144, 236, 139, 150, 12, 4, 1, 120, 1, 144, 236, 139, 150, 12, 1, 4, 1]

ydoc to json or some readable format

Is it possible with ypy to convert the ydoc contents to some readable format?

WASM wheel does not upload to PyPI

WASM version of Ypy fails to upload to PyPI with the following error message:

HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         Binary wheel 'y_py-0.5.5-cp310-cp310-emscripten_3_1_14_wasm32.whl' has 
         an unsupported platform tag 'emscripten_3_1_14_wasm32'

Resources

Hidden Transactions

Update the API to optionally hide transactions for operations

Usability

Operations that now accept transactions as a first parameter will still accept them as optional params: append(item, [txn]). When the transaction isn't supplied, it will be retrieved implicitly.
This would allow certain native operations to work, like setting elements in a YArray: y_array[0] = 1. If this sort of operation takes place outside of a transaction block, it will throw an error.

Implementation

All integrated YTypes can maintain a reference to a YDoc through something like a RefCell. We add the field: YDoc.current_txn: Option<YTransaction> which allows any type to access the currently active transaction through the YDoc. When a mutable operation takes place without an explicit transaction, the YType checks if its parent YDoc has an active transaction. If it exists, the type borrows it for its one operation and then returns it. If it doesn't exist, an error is thrown.

Concerns

If there are nested transaction blocks from the same YDoc, we will have to implicitly choose one of the transactions. We could have a stack of transactions and peek the top one, or we could just default to the outermost value.
If there are non atomic operations that wrap other implicit transactions, we could get a double borrow error:

def operationA():
    contains operationB()
...
operationA() # borrows txn
operationB() # borrows txn ERROR cannot have two mutable references
# operation B releases borrow
# operation A releases borrow

Unintuitive .to_json() behavior

The behavior for the .to_json() method on YText, YArray, and YMap is not consistent with each other. YText returns a json serialized string. The other two return Python objects. I suggest these methods return JSON strings, and a separate .to_py() method be added to return the Python objects.

import y_py as Y

ydoc = Y.YDoc()
with ydoc.begin_transaction() as txn:
    ytext = txn.get_text('text')
    ytext.extend(txn, "foo")
    
    yarray = txn.get_array('array')
    yarray.append(txn, 'bar')
    
    ymap = txn.get_map('map')
    ymap.set(txn, 'key', 'value')
    
ytext.to_json(), yarray.to_json(), ymap.to_json()
>>> ('"foo"', ['bar'], {'key': 'value'})

import json
json.loads(ytext.to_json())
>>> 'foo'

json.loads(yarray.to_json())
>>> TypeError: the JSON object must be str, bytes or bytearray, not list

Observer's subscription ID shouldn't be required

In the following code we register a callback for an observer and don't keep a reference to the subscription ID:

import y_py as Y

d = Y.YDoc()

def callback(e):
    print(e)

x = d.get_text("test")

def register_cb(x, callback):
    i = x.observe(callback)
    # if i is not returned, the callback will never be called
    # return i  # this works fine (callback called)

i = register_cb(x, callback)

with d.begin_transaction() as txn:
    x.insert(txn, 0, "abcd")

This can happen when we don't want to unobserve. But the issue is that in this case the callback is never called.

Empty updates

@Waidhoferj There is this code in your drawing example:

ypy/examples/drawing/client.py

Lines 28 to 31 in 792dc96

    
           # Sometimes transactions don't write, which means updates are empty. 
        
           # We only care about updates with meaningful mutations. 
        
           if update != b'\x00\x00': 
        
               self.send_q.put_nowait(update)

Why are empty updates emitted? This creates issues here and there.

	# Sometimes transactions don't write, which means updates are empty.
	# We only care about updates with meaningful mutations.
	if update != b'\x00\x00':
	self.send_q.put_nowait(update)