jupyter / jupyter_events

Configurable event system for Jupyter applications and extensions.

Jupyter

Read this in other languages: English, Español, Português, Français

Jupyter metapackage for installation and documents

Documentation structure

This documentation uses the Sphinx documentation engine.

The documentation is located in the docs/source folder. When you build the documentation, it will be placed in the docs/build folder. It is written in a combination of reStructuredText and MyST Markdown.

Build the documentation locally

There are a few ways to build the documentation; see below for instructions:

Build the documentation automatically with nox

The easiest way to build the documentation locally is by using the nox command line tool. This tool makes it easy to automate commands in a repository, and we have included a docs command to quickly install the dependencies and build the documentation.

To build and preview the site locally, follow these steps:

1. Clone this repository.

   $ git clone https://github.com/jupyter/jupyter
   $ cd jupyter

2. Install nox

   $ pip install nox

3. Run the docs command

   $ nox -s docs

This will install the needed dependencies in a virtual environment using pip. It will then place the documentation in the docs/build/html folder. You may explore these HTML files in order to preview the site.

Create a live server to automatically preview changes

There is another nox command that will do the above, and also create a live server that watches your source files for changes, and auto-builds the website any time a change is made.

To start this live server, use the following nox command:

$ nox -s docs-live

When the build is finished, go to the URL that is displayed. It should show a live preview of your documentation.

To stop serving the website, press Ctrl-C in your terminal

Build the documentation manually

To build the documentation manually, follow these steps:

First, install the miniconda Python distribution.

Next, navigate to the /docs directory and create a conda environment:

conda env create -f environment.yml

Activate the environment:

source activate jupyter_docs

Build the docs using Sphinx with the following commands:

make clean
make html

The docs will be built in build/html. They can be viewed by opening build/html/index.html or starting an HTTP server and navigating to 0.0.0.0:8000 in your web browser.

python3 -m http.server

Releasing the jupyter metapackage

Anyone with push access to this repo can make a release of the Jupyter metapackage (this happens very rarely). We use tbump to publish releases.

tbump updates version numbers and publishes the git tag of the version. Our GitHub Actions then build the releases and publish them to PyPI.

The steps involved:

1. Install tbump: pip install tbump
2. Tag and publish the release tbump $NEW_VERSION.

That's it!

jupyter_events's Issues

Use `yaml.safe_dump` and `_load`?

Using the default load and dump features of pyyaml can have unintended consequences, as they can use YAML tags to execute arbitrary code.

For an application, this is great as it provides a low-code way to use complex structures.

For a library, this is kinda bad, as it means any yaml source is immediately an arbitrary code execution engine.

Likely, this would just need from yaml import safe_dump, safe_load.

Additionally, when reading yaml, using read_text(encoding="utf-8") is a good way to avoid future issues.

jsonschema dependency

In schema.py you import from jsonschema.protocols import Validator for the purposes of adding a type hint. As far as I can see, this is the only reason why you need to depend on jsonschema >=4. Unfortunately, the community has not been speedy on adopting jsonschema >=4 in many, many places (for instance here and here).

I was wondering if it makes sense to remove the type hint or at least have something like the following, and then drop the dependency for jsonschema back down to 3.x?

try:
    from jsonschema.protocols import Validator
except ImportError:
    from typing import Any
    Validator = Any

While not ideal, the way it is currently means that there is a lot of packages that cannot co-exist with jupyter in the conda ecosystem right now.

pyyaml dependency

After completing the effort for #58 (via #59), this now exposes another issue with kfp's capping policy. In this case, they are capping pyyaml<6 (with a floor >=5.3). For applications using both Jupyter Server (which pulls in jupyter_events) and KFP, this generates an impossible resolution.

Similar to #59, is it possible to decrease the floor of pyyaml to >=5.3?

After doing so in my env (i.e., pip install pyyaml==5.3), both pytests and pre-commit succeed.

I'll go ahead and submit a PR and we can discuss it further.

Feature request: add modifier API to mutate an event before emitting

The modifier API would provide a hook for extension authors to modify an event in-place before it is validated and emitted.

A modifier would a be callable that would take the raw event data, modify it in-place, and return the modified version.

def modifier(event_data: dict) -> dict:
    ...

The immediate use-case for the modifier API would be to apply "redactors" to the EventLogger that remove properties from the event before they were emitted.

The EventLogger would need a new method to add these modifiers, e.g.

class EventLogger(...):

    ...

    def add_modifier(self, modifier: Callable[[dict], dict]):
        # Adds modifier to list of modifiers
        ...

    def emit(self, ..., data: dict, ...):
        # Should modify the data in-place.
        for modifier in self._modifiers:
            data = modifier(data)

        # Validate after. Ensures that validators return a valid schema to emit.
        self.validate(..., data)

        # Emit the event
        self.emit(...)

The public API would look something like:

def my_redactor(event_data: dict) -> dict:
    # Redact sensitive data.
    if "username" in event_data:
        event_data["username"] = "<masked>"
    return event_data


logger = EventLogger()
logger.add_modifier(my_redactor)

0.5.0: pytest is failing in two units

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix>
• run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-jupyter-events-0.5.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-jupyter-events-0.5.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/jupyter_events-0.5.0, configfile: pyproject.toml, testpaths: tests/
plugins: anyio-3.6.2, console-scripts-1.3.1, asyncio-0.20.2
asyncio: mode=auto
collected 49 items

tests/test_cli.py .......                                                                                                                                            [ 14%]
tests/test_listeners.py .....                                                                                                                                        [ 24%]
tests/test_logger.py .........F...                                                                                                                                   [ 51%]
tests/test_modifiers.py ......                                                                                                                                       [ 63%]
tests/test_schema.py ..F............                                                                                                                                 [ 93%]
tests/test_traits.py ...                                                                                                                                             [100%]

================================================================================= FAILURES =================================================================================
________________________________________________________________________ test_emit_badschema_format ________________________________________________________________________

    def test_emit_badschema_format():
        """
        Fail fast when an event doesn't conform to a specific format
        """
        schema = {
            "$id": "http://test/test",
            "version": 1,
            "type": "object",
            "properties": {
                "something": {"type": "string", "title": "test", "format": "date-time"},
            },
        }

        el = EventLogger(handlers=[logging.NullHandler()])
        el.register_event_schema(schema)

        with pytest.raises(jsonschema.ValidationError) as excinfo:
>           el.emit(schema_id="http://test/test", data={"something": "chucknorris"})
E           Failed: DID NOT RAISE <class 'jsonschema.exceptions.ValidationError'>

tests/test_logger.py:264: Failed
_______________________________________________________ test_bad_validations[bad-id.yaml-'not-a-uri' is not a 'uri'] _______________________________________________________

schema_file = 'bad-id.yaml', validation_error_msg = "'not-a-uri' is not a 'uri'"

    @pytest.mark.parametrize("schema_file,validation_error_msg", BAD_SCHEMAS)
    def test_bad_validations(schema_file, validation_error_msg):
        """
        Validation fails because the schema is missing
        a redactionPolicies field.
        """
        # Read the schema file
        with open(SCHEMA_PATH / "bad" / schema_file) as f:
            schema = yaml.loads(f)
        # Assert that the schema files for a known reason.
        with pytest.raises(ValidationError) as err:
>           validate_schema(schema)
E           Failed: DID NOT RAISE <class 'jsonschema.exceptions.ValidationError'>

tests/test_schema.py:38: Failed
=========================================================================== slowest 10 durations ===========================================================================
0.27s call     tests/test_cli.py::test_cli_version[subprocess]
0.27s call     tests/test_cli.py::test_cli_invalid[subprocess]
0.27s call     tests/test_cli.py::test_cli_good_raw[subprocess]
0.27s call     tests/test_cli.py::test_cli_good[subprocess]
0.26s call     tests/test_cli.py::test_cli_missing[subprocess]
0.26s call     tests/test_cli.py::test_cli_malformed[subprocess]
0.26s call     tests/test_cli.py::test_cli_help[subprocess]
0.01s setup    tests/test_listeners.py::test_listener_function
0.01s teardown tests/test_modifiers.py::test_modifier_function

(1 durations < 0.005s hidden.  Use -vv to show these durations.)
========================================================================= short test summary info ==========================================================================
FAILED tests/test_logger.py::test_emit_badschema_format - Failed: DID NOT RAISE <class 'jsonschema.exceptions.ValidationError'>
FAILED tests/test_schema.py::test_bad_validations[bad-id.yaml-'not-a-uri' is not a 'uri'] - Failed: DID NOT RAISE <class 'jsonschema.exceptions.ValidationError'>
======================================================================= 2 failed, 47 passed in 2.28s =======================================================================

Here is list of installed modules in build env

Package                       Version
----------------------------- -----------------
alabaster                     0.7.12
anyio                         3.6.2
appdirs                       1.4.4
argon2-cffi                   21.3.0
argon2-cffi-bindings          21.2.0
attrs                         22.1.0
Babel                         2.11.0
beautifulsoup4                4.11.1
bleach                        5.0.0
Brlapi                        0.8.3
build                         0.9.0
cffi                          1.15.1
charset-normalizer            3.0.1
click                         8.1.3
cloudpickle                   2.2.0
commonmark                    0.9.1
contourpy                     1.0.6
cssselect                     1.1.0
cycler                        0.11.0
defusedxml                    0.7.1
distro                        1.8.0
dnspython                     2.2.1
docutils                      0.19
doit                          0.36.0
editables                     0.3
entrypoints                   0.4
exceptiongroup                1.0.0
extras                        1.0.0
fastjsonschema                2.16.1
fixtures                      4.0.0
fonttools                     4.38.0
gpg                           1.17.1-unknown
hatchling                     1.11.1
html5lib                      1.1
idna                          3.4
imagesize                     1.4.1
importlib-metadata            5.1.0
importlib-resources           5.9.0
iniconfig                     1.1.1
Jinja2                        3.1.2
json5                         0.9.9
jsonschema                    4.17.3
jupyter_client                7.4.8
jupyter_core                  5.1.0
jupyter-server                1.23.3
jupyterlab-pygments           0.1.2
jupyterlab_server             2.16.3
kiwisolver                    1.4.4
libcomps                      0.1.19
louis                         3.23.0
lxml                          4.9.1
markdown-it-py                2.1.0
MarkupSafe                    2.1.1
matplotlib                    3.6.2
mdit-py-plugins               0.3.3
mdurl                         0.1.2
mistune                       2.0.4
myst-parser                   0.18.1
nbclient                      0.7.2
nbconvert                     7.2.6
nbformat                      5.7.0
nest-asyncio                  1.5.6
numpy                         1.23.1
olefile                       0.46
packaging                     21.3
pandocfilters                 1.5.0
pathspec                      0.10.2
pbr                           5.9.0
pep517                        0.13.0
Pillow                        9.3.0
pip                           22.3.1
pkgutil_resolve_name          1.3.10
platformdirs                  2.5.2
pluggy                        1.0.0
ply                           3.11
prometheus-client             0.15.0
ptyprocess                    0.7.0
pycparser                     2.21
Pygments                      2.13.0
PyGObject                     3.42.2
pyparsing                     3.0.9
pyrsistent                    0.19.2
pytest                        7.2.0
pytest-asyncio                0.20.2
pytest-console-scripts        1.3.1
python-dateutil               2.8.2
python-json-logger            2.0.4
pytz                          2022.4
PyYAML                        6.0
pyzmq                         24.0.0
requests                      2.28.1
rich                          12.6.0
rpm                           4.17.0
scour                         0.38.2
Send2Trash                    1.8.0
setuptools                    65.6.3
six                           1.16.0
sniffio                       1.2.0
snowballstemmer               2.2.0
soupsieve                     2.3.2.post1
Sphinx                        5.3.0
sphinxcontrib-applehelp       1.0.2.dev20221204
sphinxcontrib-devhelp         1.0.2.dev20221204
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1.dev20221204
sphinxcontrib-qthelp          1.0.3.dev20221204
sphinxcontrib-serializinghtml 1.1.5
terminado                     0.15.0
testtools                     2.5.0
tinycss2                      1.2.1
tomli                         2.0.1
tornado                       6.2
traitlets                     5.4.0
typing_extensions             4.4.0
urllib3                       1.26.12
webencodings                  0.5.1
websocket-client              1.4.2
wheel                         0.38.4
zipp                          3.11.0

Attribute Error for python 3.11

Trying to start jupyterlab on python 3.11 I see the following

AttributeError: module 'collections' has no attribute 'Hashable'

whole stack

python3 -m jupyterlab
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/opt/py/py/lib/python3.11/site-packages/jupyterlab/__init__.py", line 7, in <module>
    from .handlers.announcements import (  # noqa
  File "/opt/py/py/lib/python3.11/site-packages/jupyterlab/handlers/announcements.py", line 14, in <module>
    from jupyter_server.base.handlers import APIHandler
  File "/opt/py/py/lib/python3.11/site-packages/jupyter_server/base/handlers.py", line 23, in <module>
    from jupyter_events import EventLogger
  File "/opt/py/py/lib/python3.11/site-packages/jupyter_events/__init__.py", line 3, in <module>
    from .logger import EVENTS_METADATA_VERSION, EventLogger
  File "/opt/py/py/lib/python3.11/site-packages/jupyter_events/logger.py", line 19, in <module>
    from .schema_registry import SchemaRegistry
  File "/opt/py/py/lib/python3.11/site-packages/jupyter_events/schema_registry.py", line 3, in <module>
    from .schema import EventSchema
  File "/opt/py/py/lib/python3.11/site-packages/jupyter_events/schema.py", line 9, in <module>
    from .validators import draft7_format_checker, validate_schema
  File "/opt/py/py/lib/python3.11/site-packages/jupyter_events/validators.py", line 17, in <module>
    EVENT_METASCHEMA = yaml.load(EVENT_METASCHEMA_FILEPATH)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/py/py/lib/python3.11/site-packages/jupyter_events/yaml.py", line 25, in load
    return loads(data)
           ^^^^^^^^^^^
  File "/opt/py/py/lib/python3.11/site-packages/jupyter_events/yaml.py", line 15, in loads
    return yload(stream, Loader=SafeLoader)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/py/py/lib/python3.11/site-packages/yaml/__init__.py", line 72, in load
    return loader.get_single_data()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/py/py/lib/python3.11/site-packages/yaml/constructor.py", line 37, in get_single_data
    return self.construct_document(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/py/py/lib/python3.11/site-packages/yaml/constructor.py", line 46, in construct_document
    for dummy in generator:
  File "/opt/py/py/lib/python3.11/site-packages/yaml/constructor.py", line 398, in construct_yaml_map
    value = self.construct_mapping(node)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/py/py/lib/python3.11/site-packages/yaml/constructor.py", line 204, in construct_mapping
    return super().construct_mapping(node, deep=deep)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/py/py/lib/python3.11/site-packages/yaml/constructor.py", line 126, in construct_mapping
    if not isinstance(key, collections.Hashable):
                           ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'collections' has no attribute 'Hashable'

I understand the issue is likely in pyyaml, leaving this here for visibility, feel free to close

Feature Request: add a command line tool to ensure a custom schema is valid

We should provide schema authors a simple CLI tool to check that their schema is valid.

Using more JSON Schema format checkers

With the default jsonschema, many of the extended format arguments to string schema, such as uri, uuid, date, email would be silently ignored, but all seem rather relevant in an event sourcing system.

By adding a dependency on jsonschema[format] or [format-nogpl], these will actually be validated.

Additionally, custom validators can be added when the schema is instantiated: this could be useful, and more secure, than relying on the potentially dangerous YAML tags allowed by #26.

Defining event schema with well-known paths?

I've taken a look at the code and the docs here and upstream in Server, and only see reference to the python-side event registration.

For cases like a labextension, is it possible to register an event by putting a declarative file in a place at packaging time?

I could imagine two approaches:

event-schema-at-rest

{sys.prefix}/
  share/
    jupyter/
      events/
        my-custom-event.yaml

event-schema-in-traitlets

{sys.prefix}/
  etc/
    jupyter/
      jupyter_server_config.d/
        my-custom-event.json

{
  "EventLogger": {
    "extra_schemas": {
      "my-custom-event": {
        "$id": "http://event.jupyter.org/test",
        "version": 1,
        "title": "Simple Test Schema",
        "description": "A simple schema for testing",
        "type": "object",
        "properties": {
          "prop": {
            "title": "Test Property",
            "description": "Test property.",
            "type": "string"
          }
        }
      }
    }
  }
}

My preference, I guess, would be to the former, as nesting schema inside anything is kinda nightmarish, but there is a certain simplicity in keeping everything in traitlets, for good or bad.

Feature request: add API for event listeners

One way to extend Jupyter's Event system would be to enable extensions/plugins of Jupyter Applications to listen for specific events and trigger a callback function. One common use-case would be to enable an extension to emit additional event data that immediately follows the source event.

I think the easiest way to achieve this is to add a simple API to the main EventLogger object, something like:

class EventLogger:
    listeners: dict
    ...
    def add_listener(self, schema_id: str, version: int, listener) -> None:
        ...

    def emit(self, schema_id: str, version: int, data: dict) -> dict:
        ...
        # Event has already been validated and emitted
        for listener in self.listeners[(schema_id, version)]:
            listener(self, schema_id, version, data)

where the signature of the callback is :

def my_listener(logger: EventLogger, schema_id:str, version: int, data: dict) -> None:
    ...

data is the raw (validated) event data.

Version 0.7.0 is breaking jupyter lab build

Hi

I tried to run jupyter lab build in my Ubuntu 22 container - it was working on Friday, but now it stopped working, after the release of version 0.7.0 of jupyter_events

First I had this error:

0.783 Traceback (most recent call last):
0.783 File "/opt/conda/bin/jupyter-lab", line 6, in
0.783 from jupyterlab.labapp import main
0.783 File "/opt/conda/lib/python3.10/site-packages/jupyterlab/init.py", line 7, in
0.783 from .handlers.announcements import ( # noqa
0.783 File "/opt/conda/lib/python3.10/site-packages/jupyterlab/handlers/announcements.py", line 14, in
0.783 from jupyter_server.base.handlers import APIHandler
0.783 File "/opt/conda/lib/python3.10/site-packages/jupyter_server/base/handlers.py", line 23, in
0.784 from jupyter_events import EventLogger
0.784 File "/opt/conda/lib/python3.10/site-packages/jupyter_events/init.py", line 3, in
0.784 from .logger import EVENTS_METADATA_VERSION, EventLogger
0.785 File "/opt/conda/lib/python3.10/site-packages/jupyter_events/logger.py", line 18, in
0.785 from .schema import SchemaType
0.785 File "/opt/conda/lib/python3.10/site-packages/jupyter_events/schema.py", line 7, in
0.786 from referencing import Registry
0.786 ModuleNotFoundError: No module named 'referencing'

After running pip install referencing, I had the following error:

0.950 Traceback (most recent call last):
0.950 File "/opt/conda/bin/jupyter-lab", line 6, in
0.950 from jupyterlab.labapp import main
0.950 File "/opt/conda/lib/python3.10/site-packages/jupyterlab/init.py", line 7, in
0.950 from .handlers.announcements import ( # noqa
0.950 File "/opt/conda/lib/python3.10/site-packages/jupyterlab/handlers/announcements.py", line 14, in
0.950 from jupyter_server.base.handlers import APIHandler
0.950 File "/opt/conda/lib/python3.10/site-packages/jupyter_server/base/handlers.py", line 23, in
0.950 File "/opt/conda/lib/python3.10/site-packages/jupyter_events/init.py", line 3, in
0.951 from .logger import EVENTS_METADATA_VERSION, EventLogger
0.951 File "/opt/conda/lib/python3.10/site-packages/jupyter_events/logger.py", line 18, in
0.951 from .schema import SchemaType
0.951 File "/opt/conda/lib/python3.10/site-packages/jupyter_events/schema.py", line 18, in
0.951 from .validators import draft7_format_checker, validate_schema
0.951 File "/opt/conda/lib/python3.10/site-packages/jupyter_events/validators.py", line 41, in
0.951 JUPYTER_EVENTS_SCHEMA_VALIDATOR = Draft7Validator( # type: ignore

My installation of Jupyter Lab is made on miniconda, over a nvidia-cuda container, using the following libraries:
jupyterlab==3.6.3
jupyterlab-git==0.41.0
ipywidgets==8.0.7
nb_conda_kernels==2.3.1
jupyter_contrib_nbextensions==0.7.0

After uninstalling jupyter_events 0.7.0 and reinstalling 0.6.3 the build worked fine.

Published wheel has invalid metadata

The published wheel for jupyter_events 0.6.3 has an invalid line in jupyter_events-0.6.3.dist-info/METADATA:

Requires-Dist: jsonschema[format-nongpl]>=3.2.0

See pypa/hatch#801