datarootsio / databooks Goto Github PK

A CLI tool to reduce the friction between data scientists by reducing git conflicts removing notebook metadata and gracefully resolving git conflicts.

Home Page: https://databooks.dev/

License: MIT License

Python 83.63% Jupyter Notebook 16.37%

cli jupyter-notebook helper-tool pydantic typer rich git

databooks's People

Contributors

Stargazers

Watchers

Forkers

boaarmpit frenzymadness astronautas adrien-debray

databooks's Issues

tests don't work when run without git repository

When packaging software for Fedora, we run tests in an unpacked archive without initialized git repository for the codebase because we are testing the installed version. Would it make sense to make the tests compatible with this way of running them?

Maybe the tests can initialize their own repositories in a tmpdir or something like that.

The current log I have is:

python -m pytest
===================================== test session starts =====================================
platform linux -- Python 3.10.7, pytest-7.1.3, pluggy-1.0.0
rootdir: /home/lbalhar/Software/databooks
collected 118 items                                                                           

tests/test_affirm.py ......................                                             [ 18%]
tests/test_cli.py ..........FF.FFF.F.F.FF                                               [ 38%]
tests/test_common.py ..                                                                 [ 39%]
tests/test_conflicts.py .........                                                       [ 47%]
tests/test_git_utils.py F.                                                              [ 49%]
tests/test_metadata.py ..........                                                       [ 57%]
tests/test_recipes.py ...............                                                   [ 70%]
tests/test_tui.py ..................                                                    [ 85%]
tests/test_data_models/test_base.py .                                                   [ 86%]
tests/test_data_models/test_notebook.py ................                                [100%]

========================================== FAILURES ===========================================
__________________________________________ test_meta __________________________________________

tmpdir = local('/tmp/pytest-of-lbalhar/pytest-1/test_meta0')

    def test_meta(tmpdir: LocalPath) -> None:
        """Remove notebook metadata."""
        read_path = tmpdir.mkdir("notebooks") / "test_meta_nb.ipynb"  # type: ignore
        TestJupyterNotebook().jupyter_notebook.write(read_path)
    
        nb_read = JupyterNotebook.parse_file(path=read_path)
        result = runner.invoke(app, ["meta", str(read_path), "--yes"])
        nb_write = JupyterNotebook.parse_file(path=read_path)
    
>       assert result.exit_code == 0
E       assert 1 == 0
E        +  where 1 = <Result InvalidGitRepositoryError()>.exit_code

tests/test_cli.py:50: AssertionError
______________________________________ test_meta__check _______________________________________

tmpdir = local('/tmp/pytest-of-lbalhar/pytest-1/test_meta__check0')
caplog = <_pytest.logging.LogCaptureFixture object at 0x7fca4a63baf0>

    def test_meta__check(tmpdir: LocalPath, caplog: LogCaptureFixture) -> None:
        """Report on existing notebook metadata (both when it is and isn't present)."""
        caplog.set_level(logging.INFO)
    
        read_path = tmpdir.mkdir("notebooks") / "test_meta_nb.ipynb"  # type: ignore
        TestJupyterNotebook().jupyter_notebook.write(read_path)
    
        nb_read = JupyterNotebook.parse_file(path=read_path)
        result = runner.invoke(app, ["meta", str(read_path), "--check"])
        nb_write = JupyterNotebook.parse_file(path=read_path)
    
        logs = list(caplog.records)
        assert result.exit_code == 1
>       assert len(logs) == 1
E       assert 0 == 1
E        +  where 0 = len([])

tests/test_cli.py:83: AssertionError
______________________________________ test_meta__script ______________________________________

tmpdir = local('/tmp/pytest-of-lbalhar/pytest-1/test_meta__script0')

    def test_meta__script(tmpdir: LocalPath) -> None:
        """Raise `typer.BadParameter` when passing a script instead of a notebook."""
        py_path = tmpdir.mkdir("files") / "a_script.py"  # type: ignore
        py_path.write_text("# some python code", encoding="utf-8")
    
        result = runner.invoke(app, ["meta", str(py_path)])
>       assert result.exit_code == 2
E       assert 1 == 2
E        +  where 1 = <Result InvalidGitRepositoryError()>.exit_code

tests/test_cli.py:139: AssertionError
____________________________________ test_meta__no_confirm ____________________________________

tmpdir = local('/tmp/pytest-of-lbalhar/pytest-1/test_meta__no_confirm0')

    def test_meta__no_confirm(tmpdir: LocalPath) -> None:
        """Don't make any changes without confirmation to overwrite files (prompt)."""
        nb_path = tmpdir.mkdir("notebooks") / "test_meta_nb.ipynb"  # type: ignore
        TestJupyterNotebook().jupyter_notebook.write(nb_path)
    
        result = runner.invoke(app, ["meta", str(nb_path)])
    
        assert result.exit_code == 1
        assert JupyterNotebook.parse_file(nb_path) == TestJupyterNotebook().jupyter_notebook
>       assert result.output == (
            "1 files will be overwritten (no prefix nor suffix was passed)."
            " Continue? [y/n]: \nAborted!\n"
        )
E       AssertionError: assert '' == '1 files will... \nAborted!\n'
E         - 1 files will be overwritten (no prefix nor suffix was passed). Continue? [y/n]: 
E         - Aborted!

tests/test_cli.py:155: AssertionError
_____________________________________ test_meta__confirm ______________________________________

tmpdir = local('/tmp/pytest-of-lbalhar/pytest-1/test_meta__confirm0')

    def test_meta__confirm(tmpdir: LocalPath) -> None:
        """Make changes when confirming overwrite via the prompt."""
        nb_path = tmpdir.mkdir("notebooks") / "test_meta_nb.ipynb"  # type: ignore
        TestJupyterNotebook().jupyter_notebook.write(nb_path)
    
        result = runner.invoke(app, ["meta", str(nb_path)], input="y")
    
>       assert result.exit_code == 0
E       assert 1 == 0
E        +  where 1 = <Result InvalidGitRepositoryError()>.exit_code

tests/test_cli.py:168: AssertionError
_________________________________________ test_assert _________________________________________

caplog = <_pytest.logging.LogCaptureFixture object at 0x7fca4a66dff0>

    def test_assert(caplog: LogCaptureFixture) -> None:
        """Assert that notebook has sequential and increasing cell execution."""
        caplog.set_level(logging.INFO)
    
        exprs = (
            "[c.execution_count for c in exec_cells] == list(range(1, len(exec_cells) + 1))"
        )
        recipe = "seq-increase"
        with resources.path("tests.files", "demo.ipynb") as nb_path:
            result = runner.invoke(
                app, ["assert", str(nb_path), "--expr", exprs, "--recipe", recipe]
            )
    
        logs = list(caplog.records)
>       assert result.exit_code == 0
E       assert 1 == 0
E        +  where 1 = <Result InvalidGitRepositoryError()>.exit_code

tests/test_cli.py:203: AssertionError
__________________________________________ test_fix ___________________________________________

tmpdir = local('/tmp/pytest-of-lbalhar/pytest-1/test_fix0')

    def test_fix(tmpdir: LocalPath) -> None:
        """Fix notebook conflicts."""
        # Setup
        nb_path = Path("test_conflicts_nb.ipynb")
        notebook_1 = TestJupyterNotebook().jupyter_notebook
        notebook_2 = TestJupyterNotebook().jupyter_notebook
    
        notebook_1.metadata = NotebookMetadata(
            kernelspec=dict(
                display_name="different_kernel_display_name", name="kernel_name"
            ),
            field_to_remove=["Field to remove"],
            another_field_to_remove="another field",
        )
    
        extra_cell = BaseCell(
            cell_type="raw",
            metadata=CellMetadata(random_meta=["meta"]),
            source="extra",
        )
        notebook_2.cells = notebook_2.cells + [extra_cell]
        notebook_2.nbformat += 1
        notebook_2.nbformat_minor += 1
    
        git_repo = init_repo_conflicts(
            tmpdir=tmpdir,
            filename=nb_path,
            contents_main=notebook_1.json(),
            contents_other=notebook_2.json(),
            commit_message_main="Notebook from main",
            commit_message_other="Notebook from other",
        )
    
        conflict_files = get_conflict_blobs(repo=git_repo)
        id_main = conflict_files[0].first_log
        id_other = conflict_files[0].last_log
    
        # Run CLI and check conflict resolution
        result = runner.invoke(app, ["fix", str(tmpdir)])
>       fixed_notebook = JupyterNotebook.parse_file(path=tmpdir / nb_path)

tests/test_cli.py:268: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
databooks/data_models/notebook.py:250: in parse_file
    return super(JupyterNotebook, cls).parse_file(
pydantic/main.py:561: in pydantic.main.BaseModel.parse_file
    ???
pydantic/parse.py:64: in pydantic.parse.load_file
    ???
pydantic/parse.py:37: in pydantic.parse.load_str_bytes
    ???
/usr/lib64/python3.10/json/__init__.py:346: in loads
    return _default_decoder.decode(s)
/usr/lib64/python3.10/json/decoder.py:337: in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <json.decoder.JSONDecoder object at 0x7fca4be6b1f0>
s = '<<<<<<< HEAD\n{"nbformat": 4, "nbformat_minor": 4, "metadata": {"field_to_remove": ["Field to remove"], "another_fiel...xecution_count": 1}, {"metadata": {"random_meta": ["meta"]}, "source": "extra", "cell_type": "raw"}]}\n>>>>>>> other\n'
idx = 0

    def raw_decode(self, s, idx=0):
        """Decode a JSON document from ``s`` (a ``str`` beginning with
        a JSON document) and return a 2-tuple of the Python
        representation and the index in ``s`` where the document ended.
    
        This can be used to decode a JSON document from a string that may
        have extraneous data at the end.
    
        """
        try:
            obj, end = self.scan_once(s, idx)
        except StopIteration as err:
>           raise JSONDecodeError("Expecting value", s, err.value) from None
E           json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

/usr/lib64/python3.10/json/decoder.py:355: JSONDecodeError
__________________________________________ test_show __________________________________________

    def test_show() -> None:
        """Show notebook in terminal."""
        with resources.path("tests.files", "tui-demo.ipynb") as nb_path:
            result = runner.invoke(app, ["show", str(nb_path)])
>       assert result.exit_code == 0
E       assert 1 == 0
E        +  where 1 = <Result InvalidGitRepositoryError()>.exit_code

tests/test_cli.py:382: AssertionError
____________________________________ test_show_no_multiple ____________________________________

    def test_show_no_multiple() -> None:
        """Don't show multiple notebooks if not confirmed in prompt."""
        with resources.path("tests.files", "tui-demo.ipynb") as nb:
            dirpath = str(nb.parent)
    
        # Exit code is 0 if user responds to prompt with `n`
        result = runner.invoke(app, ["show", dirpath], input="n")
>       assert result.exit_code == 0
E       assert 1 == 0
E        +  where 1 = <Result InvalidGitRepositoryError()>.exit_code

tests/test_cli.py:457: AssertionError
________________________________________ test_get_repo ________________________________________

    def test_get_repo() -> None:
        """Find git repository."""
        curr_dir = Path(__file__).parent
>       repo = get_repo(curr_dir)

tests/test_git_utils.py:54: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
databooks/git_utils.py:38: in get_repo
    repo = Repo(path=repo_dir)
../../.virtualenvs/databooks/lib/python3.10/site-packages/git/repo/base.py:266: in __init__
    self.working_dir: Optional[PathLike] = self._working_tree_dir or self.common_dir
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <git.repo.base.Repo ''>

    @property
    def common_dir(self) -> PathLike:
        """
        :return: The git dir that holds everything except possibly HEAD,
            FETCH_HEAD, ORIG_HEAD, COMMIT_EDITMSG, index, and logs/."""
        if self._common_dir:
            return self._common_dir
        elif self.git_dir:
            return self.git_dir
        else:
            # or could return ""
>           raise InvalidGitRepositoryError()
E           git.exc.InvalidGitRepositoryError

../../.virtualenvs/databooks/lib/python3.10/site-packages/git/repo/base.py:347: InvalidGitRepositoryError
=================================== short test summary info ===================================
FAILED tests/test_cli.py::test_meta - assert 1 == 0
FAILED tests/test_cli.py::test_meta__check - assert 0 == 1
FAILED tests/test_cli.py::test_meta__script - assert 1 == 2
FAILED tests/test_cli.py::test_meta__no_confirm - AssertionError: assert '' == '1 files will...
FAILED tests/test_cli.py::test_meta__confirm - assert 1 == 0
FAILED tests/test_cli.py::test_assert - assert 1 == 0
FAILED tests/test_cli.py::test_fix - json.decoder.JSONDecodeError: Expecting value: line 1 c...
FAILED tests/test_cli.py::test_show - assert 1 == 0
FAILED tests/test_cli.py::test_show_no_multiple - assert 1 == 0
FAILED tests/test_git_utils.py::test_get_repo - git.exc.InvalidGitRepositoryError
=============================== 10 failed, 108 passed in 0.60s ================================

Unable to keep "kernelspec"

I would like to remove metadata and keep kernelspec via pre-commit. I tried the args --cell-meta-keep=kernelspec but does not work.

Add plugin system

Restructure project to allow plugins - this way databooks becomes the base models (+core functionalities) and anyone can manipulate it for their use cases (ie.: run command)

(comments welcome 😊 )

[RFE] databooks run

I like the way databooks is able to show the content of the notebook. It might make sense to implement a run command which can run all cells in a notebook and show the progress in the CLI interface cell by cell. We don't need to implement the execution because it can be done by nbconvert or nbclient packages.

What do you think?

pre-commit hook invalid value

Using pre-commit hook generates the following error:

databooks................................................................Failed
- hook id: databooks
- exit code: 2

Usage: databooks meta [OPTIONS] PATHS...

Error: Invalid value: No prefix nor suffix were passed. Please specify `--overwrite` or `-w` to overwrite files.

this is the .pre-commit-config.yaml

repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.1.0
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer
    -   id: check-yaml
    -   id: check-added-large-files
-   repo: https://github.com/psf/black
    rev: 22.1.0
    hooks:
    -   id: black-jupyter
        language_version: python3.9
-   repo: https://github.com/datarootsio/databooks
    rev: 0.1.15
    hooks:
    -   id: databooks

"Dynamic" recipes

Make recipes take variables so that they could be reused more easily.

Example:

The recipe:

max_cells = RecipeInfo(
    src="len(nb.cells) < amount_max",
    description="Assert that there are less than 'amount_max' cells in the notebook.",
)

The command:

databooks assert path/to/nbs --recipe max_cells --amount_max 10

databooks assert path/to/nbs --recipe max_cells --help

Would return the arguments of that recipe

RPM package in Fedora - problem with Typer version

Hi!

I really enjoyed your talk on PyCon PL today and I've decided that it'd be great to have databook packaged in Fedora Linux. I've tried to create a package but found a problem - we have too new type in Fedora (version 0.6.1) but databook requires version < 0.5.

databooks/pyproject.toml

Line 20 in 0b1d3e6

typer = "^0.4.0"

Would it make sense to make the requirement less strict? If thy are following semantic versioning, something like <1.0.0 should guarantee enough stability.

consider adding flake8-docstrings to ci

https://pypi.org/project/flake8-docstrings/

Databooks minimal python version to 3.7

Currently, the project only support python 3.8+

From a personal experience, I wanted to include databooks to my development environment but stumbled on dependency issues with default Sagemaker images which are still 3.7.

There are currently still a lot of projects that are being developed in 3.7, which wouldn't be able to include databooks, that is such a shame and a lost opportunity!
The sunset of Python 3.7 is also still far enough away in the future (27 Jun 2023).

pytest 7.2.0 no longer depends on py

This will no longer work with pytest 7.2.0 and higher. py lib is deprecated but you can fix the problem by requiring it explicitly if there is no other way now.

databooks/tests/test_common.py

Line 3 in 6febdba

from py._path.local import LocalPath

add example on how to strip metadata in github action

consider workflow:

user comits notebook to gb
gh actions runs that triggers metadata cleanup and commits to repo

move to pydantic v2

Add/test Python 3.12

Now that Python 3.12 is out, we should

Add it to the CI to run the tests
Remove/add imports that are 3.12 specific

Related to #50

If diff is only on code/output, then only show the split for that section
Show when only cell/notebook metadata changes - maybe make cell border yellow, etc...
For cell source diffs, show lines/character changes by changing the background color (red/green)
Add rich rendering for other outputs
- HTML (other than tables)
- PNG
- SVG
- ...

Add python 3.11 to list of supported versions

Python 3.11 is out! 🚀
Add it to CICD tests, check for updates (tomlib vs tomli, ...), etc.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.