ttung / nbencdec

Tool to encode/decode .ipynb into .py files.

NoteBook ENCode DECode

Rationale

Storing .ipynb files in code repositories has two significant drawbacks:

1. Most, if not all existing IDEs will not index .ipynb files. Therefore, if one wishes to refactor code that impacts .ipynb, it often has to be done manually.
2. It is challenging to code review .ipynb files well.

At the same time, ipython notebooks have significant usability advantages over plain python source files. nbencdec allows users to convert .ipynb into a .py file with enough hints embedded in comments such that it is possible to reconstruct the original structure of the .ipynb file.

Usage

• Install the nbencdec package.
  • Option 1: Clone the repo, and run pip install -e .
  • Option 2: run pip install nbencdec.
• Run nbencdec encode <ipynb file> <output .py file> to encode an .ipynb file as a .py file.
• Run nbencdec decode <py file> <output .ipypy file> to decode a properly encoded .py file.

Hints

There are four types of hints embedded in the comments:

1. # EPY: stripped_notebook: <json>: This represents the json structure of the original .ipynb file, except for the cells field. See the ipython file format specification for more details.
2. # EPY: START code and # EPY: END code. Lines in between these two markers should be incorporated into a single code cell in the .ipynb file.
3. # EPY: START markdown and # EPY: END markdown. Lines in between these two markers should be incorporated into a single code cell in the .ipynb file.
4. # EPY: ESCAPE. The remaining text on that line should be written verbatim to the .ipynb file. This is used for ipython magic commands, which would not be valid python.

nbencdec's Issues

Unit tests

Use the notebooks here and here to build some unit tests.

empty code cells when decoding .py -> .ipynb should trigger a warning

This can't convert properly because nbconvert is dropping empty cells when reading notebooks. jupyter/nbconvert#720

jinja2.exceptions.TemplateNotFound: encoded_python.tpl

nbencdec encode attention_model_validation_experiments.ipynb han.py

Traceback (most recent call last):
File "C:\Users\Jason\miniconda3\Scripts\nbencdec-script.py", line 33, in
sys.exit(load_entry_point('nbencdec', 'console_scripts', 'nbencdec')())
File "f:\nlp_projects\nbencdec\nbencdec\cli\commands.py", line 124, in main
args.command(args)
File "f:\nlp_projects\nbencdec\nbencdec\cli\commands.py", line 97, in encode
encoded, _ = encoder.from_file(input_fh)
File "C:\Users\Jason\miniconda3\lib\site-packages\nbconvert\exporters\exporter.py", line 199, in from_file
return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
File "C:\Users\Jason\miniconda3\lib\site-packages\nbconvert\exporters\templateexporter.py", line 384, in from_notebook_node
output = self.template.render(nb=nb_copy, resources=resources)
File "C:\Users\Jason\miniconda3\lib\site-packages\nbconvert\exporters\templateexporter.py", line 148, in template
self._template_cached = self._load_template()
File "C:\Users\Jason\miniconda3\lib\site-packages\nbconvert\exporters\templateexporter.py", line 355, in _load_template
return self.environment.get_template(template_file)
File "C:\Users\Jason\miniconda3\lib\site-packages\jinja2\environment.py", line 883, in get_template
return self._load_template(name, self.make_globals(globals))
File "C:\Users\Jason\miniconda3\lib\site-packages\jinja2\environment.py", line 857, in _load_template
template = self.loader.load(self, name, globals)
File "C:\Users\Jason\miniconda3\lib\site-packages\jinja2\loaders.py", line 429, in load
raise TemplateNotFound(name)
jinja2.exceptions.TemplateNotFound: encoded_python.tpl