quantecon / sphinxcontrib-jupyter

A Sphinx Extension for Generating Jupyter Notebooks

sphinxcontrib.jupyter

A Sphinx Extension for Generating Jupyter Notebooks

Summary

This sphinx extension can be used to:

1. build a collection of jupyter notebooks,
2. execute the Jupyter notebooks,
3. convert the executed notebooks to html using nbconvert with template support.

Note: It has mainly been written to support the use case of scientific publishing and hasn't been well tested outside of this domain. Please provide feedback as an issue to this repository.

Requires: Sphinx >= 1.7.2 (for running tests).

Examples

• https://lectures.quantecon.org/ - series of lectures on quantitative economic modeling

Installation

pip install sphinxcontrib-jupyter

to get the latest version it is best to install directly by getting a copy of the repository, and

python setup.py install

if you are wishing to make changes to the project it is best to install using

python setup.py develop

Usage

Update project conf.py file to include the jupyter extension and the desired Configuration settings (see configuration section below):

extensions = ["sphinxcontrib.jupyter"]

then run

make jupyter

Usage in RST Files

The documentation is the best resource to find usage examples.

A minimum configured sphinx repo is available here which generates a sample notebook

Configuration

The documentation is the best resource for configuration settings.

Credits

This project is supported by QuantEcon

Many thanks to the contributors of this project.

• @AakashGfude
• @mmcky
• @myuuuuun
• @NickSifniotis

LICENSE

Copyright © 2019 QuantEcon Development Team: BSD-3 All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

sphinxcontrib-jupyter's Issues

Add test capability for configuration options

PR #72 will introduce the ability to compile a set of notebooks with and without a set of solutions.

However testing the configuration side is currently difficult as sphinx needs to be run based on a matrix of configuration settings. It would be nice however to have a test case for with and without solutions (i.e. two notebooks in ipynb references)

FEAT: Include test option for compiling notebooks selective with and without code tests

Thanks @jlperla for the following:

@arnavs You can take a shot at this. The basic idea is that we would introduce a :class: test as a parallel to :class: solution`

The beautiful thing now is that we can almost use identical code wherever the word solution is used in the https://github.com/QuantEcon/sphinxcontrib-jupyter/pull/72/files

In partiular,

• in sphinxcontrib/jupyter/__init__.py

app.add_config_value("jupyter_drop_tests", False, "jupyter")

• in sphinxcontrib/jupyter/writers/translate_code.py we need to parse the test class and add a flag in the visitor

self.test= _parse_class["test"]

Then in the visitor if we are in a test and the option is set to drop them, we just pass and don't output the code. i.e.

 if self.solution and self.jupyter_drop_solutions:    
            pass

• In tests/conf.py we set the option with jupyter_drop_tests = True or jupyter_drop_solutions = False
• Add tests, of course.

You can ask @mmcky but I think a lot of the other stuff in the PR also does something with output blocks, which I believe is orthogonal?

add CI for windows

had a report of issues running this on Windows platform.

• add windows CI

Development Day @ QuantEcon Labs

We are getting a couple of RA's to work on closing a few of the issues next Tuesday (21-August)

One of our main users has requested the following priorities to help in preparing Jupiter notebooks for courses.

My priorities though would be:
1. Static files: support for directory structure and correct copying
2. Slides options in meta-data
3. :class: solution

These are Issues #69, #68, #58, #22

[Cleanup] Review `generate-xml` branch

@NickSifniotis I am just cleaning up this repo to bring it up to date and get ready for further work. When you get the chance would you please remind me of the purpose for the generate-xml branch?

Should we:

• review and merge branch
• delete branch

[Cleanup] Remove `Release-0.1` branch

@NickSifniotis I think we can remove the Release-0.1 branch as PyPI uses tags when publishing source repositories. Is there a reason to keep this?

New paragraph rendering

I'm confused why this text is not formatting paragraphs correctly

Notice the sentence "Subtracting the demand"

Support for collapsible In code blocks

Sometimes code blocks can be large (for included support code for example) and it would be nice if we could make those code blocks collapsible in the Jupyter notebook.

• research collapsible blocks in Jupyter
• enable as an option in extension

Provide a minimal example of jupinx-enabled sphinx project

Hi,
I am going through the process of setting up jupinx for my favorite sphinx documents following the instructions in the README. I am encountering a few undocumented steps (e.g. having to add a jupyter dependency in the Makefile, or a jupyter_headers variable in conf.py), and now a rougher edge:

preparing documents... done
writing output... [  1%] 2009-01-01-SienaTutorials/Worksheet01-GettingHelp                                                                                                                                           
Exception occurred:
  File "/opt/sphinxcontrib-jupyter/sphinxcontrib/jupyter/writers/translate_all.py", line 258, in depart_list_item
    if self.markdown_lines[-1][-1] == "\n":
IndexError: list index out of range
The full traceback has been saved in /tmp/sphinx-err-3ac4qcm9.log, if you want to report the issue to the developers.

To move forward, it would be helpful to have a minimal example of sphinx project with jupinx enabled for testing and comparison.

Thanks in advance!

Check installation for other projects - Makefile

Once pip install sphinxcontrib-jupyter is setup then need to check if Makefile will call the jupyter builder as part of the new default Makefile format that is generated by sphinx-quickstart.

Check if we need to register jupyter as a builder and that make jupyter works.

Currently users may need to add to their Makefile a specific call to sphinx-build

Folder structure for using static assets

It would be good to have the ability to have flexible folder structures when writing collections of lectures so that static assets could be located within a single lecture folder (for example)

CLASS -> LECTURE1 -> IMG/

rather than a single unified _static folder

CLASS -> _static -> LECTURE1 -> _static

DOCS: What does jupyter_write_metadata do

Need to document what jupyter_write_metadata does when updating README

Math is smaller when enclosed in an html table

PR #47 merged back in equation numbering. However the font size used in the current test shows that math inside and outside of tables are a different size. It would be good to add in the capability to let them be the same size.

Location of static folder

Should we accept the default used by sphinx in html_static_path instead or keep our own configuration?

I suppose one use case is to use sphinx without html configured to generate the juypter notebooks.

Recycle stuff from rst2ipynb?

Dear Sphinxcontrib-jupyter devs,

I just discovered this project. Having the same deep need for a rst to ipynb conversion, we have been dreaming for a while of exactly this: a jupyter export extension for Sphinx. Thank you!!! I am now impatient to try it out and see how it fits into our documents pipeline for SageMath. See e.g
https://more-sagemath-tutorials.readthedocs.io/ .

The jupyter notebooks that are linked from each page there are produced using a hacky solution rst-to-ipynb that we implemented on top of pandoc and notedown. You may want to briefly explore it in case you find something to recycle there; e.g. the test files.

The tricky part of such a converter is how to flatten the tree structure of a RST document into the linear (code, text, code, text, ...) structure of a Jupyter notebook, without too much degradation. The above tests exercise this a bit.

Cheers,
Nicolas
cc: @slel

FEAT: Add support for providing assignments and assignments with solutions notebooks

This could be achieved by tagging code-blocks use :class: attribute. For example we could use :class: solutions to specify a solution block that could be excluded from the notebook when using make jupyter and notebooks with solutions could be generated with make juptyer --solutions for example.

ENH: Add HTML table wrapping to support display formulae in notebook

The PR #41 removed the html wrapping for display formula. This should be reintroduced within the context of labelled display formulae. This allows for a label to be applied and appropriately positioned within the markdown codeblock.

• detect difference between labelled display formulae and non-display formulae

Update README's usage instructions re: conf.py changes

Hi all,

Thanks for the project. I'm testing it out and added the extension as mentioned in usage and then typed make jupyter since I was working from the top of the README down. It wasn't until the make threw an error that I realized that I needed a bit more ;-) in the conf.py.

I'll submit a brief PR to mention this to future users. Thanks.

ENH: Support custom mathjax preamble

Custom mathjax macros can be added in html such as.

// Declare MathJax Macros for the Appropriate Macros
MathJax.Hub.Config({
  TeX: {
    Macros: {
    	Var:     "\\mathop{\\mathrm{Var}}",
    	trace:   "\\mathop{\\mathrm{trace}}",
     }
   }

Can custom mathjax macros be supported in Jupyter notebooks?

Sphinx Extension Continuous Integration and Testing ...

Need to add testing suite for the sphinx-extension. Not entirely sure how to implement this.

Option 1

Construct a sphinx environment and run make jupyter to test known conversions between input and output?

Option 2

Direct tests on node types in the sphinx builder

any ideas are welcome

Support for RISE slides?

Hi there,
Is there any way to generate https://github.com/damianavila/RISE based slides at this point? If not, it would be a great feature for generating slides for tutorials and classes.

Develop jupyter directive for full control option

Develop a custom Jupyter directive for sphinx that allows for full control of the notebook concept.

An example would be:

.. jupyter:: {{ language }}
    :type: in
    :no-execute:
    :label: xyz

or

.. jupyter:: {{ language }}
    :type: out
    :label: xyz

Potential to add:

• test support for checking output

the downside of using a custom directive would be support for other output types such as html and pdf.

ENH: Add support for LaTeX preamble in notebooks

Some notebooks in the rst2ipynb collection have a LaTeX preamble such as

$$
\def\CC{\bf C}
\def\QQ{\bf Q}
\def\RR{\bf R}
\def\ZZ{\bf Z}
\def\NN{\bf N}
$$

it would be good to add support for this in the sphinx extension. This could be useful jupyter/notebook#534

detect changes in conf.py?

Curently if a setting is changed in conf.py, sphinx doesn't clear the cache and recompile all of the notebooks. Is this a Sphinx issue or can we address this in our extension to check for changes in configuration.

Preview add-on for vscode and atom

Some of you may be hardcore vim users, but one thing that would be extremely useful for mere mortals is a simple extension for previewing the results of a workbook. Maybe something that runs the make jupyter for the folder, and then renders the result in https://nbviewer.jupyter.org/

The preview would not need to be interactive.

Rendering of footnotes in lectures

Currently appears like this:

Add a test suite

Add a test suite for the extension and setup Travis-CI.

Currently thoughts are known RST to know IPYNB files and assert that conversion is true.

Default installation Issue with setting headers ...

@NickSifniotis is this the latest version of the sphinx jupyter converter?
I seem to recall having this issue before can't remember how to fix this.

sphinxcontrib-jupyter/sphinxcontrib/jupyter/writers/translate_code.py

Line 44 in aada7d3

self.jupyter_headers['python3'] = []

Also should there be python3 here?

Add support for more advanced RST tables and output to HTML

We need to add better support for the full RST table specification. Currently the extension does not provide support for multicolumn and multirow support.

http://docutils.sourceforge.net/docs/user/rst/quickref.html#tables

As table support becomes more advanced it would be good to have the option to output to html format for inclusion in the compiled notebooks.

Configuration of jupyter_kernels

Currently Jupyter Kernels are configured using the following:

# Configure Jupyter Kernels
jupyter_kernels = {
    "python3": {
        "kernelspec": {
            "display_name": "Python",
            "language": "python3",
            "name": "python3"
            },
        "file_extension": ".py",
    },
    "julia": {
        "kernelspec": {
            "display_name": "Julia 0.6.0",
            "language": "julia",
            "name": "julia-0.6"
            },
        "file_extension": ".jl"
    }
}

Should we build this from a list of simpler options? I think this is a good approach for now as it is a complete set of attributes when defining which kernel to use.

---

@NickSifniotis (02-August-2017):

Jupiter itself must have the metadata for all its installed kernels lying around somewhere. I'd suggest pulling directly from that data, if possible.

---

@mmcky (03-August-2017):

https://github.com/jupyter/jupyter/wiki/Jupyter-kernels
also to get list of installed kernels on a local machine is jupyter kernelspec list

Add support for RST tables

Issue #772 transfer from the Quantecon.Lectures repo.

As it stands, the Jupinx extension does not support the translation of RST tables into Jupyter markdown. It would be nice if this feature was implemented.

.. NOTE:: directives rendered in markdown

Jupinx is rendering .. NOTE:: directives whereas the lectures ignore these

Example from the permanent income lecture:

Inline :index: support not supported

Currently the extension prints the :index: identifier used in sphinx in compiled notebooks.

• short term we should ignore unknown markup
• long term we should implement support for :index: markers

ENH: Enable custom width and margins in output Jupyter Notebooks

Comment from @jstac

@mmcky A differnt issue, but I feel like the text lines are too long and margins are too small. It's well known that long lines put more strain on the eyes. If you agree I'll open a separate issue.

Error rendering some LaTeX

The QuantEcon Kalman lecture renders a bit of LaTeX incorrectly. In particular, see the line "[integral] indicates the probability that we attach to to the missile being..." For some reason, the integral is being moved to the end of line after I run Jupinx.

cc: @Nosferican, @mmcky. José, if you have time, maybe see if you could replicate the issue to confirm.

math directive isn't always getting rendered properly ...

Math directives aren't always getting placed in the $$ environment

Example:

.. math::

    \mathbb P\{z = v \mid x \}
    = \begin{cases} 
        f_0(v) & \mbox{if } x = x_0, \\
        f_1(v) & \mbox{if } x = x_1
    \end{cases} 

Config option for the default language

Add a config option jupyter_default_lang, and use it instead of python3 in JupyterCodeTranslator.__init__

@ocayrol is on it :-)

ENH: Support Sphinx Aliases for Images

Currently the following does not render well

Embedded Image Handling
-----------------------

.. |mountain| image:: _static/hood.jpg
   :scale: 10 %

Here is an image that is inline and referenced using sphinx alias |mountain|.

as it get's rendered in the notebook as

• need to figure out how to determine if a directive is in the context of an alias and pass the object in the jupyter writer
• update tests/image.rst with the test case

Remove jupyter_headers from extension

No longer necessary, we have jupyter_welcome_block

Mixed use of .. code:: and ..code-block:: renders some cells as non-execute

Currently in the test document tests/code-blocks.rst

a mixed use of .. code:: python and .. code-block:: python renders the .. code-block inclusion as a no-exectute cell. This should not be the case as the rest of the document is in python. However inclusion of code-examples for other languages should be no-execute.

Allow definition for dependencies?

Should this extension provide a dependency directive which would allow users to supply a dependent file(s) for the notebook that is not shown in any notebook cells but is brought over to be included in the _static folder.

This would be useful for support data files or python modules to import functions from

TEST: Add test for welcome.rst option in conf.py file

We should add a test for PR #56

Rendering (not ignoring) TODO

Jupinx is currently rendering TODO notes in the notebooks

From dyn_stack.rst

Sphinx crashes on code block inside itemize

Thank you again for the sphinx-contrib-jupyter example; it enabled me to reduce the crash I was getting on my own rest files to a minimal non working example:

Steps to reproduce:

• Create a file foo.rst, with the following content, e.g. inside sphinxcontrib-jupyter.minimal:

- Create the permutation

   .. code-block:: python3

       1+1

• Run make jupyter

make jupyter                                                                                                            master  ✱
Running Sphinx v1.6.6
loading pickled environment... not yet created
building [mo]: targets for 0 po files that are out of date
building [jupyter]: targets for 3 source files that are out of date
updating environment: 3 added, 0 changed, 0 removed
reading sources... [100%] simple_notebook                                                                                                                                     
looking for now-outdated files... none found
pickling environment... done
checking consistency... /opt/sphinxcontrib-jupyter.minimal/Worksheet01-GettingHelp.rst: WARNING: document isn't included in any toctree
done
preparing documents... done
writing output... [ 33%] Worksheet01-GettingHelp                                                                                                                              
Exception occurred:
  File "/opt/sphinxcontrib-jupyter/sphinxcontrib/jupyter/writers/translate_all.py", line 328, in depart_list_item
    if self.markdown_lines[-1][-1] == "\n":
IndexError: list index out of range
The full traceback has been saved in /tmp/sphinx-err-boqtxia8.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Makefile:20 : la recette pour la cible « jupyter » a échouée
make: *** [jupyter] Erreur 1

sphinx-err-boqtxia8.log

Handling of static files

Line 101 in jupyter/builders/jupyter.py has "_static" hard coded in, should probably be jupyter_static_file_path insted
Moreover, it seems that the folder structure of the static file folders is not replicated in the compiled notebooks (instead, images are linked from the root directory where index resides). Example: compile rst filed that are located in subfolders and have links to static files in those subfolders.

Bug: raw html in lectures

I found this in QuantEcon's robustness lecture, which includes an embedded youtube video

DISC: Code Block Parsing behaviour for RST Files

The current implementation will treat each code-block (or equivalent directive type) as a single input cell in the generated jupyter notebook.

Support for Python doctest code blocks::

    >>> 1+1
    2
    >>> for x in range(3):
    ...      print x
    ...
    0
    1
    2
    >>> x = 1
    >>> x = 2

will get rendered as

The rst2ipynb implementation seems to parse internally to the block and split this input out and remove the associated python syntax.

The approach taken in this extension so far is to use code-blocks as discrete In cells and then we have planned to implement a :class: output option for adding output to the code-block immediately above it. (not implemented yet). Adding output as an option gives the notebook the look and feel of an executed notebook. In my view I think the discrete block approach aligns nicely with jupyter but thought I would open this discussion.

DISC: Support two modes of operation (READABLE, COMPILED)

It might be possible to support two primary modes of operation:

READABLE: This would compile notebooks that are as simple as possible, making use of markdown and rendering markdown based tables, figure inclusions etc. This is more limited as markdown does not support all features of RST

COMPILED: This would place an emphasis on compiled output which would allow for less readable jupyter notebooks in favour of supporting more features of RST+HTML which could provide for feature filled notebooks. Given this tool is built as a tool to compile between RST and IPYNB this would be most fit for the default.

[RESOURCE] ipynb as source files to sphinx

This project looks like it allows for ipynb as sources to the sphinx compilation system. It may be a useful resource when figuring out best default behaviours from rst to ipynb conversions.

Indentation with Non-Executing Code

See below for an example in the Julia Essentials lecture. It appears the no-execute flag puts an extra level of indentation on the text. Just wanted to confirm that this is desired behavior.

Code

.. code-block:: julia

    typeof(x)


An immutable object is one that cannot be altered once it resides in memory

In particular, tuples do not support item assignment:

.. code-block:: julia
    :class: no-execute

    x[1] = 42

Output

cc: @mmcky, etc.