Support creating interactive viz and data apps about jupyterlite

[jtpio]

Looks like it is indeed missing from:

Opened https://github.com/jtpio/jupyterlite/pull/113 to add it to the main JupyterLite application.

[jtpio]

Gotcha. I'm happy to add another check if it helps :) we want Plotly to render automatically in as many places as possible.

With #171 now in, looks like Plotly now works out of the box without having to explicitly set the renderer: https://github.com/jtpio/jupyterlite/pull/220

[lrowe]

To the same end, it would be nice to support altair. I'm able to install altair with the following, but it does not display the chart:

import micropip
# Work around https://github.com/pyodide/pyodide/issues/1614 which is now fixed in pyodide
await micropip.install('Jinja2')
micropip.PACKAGE_MANAGER.builtin_packages['jinja2'] = micropip.PACKAGE_MANAGER.builtin_packages['Jinja2']
# Last version of jsonschema before it added the pyrsistent dependency (native code, no wheel)
await micropip.install("https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl")
await micropip.install("altair")
import sys
sys.setrecursionlimit(255)

import altair
import pandas
source = pandas.DataFrame({
    'a': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I'],
    'b': [28, 55, 43, 91, 81, 53, 19, 87, 52]
})
altair.Chart(source).mark_bar().encode(
    x='a',
    y='b'
)

I tried installing ipyvega with await micropip.install('vega') but it didn't help. Perhaps the javascript @jupyterlab/vega5-extension needs to be installed?

[jtpio]

Thanks @MarcSkovMadsen and @lrowe.

Support for ipywidgets (https://github.com/jtpio/jupyterlite/issues/18) should already help getting more interactive visualization libraries to work in JupyterLite. In theory most should (almost) work out of the box when the Comms are implemented.

Perhaps the javascript @jupyterlab/vega5-extension needs to be installed?

Looks like it is indeed missing from:

https://github.com/jtpio/jupyterlite/blob/2b900e958020fa68d75ec7e2250ae48f5b53f91c/app/lab/package.json#L173

JupyterLab ships it by default here:

https://github.com/jupyterlab/jupyterlab/blob/a230896fc5efcb916b4542b3457701ade7c5d805/dev_mode/package.json#L251

So we could do the same in JupyterLite and it might just do the trick.

[psychemedia]

I had a go at installing folium and it had several unmet-by-default dependencies, including markupsafe, which does not have a universal wheel available on PyPi via pip/micropip.

(With dependencies met, I could install and view an interactive map using folium.)

Would it be useful to have a simple recipe or Github Action for forcing a universal pip wheel build and pushing the wheel to a wheelhouse that can be used to load universal wheels into the pyodide kernel?

Simple wheelhouses can be published by Github Pages easily enough, eg as per https://github.com/parkin/python-wheelhouse/tree/gh-pages .

[jtpio]

Thanks @psychemedia.

(With dependencies met, I could install and view an interactive map using folium.)

Does that mean you were able to find a workaround to display a folium map? Or that if we could install all the dependencies, then we could display folium maps in JupyterLite?

[psychemedia]

@jtpio I managed to get folium running by installing the dependencies myself. I suspect it might have been smoother using the recipe at the top of this thread than the approach of installing each dependency in turn using a simple micropip call, one per requirement, in separate cells (await just threw errors for me?). FWIW, I put the missing markupsafe dependency here and installed from that. When I get a chance, I use the recipe at the top of this post to create a proper worked example.

[jtpio]

Nice, thanks for sharing 👍

If you get a chance, would you mind adding a new example notebook to the examples folder? https://github.com/jtpio/jupyterlite/tree/main/examples

Even if these micropip calls are required for now, it's still nice to have that documented somewhere.

Thanks!

[psychemedia]

Will do.. FWIW, a couple of other things that I think could make like easier for folk who can't build a distro:

• a template repo with a Github Pages ready to go environment;
• simple instruction on how / where to add their own "distributed" notebooks;
• a really simple way (maybe a requirements style file) to add their own packages that need installing into the pyodide envt for their notebooks to run cleanly. (FWIW, I found I had most success importing from a wheel URL found via a PyPi Download files page (eg https://pypi.org/project/folium/#files ) where the trick is to look for the -py2.py3-none-any.whl package and then grab that wheel link URL.)

[psychemedia]

I was just trying a requests call to an https location and it returns a SSLError("Can't connect to HTTPS URL because the SSL module is not available.") error. Is that likely to be caused by SSL support missing lower down, eg in pyodide?

[RESOLVED: https://github.com/pyodide/pyodide/issues/1585 ]

EXAMPLE resolution (raw):

from requests import Request
import pyodide
import json

p = Request('GET', "https://nominatim.openstreetmap.org/search",
            params={'q': address, 'format':'json'}).prepare()

json.loads(pyodide.open_url(p.url).read())

# Install requests and its dependencies
import micropip
await micropip.install('https://files.pythonhosted.org/packages/05/1b/0a0dece0e8aa492a6ec9e4ad2fe366b511558cdc73fd3abc82ba7348e875/certifi-2021.5.30-py2.py3-none-any.whl')
await micropip.install('https://files.pythonhosted.org/packages/19/c7/fa589626997dd07bd87d9269342ccb74b1720384a4d739a1872bd84fbe68/chardet-4.0.0-py2.py3-none-any.whl')
await micropip.install('https://files.pythonhosted.org/packages/0c/cd/1e2ec680ec7b09846dc6e605f5a7709dfb9d7128e51a026e7154e18a234e/urllib3-1.26.5-py2.py3-none-any.whl')
await micropip.install('https://files.pythonhosted.org/packages/d7/77/ff688d1504cdc4db2a938e2b7b9adee5dd52e34efbd2431051efc9984de9/idna-3.2-py3-none-any.whl')
await micropip.install('https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl')

#The requests library makes it easy to call URLs using Python
import requests

address='Open University, Walton Hall, Milton Keynes, MK7 6AA, UK'
#Construct a request to the Nominatim geolocatio service using our desired address
r= requests.get("https://nominatim.openstreetmap.org/search", params={'q': address, 'format':'json'})
r.json()



Traceback (most recent call last):
  File "/lib/python3.8/site-packages/urllib3/connectionpool.py", line 688, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/lib/python3.8/site-packages/urllib3/connectionpool.py", line 280, in _get_conn
    return conn or self._new_conn()
  File "/lib/python3.8/site-packages/urllib3/connectionpool.py", line 979, in _new_conn
    raise SSLError(
urllib3.exceptions.SSLError: Can't connect to HTTPS URL because the SSL module is not available.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/lib/python3.8/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Max retries exceeded with url: /search?q=Open+University%2C+Walton+Hall%2C+Milton+Keynes%2C+MK7+6AA%2C+UK&format=json (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available."))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<console>", line 2, in <module>
  File "/lib/python3.8/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/lib/python3.8/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Max retries exceeded with url: /search?q=Open+University%2C+Walton+Hall%2C+Milton+Keynes%2C+MK7+6AA%2C+UK&format=json (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available."))
PythonError: Traceback (most recent call last):
  File "/lib/python3.8/site-packages/urllib3/connectionpool.py", line 688, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/lib/python3.8/site-packages/urllib3/connectionpool.py", line 280, in _get_conn
    return conn or self._new_conn()
  File "/lib/python3.8/site-packages/urllib3/connectionpool.py", line 979, in _new_conn
    raise SSLError(
urllib3.exceptions.SSLError: Can't connect to HTTPS URL because the SSL module is not available.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/lib/python3.8/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Max retries exceeded with url: /search?q=Open+University%2C+Walton+Hall%2C+Milton+Keynes%2C+MK7+6AA%2C+UK&format=json (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available."))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 282, in __step
    result = coro.throw(exc)
  File "/lib/python3.8/site-packages/pyolite/interpreter.py", line 23, in run
    result = await self.run_complete
  File "/lib/python3.8/asyncio/futures.py", line 260, in __await__
    yield self  # This tells Task to wait for completion.
  File "/lib/python3.8/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/lib/python3.8/asyncio/futures.py", line 178, in result
    raise self._exception
  File "/lib/python3.8/asyncio/tasks.py", line 280, in __step
    result = coro.send(None)
  File "/lib/python3.8/site-packages/pyodide/console.py", line 253, in load_packages_and_run
    raise e
  File "/lib/python3.8/site-packages/pyodide/console.py", line 247, in load_packages_and_run
    result = await eval_code_async(
  File "/lib/python3.8/site-packages/pyodide/_base.py", line 419, in eval_code_async
    return await CodeRunner(
  File "/lib/python3.8/site-packages/pyodide/_base.py", line 270, in run_async
    coro = eval(mod, self.globals, self.locals)
  File "<console>", line 2, in <module>
  File "/lib/python3.8/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/lib/python3.8/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='nominatim.openstreetmap.org', port=443): Max retries exceeded with url: /search?q=Open+University%2C+Walton+Hall%2C+Milton+Keynes%2C+MK7+6AA%2C+UK&format=json (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available."))

    at new_error (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.js:9:175413)
    at wrap_exception (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[297]:0x1a6c94)
    at FutureDoneCallback_call (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[379]:0x1aaab3)
    at byn$fpcast-emu$FutureDoneCallback_call (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[14420]:0x7a257c)
    at _PyObject_MakeTpCall (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[760]:0x1cabed)
    at __static_1198 (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[2906]:0x2c54f1)
    at byn$fpcast-emu$__static_1198 (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[22640]:0x7bf866)
    at __static_678 (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[1621]:0x21220e)
    at byn$fpcast-emu$__static_678 (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[22120]:0x7bdce6)
    at PyVectorcall_Call (https://cdn.jsdelivr.net/pyodide/v0.17.0/full/pyodide.asm.wasm:wasm-function[763]:0x1cb006)

[jtpio]

Support is still new / experimental, but there is now an example notebook for Plotly:

https://jupyterlite.readthedocs.io/en/latest/_static/retro/notebooks/?path=examples/pyolite%20-%20plotly.ipynb

[nicolaskruchten]

We just released version 5.0 of plotly, which switches from depending on retrying to tenacity, which as I understand it should make it easier to make plotly work with Pyodide? See plotly/plotly.py#2907 ... Please let me know if there's some way I can help :)

[jtpio]

Thanks @nicolaskruchten for the heads-up!

https://github.com/jtpio/jupyterlite/pull/164 updates plotly to the latest version, and it seems to be working well:

[nicolaskruchten]

Awesome! It'd be great for our renderer system to auto-detect that it's running in JupyterLite... what kind of heuristic could we use? Here's where we do the auto-detection right now: https://github.com/plotly/plotly.py/blob/master/packages/python/plotly/plotly/io/_renderers.py#L448

[jtpio]

what kind of heuristic could we use?

Ideally JupyterLite should fall in the same category as the other notebook frontends, which seems to be here in the code:

https://github.com/plotly/plotly.py/blob/24cda54b22b7c541a30fd6a080e4ccf5a0684106/packages/python/plotly/plotly/io/_renderers.py#L519-L523

For now ipython does not work yet in JupyterLite (but might), which could explain why setting the renderer explicity is required for now.

[nicolaskruchten]

Gotcha. I'm happy to add another check if it helps :) we want Plotly to render automatically in as many places as possible.

[bollwyvl]

closing as:

• all the python/renderer dependency stuff is pretty much "solved" (as much as it can be at this point) or can be by "normal means" extensions
• #148 provides a concrete path forward on "apps" (e.g. a single cell output)