mljar / mercury

Convert Jupyter Notebooks to Web Apps

New way for visual programming!

We are working on a new way for visual programming. We developed a desktop application called MLJAR Studio. It is a notebook based development environment with interactive code recipes and managed Python environment. All running locally on your machine.

---

Build Web Apps in Jupyter Notebook

Mercury allows you to add interactive widgets in Python notebooks, so you can share notebooks as web applications. Forget about rewriting notebooks to web frameworks just to share your results. Mercury offers a set of widgets with simple re-execution of cells.

You can build with Mercury:

• Turn your notebook into beautiful Web Apps,

• Create interactive Presentations with widgets, you can recompute slides during the show,

• Share notebooks as static Websites,

• Build data-rich Dashboards with widgets,

• Create Reports with PDF exports, automatic scheduling, and email notifications (coming soon),

• Serve Python notebooks as REST API endpoints (coming soon).

Mercury features:

• add widgets with Python code - no frontend experience needed!
• hide or show the notebook's code,
• export executed notebook to PDF or HTML,
• share multiple notebooks - no limits!
• embed notebook on any website,
• easy file upload and download from the notebook,
• add authentication to notebooks (coming soon),
• schedule automatic notebook execution (coming soon).

Widgets

Mercury provides multiple widgets. There are 3 types of widgets:

• Input widgets are components that will appear in the sidebar when running the notebook in Mercury. They can be used to provide user input or trigger action in the notebook.
• Output widgets help present notebook results to the user and control execution flow.
• Custom Widgets - you can use many custom widgets, for example, PyDeck, and Pivot Tables.

Integrations

Mercury works with virtually every Python package! Among the most important are machine learning libraries such as Scikit-Learn, Pandas, and Seaborn or visualization libraries: Plotly, matplotlib, Vega-Altair, and Ipyvizzu.

Example

Simple code example that creates a widget and displays its value. You can interact with a widget in the Jupyter Notebook. Its value will be updated. However, to see the update in other cells you need to manually execute them.

Import package:

import mercury as mr

Create a Text widget:

name = mr.Text(value="Piotr", label="What is your name?")

Print widget value:

print(f"Hello {name.value}")

Code in Jupyter Notebook

Mercury App

Use Mercury to run notebook as web application. Cells are automatically re-executed after widget change. Mercury re-executes only cells with widget definition and below it. In the example, cells 2 and 3 are re-executed after widget update.

Documentation

📚 Read more about Mercury on RunMercury.com.

Installation

Compatible with Python 3.7 and higher.

Install with pip:

pip install mercury

Install with conda:

conda install -c conda-forge mercury

Demo

Run Mercury with demo notebooks.

mercury run demo

Please check 127.0.0.1:8000 to see demo notebooks.

Deployment

You have several options to deploy the notebook. You can use the self-hosted option where you use docker-compose on a VPS machine or use ngrok. There's also a possibility to use Hugging Face Spaces.

Another option is a Self-hosted commercial where you get access to the deployment dashboard where you manage notebooks and user access. In addition, you have access to user analytics; you can freely customize the style of your application. You benefit from private forks and email support.

The third option is to use Mercury Cloud. It's the easiest way to share notebooks online. You will be able to create a website with a few clicks.

Mercury with your notebooks

To run Mercury with your notebook, please execute the following:

mercury run

The command should be run in the same directory as notebooks. You can change code in Jupyter Notebook, and Mercury will instantly update web app.

Mercury License

Mercury is released with AGPL v3 license.

Looking for dedicated support, a commercial-friendly license, and more features? The Mercury Pro is for you. Please see the details at our website.

mercury's Issues

Dynamic parameter configuration

If we can add one preprocess block on top that runs only one time when notebooks is added. In this we can get some values which can be used to set configurations of parameter.

An example.

suppose we want to make "select" with choices as name of files in folder we can do something like this.

Convert Notebook to REST API

Problem

• How to reuse those models in microservices in Kubernetes (Containers)

Solution

• Reuse Mercury's capability through an API-driven interview
• Instead of a Form, users would use an API
  • Either HTTP REST/GRPC
• Container-level through Environment vars

add option to share notebooks with selected users - available for pro users

There will be an additional parameter in the YAML: share. By setting the share you can specify who can see your notebooks.

• share: public - the default available to all (open source and pro users)
• share: private - notebook available only to authenticated users (only pro)
• share: username1, username2 - notebook available only to username with names: username1 or username2 (only pro)
• share: group1, group2, username1, username2 - notebook available to all users in group1 or group2 or having a name username1 or username2 (only pro)

All permissions are for view and execute. There is no option to View only the notebook.

The authenticated users are available only to commercial (pro) users. Please check the license pricing at our website.

mercury.exe instead of mercury

During:
mercury watch

c:\programdata\anaconda3\python.exe: can't open file 'C:\Users<username>\AppData\Roaming\Python\Python38\Scripts\mercury': [Errno 2] No such file or directory

instead mercury it should look for mercury.exe

I have to copy paste mercury.exe and rename it to mercury to make it work

temporary fix:

access previous execution

Please add option to access notebooks from previous execution. The parameters values should be displayed and the notebook.

Remove error task

If there was an error it will be displayed all the time. There should be some delete button to clear the error message.

Clear celery.sqlite file after notebook watch

Hide console.log fetch notebooks

remove mercury_red_2.svg file

remove unused file and change the name of the logo file

Add delete command

There is add command to add notebook to Mercury. Please add delete command to remove unwanted notebook.

allow to define YAML prams as comment with """ or '''

add anaconda recipe

Opening / refreshing Mercury page automatically downloads `html` file

Hi @pplonski , thank you for the wonderful tool.

I built a test page and noticed that every time the page loads (in either chrome or edge) it downloads a file called html which contains the following:

<style type="text/css">
.jp-mod-noOutputs {
    padding: 0px; 
}
.jp-mod-noInput {
  padding-top: 0px;
  padding-bottom: 0px;
}
</style>

Happy to help debug or test if you cannot reproduce on your end!

I am using version: 0.3.0

Cheers,
John

Some really bad error

Notebook is executed with errors.

2022-02-17 10:45:30.028511: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2022-02-17 10:45:30.154158: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 155539800 exceeds 10% of free system memory. 2022-02-17 10:45:30.223060: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 155539800 exceeds 10% of free system memory. 2022-02-17 10:45:30.543504: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 143824140 exceeds 10% of free system memory. 2022-02-17 10:45:31.503299: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 575296560 exceeds 10% of free system memory. 2022-02-17 10:45:31.763055: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 575296560 exceeds 10% of free system memory. [NbConvertApp] ERROR | Kernel died while waiting for execute reply. Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/nbclient/client.py", line 949, in async_execute_cell exec_reply = await self.task_poll_for_reply asyncio.exceptions.CancelledError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/bin/jupyter-nbconvert", line 8, in sys.exit(main()) File "/usr/local/lib/python3.8/dist-packages/jupyter_core/application.py", line 264, in launch_instance return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs) File "/usr/local/lib/python3.8/dist-packages/traitlets/config/application.py", line 846, in launch_instance app.start() File "/usr/local/lib/python3.8/dist-packages/nbconvert/nbconvertapp.py", line 361, in start self.convert_notebooks() File "/usr/local/lib/python3.8/dist-packages/nbconvert/nbconvertapp.py", line 533, in convert_notebooks self.convert_single_notebook(notebook_filename) File "/usr/local/lib/python3.8/dist-packages/nbconvert/nbconvertapp.py", line 498, in convert_single_notebook output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer) File "/usr/local/lib/python3.8/dist-packages/nbconvert/nbconvertapp.py", line 427, in export_single_notebook output, resources = self.exporter.from_filename(notebook_filename, resources=resources) File "/usr/local/lib/python3.8/dist-packages/nbconvert/exporters/exporter.py", line 186, in from_filename return self.from_file(f, resources=resources, **kw) File "/usr/local/lib/python3.8/dist-packages/nbconvert/exporters/exporter.py", line 204, in from_file return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw) File "/usr/local/lib/python3.8/dist-packages/nbconvert/exporters/html.py", line 145, in from_notebook_node return super().from_notebook_node(nb, resources, **kw) File "/usr/local/lib/python3.8/dist-packages/nbconvert/exporters/templateexporter.py", line 376, in from_notebook_node nb_copy, resources = super().from_notebook_node(nb, resources, **kw) File "/usr/local/lib/python3.8/dist-packages/nbconvert/exporters/exporter.py", line 143, in from_notebook_node nb_copy, resources = self._preprocess(nb_copy, resources) File "/usr/local/lib/python3.8/dist-packages/nbconvert/exporters/exporter.py", line 323, in _preprocess nbc, resc = preprocessor(nbc, resc) File "/usr/local/lib/python3.8/dist-packages/nbconvert/preprocessors/base.py", line 47, in call return self.preprocess(nb, resources) File "/usr/local/lib/python3.8/dist-packages/nbconvert/preprocessors/execute.py", line 84, in preprocess self.preprocess_cell(cell, resources, index) File "/usr/local/lib/python3.8/dist-packages/nbconvert/preprocessors/execute.py", line 105, in preprocess_cell cell = self.execute_cell(cell, index, store_history=True) File "/usr/local/lib/python3.8/dist-packages/nbclient/util.py", line 84, in wrapped return just_run(coro(*args, **kwargs)) File "/usr/local/lib/python3.8/dist-packages/nbclient/util.py", line 62, in just_run return loop.run_until_complete(coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/local/lib/python3.8/dist-packages/nbclient/client.py", line 953, in async_execute_cell raise DeadKernelError("Kernel died") nbclient.exceptions.DeadKernelError: Kernel died

If the notebook is in sub folder the mercury will not accept it as valid app

Add text input

Add text input. Please remember to sanitize the input.

Add scrolling if many parameters in the sidebar

In the case of many widgets in the sidebar the Run and Donwload buttons are not available.

There should be some scroll available.

Create a youtube video showing how Mercury works

Add `demo` in `run` command

Please add the demo option in the run command that will create a demo notebook and add it to Mercury.

Add rescue if no YAML config provided

If there is no YAML code provided in the notebook then some message should be displayed and default values should be used for title and description.

create rich demo

The rich package works in the notebook - it can be a nice demo with it.

https://github.com/Textualize/rich

mercury delete & mercury list not working

Unable to use
mercury delete
mercury list

even these commands delete & list doesn't show up in help menu

https://github.com/ngoclinh8123/ngoclinh8123.github.io

add ability to change the PORT for mercury watch command

As of 0.5.1 there is no way to change the port for mercury when using watch. There are various work arounds using runserver and something external to control restarts, but this would be a nice convenience.
#50

Mercury doesn't show the executed notebook if error occurred

If there are some errors (text in stderr) during notebook execution the notebook is not displayed in the Mercury. Please display the notebook if executed.

Only case when do not display the notebook is when the output HTML file is not created.

Add support to Julia notebooks

Please add option to convert Julia notebooks into standalone web apps.

Error while writing the code in python functions

Hi,
My Jupiter notebook code looks like follows:

---
title: My first notebook
description: my first notebook on mercury
show-code: False
params:
    a:
        input: slider
        label: This is slider label
        value: 5
        min: 0
        max: 100
    b:
        label: This is numeric label
        input: numeric
        value: 5.5
        min: 0
        max: 100
        step: 1
---
a=5
b=6
def add(a,b):
    return a+b
print(add(a+b))

while running it in the web app it gives an error add is not defined.
can any one help me with how to overcome this?

add command `run`

Please add a command run that will simplify the process of deployment. The run should start server and worker. It should set some defaults for:

• ALLOWED_HOSTS=*
• SERVE_STATIC=*
• NOTEBOOKS=*.ipynb
• WELCOME=welcome.md

User still will be able to overwrite the env variables but it will get good defaults to start.

Add file input

Please create a new widget that will allow file input.

Missing cell with variables or missing variable

When there is missing cell with variables or variables definitions are not complete there should be some rescue code.

Clear all tasks on refresh in watch mode

Invalid X-Frame-Options header was found when loading .../output_1.html : ALLOWALL is not a valid directive

error message when loading iframe

Add date and time input

Please check this react package https://github.com/Hacker0x01/react-datepicker/

Error when loading packages

I am having some problems importing packages. I encounter 0 problems when running the jupyter notebook locally. I am getting errors when trying to run my notebook in the local server. Is there something that I am doing wrong or is this a general issue? I did not see any issues on the internet related to this so I wanted to raise this problem as an issue. Can you show me how to deal with this?

Reading notebook without utf-8 encoding

Hi,

Thank you for your amazing work !

I have a problem when I try to convert my notebook with mercury run, I have the following message :
Error during notebook initialization. 'charmap' codec can't decode byte 0x9d in position 3522922: character maps to

the notebook runs fine in jupyter and I have no issues ... I can't understand from where it can come from ...

Please help !

Thank you in advance.

Best Regards

Accept HOST or add to the list of hosts via a configuration file

Could there be a way to read a config file for the application run? Essentially a .env file or other configuration for ports and so forth.

add login view for pro users

The UI code for pro users will ba available in the main repo.

The backend code for pro (commercial users) is in the separate repo, available only with a license.

YAML RawNBConvert cell not detected by Mercury

Not sure if this is related, but some of the cli commands appear to not be working either. I see the same behavior on 0.3.0 and 0.5.1. Using python 3.9

add option to change home view welcome message above notebooks gallery

Right now there is a h1 Welcome message at the top of the notebooks gallery. Please add an option to specify a custom message there. Ideally, it will be nice if Mercury admin can define a markdown file (for example welcome.md) with custom text that will be displayed above the notebooks gallery.

make footer as separate component

Error when uploading file

I'm getting this error when uploading a file through the widget

Forbidden: /api/v1/fp/process/

Wiki Documentation : Deployment on LAN/Intranet

Prerequisite

Mercury is setup and accessible at 127.0.0.1:8000 (https://github.com/mljar/mercury#readme)

Background

Here our aim is to make mercury accessible within LAN. Currently it can only be accessed on the same computer it is hosted on at Localhost (127.0.0.1:8000). To achieve this please follow along. These steps are for windows Computer but same can be applied for other OS with some tweaks (for ubuntu we do not require firewall configuration discussed later)

Adding you IPV4 address to Allowed host

Get IPV4 address

On cmd

ipconfig

• Get this ipv4 address.

Firewall access

Now we have to allow firewall to accept incoming connection to mercury port (8000 by default)

• Go To windows firewall settings by search. (Control Panel\System and Security\Windows Defender Firewall)

• Click on Advanced settings in left tab
  

• Click on inbound Rules
  

• Click on add New Rule
  

• Select Port from the menu
  

• Input port number on which you want mercury to be hosted (Default is 8000)
  
  

• Provide some name and Finish
  

Bind your IP with mercury instead localhost

We can not access 127.0.0.1 from other computer, we can only access it with it's ip-address once we have access to the computer
port number will help us access the process we are looking for. For us port is 8000 (for process mercury)

• Set environment variable to add our ipaddress (where the mercury is hosted) in ALLOWED_HOSTS.

set ALLOWED_HOSTS=192.168.29.186

-To start mercury on specific ip use below command which running mercury.

mercury runserver <yourIPaddress:portNumber> --runworker

mercury runserver  192.168.29.186:8000 --runworker

Voila you are done! now you can access it from anywhere on LAN

mercury emedding

check the way to embed mercury framework in any website - it will be super useful if users can embed interactive notebooks on any website.

An issue for text input

There is a problem with text input. It does not accept some symbols such as =, ?, >, ! etc. Can you change the character map for the text widget? I have to use these symbols.

Installation error : Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools

Used pip install mljar-mercury inside an anaconda command line
Got error for psutil
Error message below

1. building 'psutil._psutil_windows' extension
   error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

2. I Installed C++ 14 and now i get this error
   c:\program files\python310\include\pyconfig.h(200): fatal error C1083: Cannot open include file: 'basetsd.h': No such file or directory
   error: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe' failed with exit code 2

Add the same notebook should update the notebook instead of duplicate

Right now if we run twice the following command:

mercury add notebook_1.ipynb

We will have two notebooks as apps in the Mercury. Please do not add the duplicate, just update the current notebook. By update I mean:

• use the old ID
• run nbconvert on the new notebook
• display a message that notebook is updated (not added)

Precise URL of watched notebook app

Please display (in the terminal after running a command) to the user a precise URL address of watched notebook.

TemplateDoesNotExist at / error

I got TemplateDoesNotExist at / error. Both the code in github main repo and pypi repo have same error. It may be related to 0.3.0 release. Can you check it out please? This is an internal server error related django.

Add option to create files for download

Please add feature to create a directory that will allow to download all files from it.

Example:

title: My app
description: App with file download
params:
    output_dir:
        output: dir

# %% next cell
output_dir = "output/directory/path"

# %% next cell
import os
with open(os.path.join(output_dir, "my_file.txt"), "w") as fout:
    fout.write("This is a test")

In the above example the file my_file.txt should be available to download in the Mercury app. Different users should have different output folders.

Forbidden: /api/v1/execute/31

Forbidden: /api/v1/execute/31
[04/Feb/2022 09:04:59] "POST /api/v1/execute/31 HTTP/1.1" 403 58

Running in GCP AI Platform, This is happening when we are editing some form value and clicking on Run,

PS : Added one extra IP in Allowed Host to run in GCP AI Platform