nteract / commuter

🚎 Notebook sharing hub

the interactive computing suite for you

nteract is an open-source organization committed to creating fantastic interactive computing experiences that allow people to collaborate with ease.

We build SDKs, applications, and libraries that help you and your team make the most of interactive (particularly Jupyter) notebooks and REPLs.

To learn more about the nteract open source organization and the rest of our projects, please visit our website.

What's in this repo?

This repo is a monorepo. It contains the code for the nteract core SDK and nteract's desktop and web applications. It also contains the documentation for the SDK and the applications. Here's a quick guide to the contents of the monorepo.

Folder	Description
applications/desktop	Source code for the nteract desktop application. The desktop application is a cross-platform app built using Electron.
applications/jupyter-extension	Source code the nteract Jupyter extension. This extension can be installed alongside Jupyter classic and JupyterLab in your Jupyter deployments or personal Jupyter server.
packages	JavaScript packages that are part of the nteract core SDK.
changelogs	Changelogs for each release of the nteract core SDK and applications.

How do I contribute to this repo?

If you are interested in contributing to nteract, please read the contribution guidelines for information on how to set up your nteract repo for development, how to write tests and validate changes, how to update documentation, and how to submit your code changes for review on GitHub.

How do I use the nteract core SDK?

If you are a developer who wants to build an nteract-based notebook application, check out the following documentation resources for more info.

Link	What's in it?
docs.nteract.io	This page contains our how-to and tutorial style documentation. Get started learning about the nteract core SDK here.
packages.nteract.io	This page contains the API documentation for packages in our core SDK. Bookmark this and use it as a reference when building your nteract-based UI.
components.nteract.io	This page contains the documentation for our suite of composable React components. It contains code samples that you can reference when building your nteract-based UI.

Our documentation is living. We are always making changes and adding more content. If you have feedback about the documentation, please open an issue in this repo. If you are interested in submitting a change to our documentation page, please review the contribution guidelines and submit a pull request.

How do I use the nteract desktop application?

To get started with the nteract desktop app, head over to the nteract homepage to download the application for your operating system.

Once you've download the app, head over to our documentation page for tutorials and guides on using the app for your data analysis and science workflows.

Supporting nteract

nteract is a non-profit open-source organization fiscally sponsored by NumFOCUS. If you are interested in supporting development on nteract, please consider making a recurring donation.

Development on nteract is also supported by the following organizations.

commuter's Issues

Development flow broken at head

When I follow the developer instructions for commuter I get the error below. The error first appears with the Oct 2nd update, which leads me to believe this is a problem with updating packages.

I rolled back the Sept 11th update and everything works. I also confirmed that this is not only my computer.

Error:
error in ../frontend/icons/octicons.js
Module build failed: TypeError: .../octicons.js: Cannot read property '0' of null

Ability to run commuter on a url prefix

e.g https://host.tld/commuter/<path_to_ipynb>

Prefix in this case will be /commuter/

npm run build:frontend takes forever*

Ok, not forever. Long enough that it's well past the timeout on Heroku, and, at least locally, I haven't seen finish directly.

[bug] Author field not getting populated on the Discover UI

Check Demo for NYC taxi cab example

I couldn't get the NYC taxi cab example to load on the Heroku demo. Not sure if it was huge latency or not finding something. Other examples worked fine.

Password / Authentication ?

Is it possible to lock down the notebook via password

Heroku deployment failing

The Heroku pipeline has been failing since Lerna 2.0.0-rc.1 was brought in.

       Lerna v2.0.0-rc.1
       Independent Versioning Mode
       This is not a git repository, did you already run `git init` or `lerna init`?
       
       npm ERR! Linux 3.13.0-112-generic
       npm ERR! argv "/tmp/build_6b2b2babbbb3cc8f0a897e4e530ce41d/nteract-commuter-61cfbaf/.heroku/node/bin/node" "/tmp/build_6b2b2babbbb3cc8f0a897e4e530ce41d/nteract-commuter-61cfbaf/.heroku/node/bin/npm" "run" "lerna"
       npm ERR! node v6.9.5
       npm ERR! npm  v3.10.10
       npm ERR! code ELIFECYCLE
       npm ERR! @nteract/[email protected] lerna: `lerna bootstrap`
       npm ERR! Exit status 1
       npm ERR!
       npm ERR! Failed at the @nteract/[email protected] lerna script 'lerna bootstrap'.
       npm ERR! Make sure you have the latest version of node.js and npm installed.
       npm ERR! If you do, this is most likely a problem with the @nteract/commuter-client package,
       npm ERR! not with npm itself.
       npm ERR! Tell the author that this fails on your system:

It looks like lerna is not pleased with the fact that it isn't a git repo by the time it reaches Heroku.

Top level way to hide all and show all code

User story: want to link to a notebook document as a standalone report -- an interactive data memo. To facilitate this, we want a way to hide and show code quickly.

Users also want:

• availability as a URL param ("so I can share links that open a clean notebook without code cells")
• ability to toggle it per cell (in the live view)

frontend build error with octicons.js

➜  commuter git:(master) node --version
v8.9.0
➜  commuter git:(master) npm --version
5.5.1
➜  commuter git:(master) git rev-parse HEAD
484d26e83d5d1a25001c0d21566c39b8f49cea88

npm run dev

 DONE  Compiled successfully in 1998ms                                                                                16:08:11

[Wed Nov 01 2017 16:08:11 GMT+0800 (CST)] INFO Node env: development
[Wed Nov 01 2017 16:08:11 GMT+0800 (CST)] INFO Commuter server listening on port 4000
> Building page: /


 DONE  Compiled successfully in 1189ms                                                                                16:08:22

::1 - - [01/Nov/2017:08:08:22 +0000] "GET / HTTP/1.1" 302 -
> Building page: /view


 ERROR  Failed to compile with 1 errors                                                                               16:08:28                                                                          

 error  in ../frontend/icons/octicons.js

Module build failed: TypeError: /home/*/Repos/github/nteract/commuter/packages/frontend/icons/octicons.js: Cannot read property '0' of null
    at Buffer._append (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/buffer.js:95:25)
    at Buffer.append (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/buffer.js:72:10)
    at Generator._append (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/printer.js:206:52)
    at Generator.word (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/printer.js:131:10)
    at Generator.Identifier (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/generators/types.js:38:8)
    at /home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/printer.js:298:23
    at Buffer.withSource (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/buffer.js:168:5)
    at Generator.withSource (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/printer.js:189:15)
    at Generator.print (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/printer.js:297:10)
    at Generator.MemberExpression (/home/*/Repos/github/nteract/commuter/packages/frontend/node_modules/babel-generator/lib/generators/expressions.js:216:8)

 @ ../frontend/components/contents/directory-listing.js 19:16-47
 @ ../frontend/components/contents/index.js
 @ ../frontend/pages/view.js?entry
 @ multi ../frontend/pages/view.js?entry

Run flow and prettier eslint on Travis

It would be great to see flow and prettier get checked on travis.

Note: only the commuter-server has flow setup.

Limit large files

We can only render files of a certain size with reliability. Some users end up having large CSVs that they shouldn't try to render on here (yet). For now, there should be a cap. Related: #93.

commuter screen shots

Custom CSS

There are a few levels of CSS that we should think about configuring:

1. At the notebook level, be able to set CSS for the notebook (in metadata), as opposed to outputting it in a cell:

metadata: {
  nteract: {
    css: {
      url: "someCustom.css"
    }
  }
}

2. Some users have voiced that they would like to see their own styling across the entire deployment of notebooks so it matches their personal style. Perhaps this means a toggle for now with our current themes, while allowing them to in the future set up a custom theme.

npm run start script doesn't restart failed processes

If the server dies, the client stays running and you have to restart the entire thing.

Return standard jupyter API response when "directory" not found

Right now when S3 "directories" are not found they return an empty content directory when they should return:

{
  "reason": null,
  "message": "No such file or directory: someName"
}

Conformance Testing

Based on #30, we need to make sure our API is consistent and correct. At the very least we need to make sure our contents API is 100% compatible with the Jupyter Contents API.

Large files proxied may need additional handling

When opening nyc_taxi.ipynb (from the demo) now, this error message is logged to the console:

[0] ::ffff:127.0.0.1 - - [19/Feb/2017:08:26:06 +0000] "GET /api/contents/ HTTP/1.1" 304 -
[0] /Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/request.js:31
[0]             throw err;
[0]             ^
[0]
[0] SyntaxError: Unexpected end of JSON input
[0]     at Object.parse (native)
[0]     at Response.s3.getObject (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/src/services/s3.js:70:30)
[0]     at Request.<anonymous> (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/request.js:358:18)
[0]     at Request.callListeners (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
[0]     at Request.emit (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
[0]     at Request.emit (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/request.js:671:14)
[0]     at Request.transition (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/request.js:22:10)
[0]     at AcceptorStateMachine.runTo (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/state_machine.js:14:12)
[0]     at /Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/state_machine.js:26:10
[0]     at Request.<anonymous> (/Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-server/node_modules/aws-sdk/lib/request.js:38:9)
[1] Proxy error: Could not proxy request /api/contents//nyc_taxi.ipynb from 127.0.0.1:3000 to http://localhost:4000.
[1] See https://nodejs.org/api/errors.html#errors_common_system_errors for more information (ECONNRESET).
[1]
[0]
[0] npm
[0]  ERR! Darwin 16.4.0
[0] npm ERR! argv "/Users/kylek/.nvm/versions/node/v6.9.2/bin/node" "/Users/kylek/.nvm/versions/node/v6.9.2/bin/npm" "run" "server"
[0] npm ERR! node v6.9.2
[0] npm ERR! npm  v4.0.5
[0] npm ERR! code ELIFECYCLE
[0] npm ERR! @nteract/[email protected] server: `./src/index.js server`
[0] npm ERR! Exit status 1
[0] npm ERR!
[0] npm ERR! Failed at the @nteract/[email protected] server script './src/index.js server'.
[0] npm ERR! Make sure you have the latest version of node.js and npm installed.
[0] npm ERR! If you do, this is most likely a problem with the @nteract/commuter-cli package,
[0] npm ERR! not with npm itself.
[0] npm ERR!
[0]  Tell the author that this fails on your system:
[0] npm ERR!     ./src/index.js server
[0] npm ERR! You can get information on how to open an issue for this project with:
[0] npm ERR!     npm bugs @nteract/commuter-cli
[0] npm ERR! Or if that isn't available, you can get their info via:
[0] npm ERR!     npm owner ls @nteract/commuter-cli
[0] npm ERR! There is likely additional logging output above.
[0]
[0] npm ERR! Please include the following file with any support request:
[0] npm ERR!     /Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-cli/npm-debug.log
[0] npm run server exited with code 1

Other notebooks load just fine. Since I know that the nyc_taxi notebook is huge (due to it embedding BokehJS + data, quadruple encoded in many places), it's likely that S3 is expecting us to get the data in chunks (this is my hunch at least). As an aside to this, we'll want to prevent fully exiting/crashing during errors like these (unless we're going to set up some forever for this handling...)

Ensure server is built before launching with `start`

If the server package hasn't had a fresh build, whatever old version in lib is the one that runs on npm run start.

Add tests for react components

And use jest

Create a default rendered view when the content is not known

Right now users end up at a blank page, we should give them at least a <pre>. Need some way of dealing with binary files though. 🤔

Docker ?

I can see Commuter being used as a docker server, while running a commuter instance from within a docker container. If there isn't a plan in place already i'd be happy to contribute a Pull Request, that installs, and runs commuter via docker.

Please let me know your thoughts.

Include nteract fonts on notebook view

Should be as easy as adding:

    <link href='https://fonts.googleapis.com/css?family=Source+Code+Pro:400,700,300,200,500,600,900' rel='stylesheet' type='text/css'>
    <link href='https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,200,200italic,300,300italic,400italic,600,600italic,700,700italic,900,900italic' rel='stylesheet' type='text/css'>

to the index.html

Configure Redux Devtools

Allow deploying commuter on a basepath

In order to deploy commuter as a service for JupyterHub (and generally on the configurable HTTP proxy), we need the ability to run commuter on a prefix.

/cc @minrk @cabhishek

Symlink Handling

When commuter comes across a symlink, whether it's a file or directory, it fails. It also fails hard, not returning any results on a directory listing. If a symlink exists and we have access to what it resolves to, we should support it.

Niceties for Zeppelin notebook URLs

• Make the URL be /YAZXC1425/note.json#Name-of-zeppelin-notebook
• Within the directory listing show links for zeppelin/${ID}/note.json as the name of the Zeppelin notebook.

Render other content

• HTML - do it in an iframe (#88)
• Images
• RMarkdown - convert to an in-memory notebook (?)

Display last_modified and other metadata on file view

Especially for notebooks, it would be nice to know when the file was last modified.

Paths that don't exist should 404

Right now it shows an empty listing, it should say 🙅 .

install on master not working

$ npm i

> @nteract/[email protected] install /Users/kylek/code/src/github.com/nteract/commuter
> npm run lerna && npm run build


> @nteract/[email protected] lerna /Users/kylek/code/src/github.com/nteract/commuter
> lerna bootstrap

Lerna v2.0.0-beta.36
Independent Versioning Mode
Bootstrapping 5 packages
Preinstalling packages
Installing external dependencies
NpmUtilities.installInDir     ("./packages/commuter-cli", ["copy-dir@^0.3.0"])
Command exited with status 1: npm install copy-dir@^0.3.0
npm ERR! Darwin 15.6.0
npm ERR! argv "/Users/kylek/.nvm/versions/node/v6.9.2/bin/node" "/Users/kylek/.nvm/versions/node/v6.9.2/bin/npm" "install" "copy-dir@^0.3.0"
npm ERR! node v6.9.2
npm ERR! npm  v4.0.5
npm ERR! code E404

npm ERR! 404 Not found : @nteract/commuter-directory-listing
npm ERR! 404
npm ERR! 404  '@nteract/commuter-directory-listing' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 It was specified as a dependency of '@nteract/commuter-client'
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/kylek/code/src/github.com/nteract/commuter/packages/commuter-cli/npm-debug.log

Errored while running BootstrapCommand.execute
Command exited with status 1: npm install copy-dir@^0.3.0

Display section numbers

As reported by @jeffwong, commuter doesn't display section numbering like the classic notebook does.

Admittedly, neither does the nteract app since we're using commonmark which has no opinionated stance on sections. We should probably do some mangling of the AST to take a stance on the header links.

/cc @mpacer who's done a bunch of thinking on this

Reduce margins for notebook display

Right now it's putting notebooks in the middle and not using whitespace well.

Non-notebook files should adhere to the contents API

Right now the API is trying to JSON.parse all files when it should only decode notebook content. Here are the expected responses for a few files:

/api/contents/trial.json

{
  "type": "file",
  "name": "trial.json",
  "format": "text",
  "created": "2017-03-07T21:36:34.692957Z",
  "path": "trial.json",
  "last_modified": "2017-03-07T21:36:34.692957Z",
  "content": "{\n    \"stuff\": \"cool\"\n}",
  "writable": true,
  "mimetype": "application/json"
}

/api/contents/test.html

{
  "type": "file",
  "name": "test.html",
  "format": "text",
  "created": "2017-03-07T21:19:44.792556Z",
  "path": "test.html",
  "last_modified": "2017-03-07T21:19:44.792556Z",
  "content": "<h1>Hey</h1>\n\n<p>Is this real life?</p>",
  "writable": true,
  "mimetype": "text/html"
}

Designing the Search API

CC @parente and @mariusvniekerk who were interested in this.

Are there current search APIs we'd want borrow from for this?

Handle inline maths in Markdown

Notebooks with embedded maths in Markdown aren't rendering on commuter.

Persist notebooks back to S3

Right now notebooks can only be listed from S3. Implement notebooks saving feature in a way that's compatible with notebooks content API using rx-jupyter

Bring in the plotly renderer

Otherwise commuter tries to render the embedded plotly HTML which tries to use requirejs.

Notebooks failing to load

commit used to reproduce the errors:

Directory listing works:

Notebook fails:

Render source code with a syntax highlighted view

For .js, .py, and all the others, use CodeMirror or something lightweight for syntax highlighting.

Hide certain files

In particular, don't show .ipynb_checkpoints, maybe other hidden files.

Ability to search for notebooks?

Greetings,

This may already be documented somewhere , but it it possible to search for a notebook.
As a project or instance grows, it may make it easier to share amongst a larger team if one could search for a notebook they're looking for.

Create a "file-based" discovery backend

Allow users to configure COMMUTER_DISCOVERY_FILE that is a JSON document used to respond to /v1/api/discovery. Alternatively, they could put it on S3 in their bucket, maybe at the base path?

Local Storage Backend

Right now we have an S3 backend. In order to support EFS and other NFS based storage options, we'll want another backend.

Turn off the search/replace from the Editor

The CodeMirror find/replace is currently hijacking the entire Find view if a user clicks inside the cell. Oops!

Make iframe for HTML viewer more seamless

In #88, I used absolute positioning and left: 10rem to put the iframe in place. I'd like to see if there's a better way to make the iframe take up the rest of the page while being positioned like our other renderers (ok, the one notebook renderer).

Organized collections

Something I've noted from others who are familiar with products like Tableau -- even if an overall organization is sharing notebooks all over, their suborganizations each want to share collections.

In an almost nonfictional example, imagine an org* is split across several product lines, disciplines or customer segments, each with their own staff of data scientists, statisticians, and data engineers:

• Latin America
• Asia
• Europe
• Marketing
• Infrastructure Operations

* Not completely non-fictional, I've worked at several companies that were large enough to split across lines like these

Error Handling

Bits for cleanup:

• Show loading state / progress
• When there is no discovery backend, API should possibly return

{
  results: [],
  error: "Message about the discovery backend not being connected"
}

• Handle S3 timeouts, errors, etc.
• Any other miscellaneous errors need to be logged as well as displayed for users

CSV Rendering

• Render .csv files.
• Provide data resource renderer from nteract/nteract
• Render the zeppelin tsv format with the above CSV renderer

Support auth via aws_access_key_id & aws_secret_access_key

Implementation details here

Right now only IAM_ROLE auth works.

Stream big data over `/files/` instead of creating a massive string

When a file is large enough we hit a limit in calling Buffer.toString() for the contents response. We should handle the contents API call with no response and expect to use the streaming /files/ API to get larger data sizes.