leohmoraes / datalab

Interactive tools and developer experiences for Big Data on Google Cloud Platform.

Google Cloud DataLab

Google Cloud DataLab provides a productive, interactive, and integrated tool to explore, visualize, analyze and transform data, bringing together the power of Python, SQL, JavaScript, and the Google Cloud Platform with services such as BigQuery and Storage.

DataLab builds on the interactive notebooks, and the foundation of Jupyter (formerly IPython) to enable developers, data scientists and data analysts to easily work with their data from exploration to developing and deploying data pipelines, all within notebooks.

DataLab deeply integrates into Google Cloud Platform to allow users to extract insights and harness the full value of their data. It provides a secure environment for members of a cloud project to effortlessly access data and resources accessible from the project, and to share notebooks via git.

You can see an example of the notebooks by browsing through the samples and documentation, which are themselves written in the form of notebooks.

Using DataLab and Getting Started

DataLab is packaged as a docker container which contains Jupyter/IPython, and a variety of python libraries such as numpy, pandas, scikit-learn and matplotlib, in a ready-to-use form.

You can run the docker container locally or in GCE, as described in the wiki.

Contacting Us

For support or help using DataLab, please submit questions tagged with google-cloud-datalab on StackOverflow.

For any product issues, you can either submit issues here on this project page, or you can submit your feedback using the feedback link available within the product.

Developing DataLab

Building and Running

The wiki describes the process of setting up a local development environment, as well as the steps to build and run, and the developer workflow.

Contributing

Contributions are welcome! Please see our roadmap page. Please check the page on contributing for more details.

You can always contribute even without code submissions by submitting issues and suggestions to help improve DataLab and building and sharing samples and being a member of the community.

Testing

Please take a look at the test directory for instructions on how to run tests locally.

Repository Overview

This is a quick description of the repository structure to help understand and discover the relevant pieces.

All source code corresponding to product functionality that is built exists within /sources. The following is a list of the individual components:

• /sources/lib - set of python libraries used to implement APIs to access Google Cloud Platform services, and implement the DataLab interactive experience.

  • api: Google Cloud Platform APIs (currently: BigQuery and Cloud Storage).
  • datalab: interactive notebook experience to plug into Jupyter and IPython.

• /sources/web - the DataLab web server. This is implemented in node.js and serves the DataLab front-end experience - both content and APIs, as well as backend infrastructure such as notebook source control. Some of the requests are proxied to the Jupyter notebook server, which manages notebooks and associated kernel sessions.

• /sources/tools - miscellaneous other supporting tools.

Source code builds into the /build directory, and the generated build outputs are consumed when building the DataLab docker container.

The build outputs are packaged in the form of a docker container.

• /containers/datalab - the only container for now. This is the container that is used as the DataLab AppEngine module.