galuszkak / hopsworks

Hopsworks - Data-Intensive AI platform with a Feature Store

Give us a star if you appreciate what we do

What is Hopsworks?
Quick Start
Development and Operational ML on Hopsworks
Docs
Who’s behind Hopsworks?
Open-Source
Join the community
Contribute

What is Hopsworks?

Hopsworks and its Feature Store are an open source data-intensive AI platform used for the development and operation of machine learning models at scale.

Quick Start

bash <(curl -s https://repo.hops.works/installer/latest/hopsworks-installer.sh)

Recommended minimum specification.

The Hopsworks Installer takes roughly 1-2 hrs to complete, depending on your bandwidth.

Installer Quick Start with Azure CLI or GCP CLI

bash <(curl -s https://repo.hops.works/installer/latest/hopsworks-cloud-installer.sh)

If you have the Azure or GCP CLI utilities installed (on a Linux machine), then the Hopsworks Cloud Installer (Hopsworks-cloud-installer.sh) will both install Hopsworks and provision the VMs in one command.

Hopsworks IDE Plugin

To work with the Hopsworks IDE plugin for IntelliJ/PyCharm, you can install it directly from the plugins menu of the IDE or clone it and follow the README.

Build with Maven

mvn install

Maven uses yeoman-maven-plugin to build both the front-end and the backend. Maven first executes the Gruntfile in the yo directory, then builds the back-end in Java. The yeoman-maven-plugin copies the dist folder produced by grunt from the yo directory to the target folder of the backend.

You can also build Hopsworks without the frontend (for Java EE development and testing):

mvn install -P-web

Development and Operational ML on Hopsworks

You can develop and run Python, Spark, and Flink applications on Hopsworks - in Jupyter notebooks, as jobs, or even notebooks as jobs. You can build production pipelines with the bundled Airflow, and even run ML training pipelines with GPUs in notebooks on Airflow. You can train models on as many GPUs as are installed in a Hopsworks cluster and easily share them among users.

Docs

Hopsworks documentation includes a user-guide, Feature Store documentation, and an Administrator Guide. There is also dedicated documentation for the Hopsworks Feature Store.

Hopsworks REST API is documented with Swagger and hosted by SwaggerHub.

• hopsworks-api - https://app.swaggerhub.com/apis-docs/logicalclocks/hopsworks-api
• hopsworks-ca - https://app.swaggerhub.com/apis-docs/logicalclocks/hopsworks-ca

To build and deploy swagger on your own Hopsworks instance you can follow the instructions found in this guide.

Who’s behind Hopsworks?

Hopsworks started as an open-source collaborative project at KTH University, RISE, and has more recently been taken on by Logical Clocks. Several funding bodies have helped contribute to its development including: European Commission (FP7, H2020), EIT, SSF, Vinnova and Celtic-Next.

Open-Source

Hopsworks is available under the AGPL-V3 license. In plain English this means that you are free to use Hopsworks and even build paid services on it, but if you modify the source code, you should also release your changes and any systems built around it as AGPL-V3.

Join the community

• Ask questions and give us feedback in the Hopsworks Community
• Follow us on Twitter
• Check out all our latest product releases

Contribute

We are building the most complete and modular ML platform available in the market and we count on your support to continuously improve Hopsworks. Feel free to give us suggestions, report bugs and add features to our library anytime.

We’re the best in what we do and want our community to succeed as well.
Our many thanks to the top contributors of Hopsworks!

Enjoy!