inseefrlab / onyxia

🔬 Data science environment for k8s

Don't know Onyxia yet? Learn about the project
Looking for our public instance? datalab.sspcloud.fr
Let's get in touch! Join our Slack
What's next? Checkout our roadmap

Onyxia is a web app that aims at being the glue between multiple open source backend technologies to provide a state of art working environment for data scientists.
Onyxia is developed by the French National institute of statistics and economic studies (INSEE) and supported by the Interministerial Digital Directorate (DINUM, CodeGouvFr).

Core feature set:

• An interface for helm charts (e.g: Jupyter, RStudio) on demand on a Kubernetes cluster.
  The catalog of available images is not part of the app, you can create your own. (here is the catalog we build for the institute's needs.)
• Users can define the amount of RAM, CPU and GPU they would like to allocate to their containers.
• Specify a custom init script to be executed at launch.
• Define environment variables to be made available in the containers.
• Save and restore your service configurations.
• Deep integration with S3 for working with data (S3 as the open standard, not the AWS service) and with Vault (for secret management).
• Keycloak integration.

Media

Devoxx France 2023

KCD France 2023

Open Source Experience 2022

OW2Con'23

Energy Data Hack

OpenLAB - Download slides

Article d'acteurs publics

Les Entrepreuneurs d'intérêt général - Découvrir le programme

Contributing

If your are a new contributor, please refer to the technical documentation.

📣 Monthly Onyxia Community Calls! 📣 Starting November 2023, we're thrilled to introduce community calls on the last Friday of every month at 1pm Paris time. This is your chance to engage, ask questions, and stay updated on the newest Onyxia advancements. Don't forget to set a reminder! 📅🕐

onyxia's Issues

Keycloak migration - 8.x to 9.x

Console warning:
[KEYCLOAK] Usage of legacy style promise methods such as .error()and.success()has been deprecated and support will be removed in future versions. Use standard style promise methods such as.then() and .catch() instead.`

See keycloak doc

Username and password for rStudio service

When you want to connect to the RStudio service, it is not obvious that the username is "rstudio" and that the password can be found in the "Mon compte" section.

This should be explained somewhere on the "RStudio Sign in" form.

Use case : generate script for minio

The implementation is not clear. User could believe that they should do this configuration inside containers in the datalab. But in fact they only need this configuration for outside access because all containers are already configured

Maybe a label containing the idea "access minio from outside" should clarify where to use this manual configuration

regression on self-service details

When we access to detailed self-services, URLs are not displayed.
It's a big regression because secondary url are never display elsewhere.

Get rid of alpha / beta / stable status

See InseeFrLab/onyxia-api#15, the API will soon stop using services status so we should meld the 3 categories from the "shared services" page together.

List and document all injectable values

Thanks to #28, we should soon be able to inject user data into contracts.
We should standardize the data structure we are able to inject.
The data currently stored in redux is a good starting point but should be discussed as it's far from perfect.

A markdown file listing each variable we intend to inject and their source / description would be perfect !
This is needed to prevent people from using different keys within their catalogs.

add prometheus metrics on bucket usage

In the new release of minio, usage counts about each bucket is reported.
Available grafana graph are here :

https://grafana.lab.sspcloud.fr/d/PhCwEJkMz/minio-user?orgId=1&var-username=tm8enk

We should add a monitoring link on each bucket cards.

add formations

We should create a new top level component to list available formations for users.
All formations are simple json file like :

{ "id": "python onyxia", "title": "Python sur le datalab onyxia", "description": "Formation en python", "maintainer": "Insee", "modules": [ { "title" : "Jupyter", "description": "Decouverte de l'environnement Jupyter", "deploiement" : "https://spyrales.sspcloud.fr/my-lab/catalogue/inseefrlab-datascience/jupyter/deploiement" }, { "title" : "S3", "description": "Decouverte de S3", "deploiement" : "https://spyrales.sspcloud.fr/my-lab/catalogue/inseefrlab-datascience/jupyter/deploiement" } ] }

Idealy the list of formations come from an environnement variable.

Git avec une clef SSH associée au profil

Merci pour l'infrastructure, elle est très bien.
Est-il possible de s'ajouter une clef SSH depuis "mon compte" pour giter depuis la console dans Jupyter plus aisément ?

universe deployed apps does not render images

When a universe app is deployed by a user we should render the image url from the ONYXIA_LOGO label

beta features

Training should leave beta /advanced features.
The dynamic url in forms should integrated beta/advanced features

migration to vault kv engine v2

We should enforce a password rotation each week in order to keep them secure.
vault kv engine v1 does not allow versionning but v2 does.

We should migrate to it and renew passwork when it is too old

bandeau d'avertissement

Il serait utile de pouvoir afficher un message aux utilisateurs à partir d'une variable d'environnement sous la forme d'un bandeau en haut de chaque page.

Bug: minio is called after each change in the service form

this bug doesn't exist before a recent update

username validation

We do not enforce a username validation.
We should notify user if the username is not valid.

Add a "enable beta features" switch button

Hello,

We may need to beta test some features that are not fully production-ready.
At the moment, some of thoses features are "hidden" and people need to know the exact URL to access them.
I know of at least two of them at the moment :

• Accessing test catalogs that are sent by the API but not displayed in the UI
• Accessing the my secrets page (vault integration which is still a Work In Progress)

We could add a switch button in my account (or somewhere else ?) to let advanced users enable beta features. The beta opt-in choice would be stored client-side (for example in application storage).
Users that have opted-in would see the features without restrictions (ie my-secrets item to the menu, display the test catalogs ...), users that opted-out would not see them at all.

add support for grafana for each self service

We should use the GRAFANA_BASE_URI to give a link to the user for a dashboard of his service

User confirmation when deleting or stopping a service

When deleting or stopping a service via Onyxia, it would be nice to ask for confirmation, using a modal window for example.

images of universe package is not render

Image defined in resource.json is not render in the catalog panel.

Bug : enum type is not supported

• Steps to reproduce :

Use https://inseefrlab.github.io/Universe-Datascience/universe.json as universe.
Select vscode.
Go to vscode category.
Crash.

TypeError: Cannot read property 'map' of undefined

Handling loading state

Improve UI when app is waiting API response, displaying Loader locally.

Régression constatée au lancement du service rstudio

Après avoir créé un nouveau service rstudio, le service n'affiche plus comme auparavant l'icône d'ouverture mais celle de redémarrage (alors que le service a bien été démarré).
Voir le service de droite dans la capture ci-dessous :

Lorsqu'on rafraichit la page, on obtient bien l'icône d'ouverture :

remove of service endpoint from oyxia-api

We should create an external url to get the services.json.
This is the ui issue corresponding to InseeFrLab/onyxia-api#16

The specification could be :
`

{
    "services" : [
        {
            "title": "",
            "subtitle": "",
            "logo": "",
            "caution": "",
            "description": "",
            "link": "",
            "urls" : [
                {
                    "name": "",
                    "url": "",
                    "description": "",
                    "linked": "true"

                }
            ],

            "dashboard": "",
            "channel": "",
            "scm": "",
            "maintainer": "",
            "email": ""
        }
    ]   
}

The link value is the direct link for fast access to the services. The urls may be used to show additionnel url. If they are browsable with external explorer then we can display a link.

Bug : service name is not displayed in "my services"

See :

Not sure what's broken but something broke 😢
(ONYXIA_NAME is properly defined so it's not API's fault)

Cloudshell : inject values

At the moment, the cloudshell is deployed using default values.
We should use the same logic as traditional deployment. ie : we should inject user information.

create a button to reset user.password

We should implement a button to help user to reset password inside his vault profile

Investigate requests to /undefined

For some reason, the app makes random requests to /undefined (note that those requests are made to the webserver itself, not to the API) :

We should investigate and track down those undefined requests. It does not make the app crash but it adds some noise and overhead.

🕵️‍♂️ 🕵️‍♀️

bug with formation cards

formation cards should not show " Version : " if version is empty.

Bug : vault init is broken

Since v0.2, vault profile initialization is broken, data sent is empty.
404 means user's profile does not exist so we should create it (this part is working as intended) :

But the creation request leads to a 400 as no data is sent :

This breaks the deployment page for some packages that depend on vault-stored data. We should create a separate issue to make this error non-blocking.

env-config.js should not be cached

env-config.js is included as a script in index.html. It contains the whole app configuration which may change when the app is redeployed.
This file should not be cached (and at the moment it is as any js file would).

This can either be done client-side (apparently by appending fake get parameters to trick the browser like <script src="env-config.js?v=42" />) or server-side (by changing the nginx conf).

Not sure which one is best ?

css broken

Add version / build number

We should add the version and / or build number and display it somewhere in the UI.
We could also display the API version (see InseeFrLab/onyxia-api#42 )

Conform to the new specification

The new specification was just published : https://github.com/InseeFrLab/onyxia-api/tree/master/docs/specification
It has some changes from the previous convention (mostly keys names changes).
While the implementation server-side (and in the inseefrlab/Universe-Datascience) may still be a work in progress (see InseeFrLab/onyxia-api#11 and InseeFrLab/Universe-Datascience#4) we can start to work on it right now to be ready.

vault password is not correctly assign in universe contrat in a certain condition

if a user go directly to a package deployment page..the user password is not correctly set ..
the user saw the default one

user with no self service have a bad UX

When a user has no more self service , he lose his toolbar with the + button...
We should keep it

Fix guided visit

Since the opensourcing, the guided visit is broken.
For example, it tries to create a "draw.io" service that does not exist anymore.

fix card for services

The title of each is not display

have a button to delete all user services

On page /my-lab/mes-services we need a button that deletes all services. Of course it also needs a confirmation page.

Reinit configuration button may be too confusing

A recent and great feature generates a quicklink that users can use to deploy again later with the same configuration :

This feature comes with a button to let the user reset the configuration back to default but this button uses the same color's scheme as the "deploy service" button.
The user may be confused and click by mistake on the reset configuration button instead of the deploy button. I suggest we use something more alerting like red or yellow but any suggestion is welcome :)

Additionnaly, any design improvement and internationalization is obviously also welcome on this page 😃

externalize information for the home page

The json for the home is inside onyxia-js : public/accueil.json
We should make it possible to externalise this data to another endpoint and configurable with an environnement variable.
We could have a default one inside Onyxia-js at the same path.

If vault is unavailable, package is undeployable

See #123, we should make Onyxia more robust by making sure that packages are still deployable even if vault is unavailable.
We should probably do the same for S3 / Minio.

Feature : add S3 config export button

Hi,

It would be great to have an export feature for S3 credentials.
At the moment, users can export individual values like token, key, session token from the my account page but they have to manually configure their client.
We could add a button to export the config (copy to clipboard or download a config file) for various formats. Of the top of my head : python script, R script, .s3cmd config file ...
The UI could be dead simple like a <select> to choose the format and a button to export it.
This could be done 100% client-side.

Initialization of a user password inside vault

Whenever a user connect to onyxia-ui, we should look inside his vault and detects if it's a first connection and initializes an access password inside vault.

modify icon of the main grafana dashboard on the foot page

We should not use the grafana icon but the monitoring one also present in JS here

training ui

if a formation as too much description , the user can not see the link

JSON contract preview does not refresh

On the create service page, changing the parameters and clicking again on the preview (eye) icon does not refresh the displayed generated JSON.
The data received from the API are fine but the UI does not refresh.

Steps to reproduce :

• Go to any service to deploy (e.g : /my-lab/catalogue/sample/wordpress/deploiement)
• Fill any parameter you want
• Click eye icon : generated JSON is fine
• Change any parameter
• Click eye icon : generated JSON does not change although the new data received have changed

This may be a regression caused by 4e81d5d (moving from jsoneditor to jsoneditor-react)

create folder into S3 is not possible

We can't create folder inside a folder with onyxia-ui.
It's seem to be a double / in stead of a single in the put path

prepare release version 0.2

some bug fixes and features are needed before releasing :

• fix 'fil d'arianne' for groups
• add onyxia url and private endpoint to each description of self services
• add support for delete group
• add support for Onyxia title that help user to set a title for their self services.
• fix debug information

add git information into vault

In order to implement git facilities in the datalab we should create a git profil into vault.
First implementation should at first login :

• create two keys into the secret .onyxia/profile
  GIT_USER_NAME : initialize to the name of the user we know from auth
  GIT_USER_EMAIL : initialize to the mail of the user we know from auth

• show this two keys into the "Profil Onyxia" component

• let user modify this two keys if email and name on git is not the same as the one in our plateform.

This answers #102

generate the link of a deploiement

When a user fill the form of a self service we should let him save the link that deploy the same service with same parameter..
ex: if a user modify cpu and mem then the link would be :
https://..../....?service.cpus=1&service.mem=4096

This feature will allow more fonctionnality like :

• create link for formation for each user
• create a list of service to run fast
• prefill forms with last parameter used

all this fonction should be discuss