onepanelio / onepanel

The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.

End-to-end computer vision platform

Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.

onepanel-feature-highlights.mp4

Why Onepanel?

Quick start

See quick start guide to get started.

Community

To submit a feature request, report a bug or documentation issue, please open a GitHub pull request or issue.

Contributing

Onepanel is modular and consists of multiple repositories.

See contribution guide and CONTRIBUTING.md in each repository for additional contribution guidelines.

Acknowledgments

Onepanel seamlessly integrates the following open source projects under the hood:

Argo | Couler | CVAT | JupyterLab | NNI

We are grateful for the support these communities provide and do our best to contribute back as much as possible.

License

Onepanel is licensed under Apache 2.0.

onepanel's Issues

SSH in base docker image

This will enable VSCode remote extenstion

Add Argo CronWorkflow

argoproj/argo-workflows#1758

Add endpoint to verify token

Add an endpoint to verify a token that the user provides.

This helps clients like web know if the token they provide is valid or not so they can be prompted to provide a valid token.

Make sure onepanel deployment works with https with minikube

We might be able to get around that if a host file is set

Help links need to link to real content

They are just placeholders right now.

Locations:

workflow-templates - top right
workflow-templates/create - middle right

Add CLI command for getting istio-ingressgateway external IP

Need to display the istio gateway external IP/host before displaying the URL to user. This is only if the provider is cloud:

Message would be:
Please point your Wildcard FQDN (i.e. *.example.com or *.sub.example.com) in your DNS provider to as outlined in our docs.
Once the DNS propagates, your application will be available at /

Sample kubectl command to get gateway external IP:

kubectl get service istio-ingressgateway -n istio-system

Relates to #89

System resources should only run on specific nodes

These nodes will have to either have labels or taints.

Secret not auto-generated

I ran into this while deploying on microk8s. I got to executing a workflow and then the web failed (with no error message) because the secret was not created.

I propose testing if the secret exists on core startup and if it doesn't, create it.

Thoughts @rushtehrani @aleksandrmelnikov ?

Logging

There are two types of logging:

1. Pod logs streaming from API - the limitation here is only lines are streamed, not character.
2. TTY logging - in this case we log directly from the Pod stdout so that we can do character by character logging.

We need to support 2 not only because it allows character by character logging, but it also offloads logging to containers instead of the API. Note that we still need to pull archived logs from the S3 compatible object storage

Switch Argo to PNS

See workflow executors documentation for comparison and configuration.

Centralize all metrics and logging

Add Cloudflare, AzureDNS and Google CloudDNS

https://cert-manager.io/docs/configuration/acme/dns01/

No files showing for a completed node in running workflow

Display full Kustomize errors instead of a simple error message

Metrics endpoint should return 404 when there are none

Right now the metrics endpoint returns a 500 if there are no metrics. It should return a 404.

Upgrade to Istio 1.4.6

This resolves an issue with GKE:

The issue with open source Istio version 1.4 and higher on Google Kubernetes Engine has a mitigation available as of Wednesday, 2020-01-08 14:30 US/Pacific.
A patched version of open source Istio has been released which corrects the issue with the "mixer" component creating too many watches: https://istio.io/news/releases/1.4.x/announcing-1.4.3/

Eventing

Explore eventing options:

• https://knative.dev/docs/eventing/
• https://argoproj.github.io/argo-events/

Workspace workflow requirements

Need to have these abilities in a standard way:

• Generate default workspace workflows when "Core" is deployed
• Ability to pass in system-wide configuration variables to every workflow. Examples: domain name, smtp, different types of creds (this might need to be in secret instead)
• Mount environment variables
• Ability to pause statefulsets, keeping volume intact
• Ability to switch nodes, keeping volume intact

Add parameters to API call to execute a workflow

To create/execute we need to pass in parameters.

Update the API request to take these into consideration.
JSON POST will look like

[
   {
       name1: "value1" 
   }, 
   {
       name2: "value2" 
   },    
...
   {
       nameN: "valueN" 
   }, 
]

Check Params.yaml to see which optional component is enabled

Check Params.yaml to see which optional component is enabled and update "status" command checking to include those optional components.

Enable GPU flag

For consistency, instead of relying on cloud provider deploying the NVIDIA device plugin, we need to make it available as a flag.

System environment variables

These environment variables will mount to every container to enable integrations from within the containers:

• ONEPANEL_API_URL - Format: <http-scheme>://<host>[:<apiHTTPPort>]/[apiPath]
• PROVIDER_TYPE - local or cloud

Use Kubectl package instead of calling exec.Command("kubectl")

GKE auth token first time availability and expiration

This happens in two scenarios:

1. When you get kubeconfig for newly created GKE cluster, this token is empty unless any kubectl command is run.
2. After an hour or so, the token gets stale and when you run opctl auth token, you still get the stale token.

Here's an example of how Argo addresses the second case, it is possible that this might work for first case as well: https://github.com/argoproj/argo/pull/2327/files

Local font references in UI

• Add local reference to Roboto
• Make reference to other fonts (e.g Material Icons) local as well.

test

Kustomize packaging

See https://github.com/kubeflow/manifests for reference manifests for some of these components.

Components we initially need, note that some may need to be updated:

• role related components
• system storage class for system logs/metrics - have to decide whether this should be use for Postgres or not.
• ssd storage class for workloads (provider dependent)
• EFK logging
• system metrics
• monitoring and alerts components (optional)
• cluster autoscaler (provider dependent) also this and this
• Is horizontal pod scaler built into master api?
• default network policy to limit access to onepanel namespace - note that this needs to be updated to run in onepanel namespace and we should change the label as well.
• nvidia device plugin - is this provider dependent?
• cert-manager, see this and this
• istio with specific settings
• dex
• We also need Calico with Typha, however this will already be taken care of in managed offerings like GKE, AKS and EKS

Show the URL for app once deployment is complete

The message should say:

Your application is now running at: (http|https)://(host):(uiHTTPPort|uiPath)

Help Links are wrong style when unvisited

They should be black, but are blue. They are the correct color when visited.

Argo workflow GC

Argo workflows should be cleaned up after completion and their logs and data archived.

The steps to make this work:

1. Setup S3 or equivalent store for Artifacts, see this doc
2. Set archiveLogs: true under this line. See configmap, also see argoproj/argo-workflows#1860 that fixes an issue with this flag.
3. Set Garbage collection in workflow. See this issue regarding setting global defaults for workflows.

Release a CLI build with cross platform binaries

Posix shared memory across containers in pods

Every Pod needs this added to support Posix Shared Memory across containers in pods

volumeMounts:
  - mountPath: /dev/shm
    name: dshm

volumes:
 - name: dshm
      emptyDir:
        medium: Memory
        sizeLimit: "256Mi"

Note that sizeLimit is optional and depends on: kubernetes/kubernetes#63641

Last line of logs keeps repeating when a workflow node completes

To reproduce, run the same workflow referenced in #90 and open the logs for node A. Then keep it open until the node completes.

Build Application manager image

Use this Dockerfile to build Application manager image and push to Docker Hub.

Onepanel ConfigMap and Secret

This ConfigMap will contain configuration for Onepanel's API/UI/Workflows.

CLI

Initially we need the following:

• Update params.env to include:
  • namespace - this is the default namespace that user selects to execute workflows in. This should default to default.

Once onepanel deploy is run:

• Create the user selected namespace above (note that default namespace is autogenerated)
• Create these resources after namespace is created.
• Create a ConfigMap named onepanel in onepanel namespace
  • namespace - default namespace mentioned above
  • We also need the Argo S3 config information in here: keyFormat, bucket, secretKey, accessKey, the last two referencing the secret name and keys in the user selected namespace.
• Create a Secret named onepanel in onepanel namespace
• Create a Secret named onepanel in the user selected namespace mentioned above:
  • This will contain the S3 compatible storage information.

API

• Expose this ConfigMap via an API call.

Validate token via API on login page before proceeding

Right now, any token is accepted. This can be frustrating as you login and get still 403s, so you have to log out and then back in. It'd be nice if you knew right then and there if your token was valid or not.

Relates to #96

Use Kustomize package instead of calling exec.Command("kustomize", "build")

An example of how kubectl does this:
https://github.com/kubernetes/cli-runtime/blob/master/pkg/kustomize/builder.go#L28

Move `defaultNamespace` under `application`

In its current location, it is almost always missed.

Secret management and mounting

Management

Code for reference - this only supports secret creation:
https://github.com/onepanelio/core/blob/master/api/secret.proto
https://github.com/onepanelio/core/blob/master/server/secret_server.go
https://github.com/onepanelio/core/blob/master/manager/secret_manager.go
https://github.com/onepanelio/core/blob/master/model/secret.go
https://github.com/onepanelio/core/blob/master/kube/secret.go

Mounting

Argo support for secret mounts

Display select sidecars in "Artifacts" or a dedicated section

An example sidecar would be TensorBoard. Ideally these would be Argo sidecars and we can identify whether to expose them in the UI with labels.

`opctl apply` should check pod statuses before printing that application is ready

We need to check that every pod in onepanel namespace is running. Generally if all pods in onepanel are running we should be ok, however we need to research whether we need to check every pod in istio-system and cert-manager as well as select pods in kube-system.

Certificate deployment options

Add opctl init flags to support the following scenarios:

• HTTP only
• HTTPS with external cert management
• HTTPS with cert-manager

HTTP only

Default option, In this case:

• insecure field in params.yaml is set to true
• istio ingressgateway does not redirect to HTTPS

HTTPS with external cert management

In this case:

• CLI --enable-tls flag set
• insecure field in params.yaml is set to false
• Istio ingressgateway will redirect to HTTPS

HTTPS with cert-manager

• CLI --enable-tls flag set
• CLI --enable-cert-manager flag set
• CLI --dns-provider flag is required
• Istio ingressgateway will redirect to HTTPS

Initial security considerations

• Policy to allow system namespaces only
• Namespace per project limits
• Policy to deny access to cloud provider metadata servers - may be optional.
• Setting istio-injection=enabled only on non-system namespaces.
• Disable service account automount by default, admin can override.

Add `type` to `arguments.parameters` in Argo Workflows

Support HTML input and select types for Argo Workflow YAMLs. Example:

arguments:
    parameters:
    - name: name
       value: 'workspace name'
       type: input.text
    - name: description
       value: 'workspace description'
       type: textarea.textarea
    - name: machine-type
       value: p3.2xlarge
       type: select.select
       options:
          - name: 'V100 GPU'
             value:  p3.xlarge
    - name: image-type
       value: pytorch-1.4
       type: input.radio
       options:
          -  pytorch-1.4
          -  tensorflow-2.0

We could use Gantry Field Types as reference but only need to support basic HTML fields listed above for now.

On the API side, the field types and YAML structure need to be validated, but removed before they are passed into Argo validation

Allow comments in params.yaml

Allow params.yaml to contain default comments explaining each field.

cert-manager upgrade - version TBD

Currently, cert-manager 0.13.0 is installed without the webhook. Any version >= 0.14.0 requires webhook but has this limitation.

StorageClass for Minikube

Add version command to CLI, which will report current version

Workflow UI stops updating

When you run a workflow on GKE, it seems that websocket connection gets lost and the workflow UI stops updating.

Example workflow:
wf-test.txt

Add file or folder `lookup` params

This will allow the user to select a location in object storage to lookup and display select list of files from.