Coder Social home page Coder Social logo

lminzhw / volcano Goto Github PK

View Code? Open in Web Editor NEW

This project forked from volcano-sh/volcano

0.0 0.0 0.0 56.78 MB

A Kubernetes native system for high performance workload, e.g. AI/ML, BigData, HPC

Home Page: http://volcano.sh

License: Apache License 2.0

Makefile 0.38% Shell 33.56% Go 65.37% Dockerfile 0.68%

volcano's Introduction

volcano-logo


Build Status Go Report Card RepoSize Release LICENSE

Volcano is system for runnning high performance workloads on Kubernetes. It provides a suite of mechanisms currently missing from Kubernetes that are commonly required by many classes of high performance workload including:

  1. machine learning/deep learning,
  2. bioinformatics/genomics, and
  3. other "big data" applications.

These types of applications typically run on generalized domain frameworks like Tensorflow, Spark, PyTorch, MPI, etc, which Volcano integrates with.

Some examples of the mechanisms and features that Volcano adds to Kubernetes are:

  1. Job management extensions and improvements, e.g:
    1. Multi-pod jobs
    2. Lifecycle management extensions including suspend/resume and restart.
    3. Improved error handling
    4. Indexed jobs
    5. Task dependencies
  2. Scheduling extensions, e.g:
    1. Co-scheduling
    2. Fair-share scheduling
    3. Queue scheduling
    4. Preemption and reclaims
    5. Reservartions and backfills
    6. Topology-based scheduling
  3. Runtime extensions, e.g:
    1. Support for specialized continer runtimes like Singularity, with GPU accelerator extensions and enhanced security features.
  4. Other
    1. Data locality awareness and intelligent scheduling
    2. Optimizations for data throughput, round-trip latency, etc.

Volcano builds upon a decade and a half of experience running a wide variety of high performance workloads at scale using several systems and platforms, combined with best-of-breed ideas and practices from the open source community.

Overall Architecture

volcano

Quick Start Guide

The easiest way to deploy Volcano is to use the Helm chart.

Pre-requisites

First of all, clone the repo to your local path:

# mkdir -p $GOPATH/src/volcano.sh/
# cd $GOPATH/src/volcano.sh/
# git clone https://github.com/volcano-sh/volcano.git

1. Volcano Image

Official images are available on DockerHub, however you can build them locally with the command:

cd $GOPATH/src/volcano.sh/volcano
make images

## Verify your images
# docker images
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
volcanosh/volcano-admission     latest              a83338506638        8 seconds ago       41.4MB
volcanosh/volcano-scheduler     latest              faa3c2a25ac3        9 seconds ago       49.6MB
volcanosh/volcano-controllers   latest              7b11606ebfb8        10 seconds ago      44.2MB

NOTE: You need ensure the images are correctly loaded in your kubernetes cluster, for example, if you are using kind cluster, try command kind load docker-image <image-name>:<tag> for each of the images.

2. Helm charts

Second, install the required helm plugin and generate valid certificate, volcano uses a helm plugin gen-admission-secret to generate certificate for admission service to communicate with kubernetes API server.

#1. Install helm plugin
helm plugin install installer/chart/volcano/plugins/gen-admission-secret

#2. Generate secret within service name
helm gen-admission-secret --service <specified-name>-admission-service --namespace <namespace>

## For eg: 
kubectl create namespace volcano-trial

helm gen-admission-secret --service volcano-trial-admission-service --namespace volcano-trial

Finally, install helm chart.

helm install installer/chart/volcano --namespace <namespace> --name <specified-name>

For eg :
helm install installer/chart/volcano --namespace volcano-trial --name volcano-trial

NOTE:The <specified-name> used in the two commands above should be identical.

To Verify your installation run the following commands:

#1. Verify the Running Pods
# kubectl get pods --namespace <namespace> 
NAME                                                READY   STATUS    RESTARTS   AGE
<specified-name>-admission-84fd9b9dd8-9trxn          1/1     Running   0          43s
<specified-name>-controllers-75dcc8ff89-42v6r        1/1     Running   0          43s
<specified-name>-scheduler-b94cdb867-89pm2           1/1     Running   0          43s

#2. Verify the Services
# kubectl get services --namespace <namespace> 
NAME                                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
<specified-name>-admission-service       ClusterIP   10.105.78.53   <none>        443/TCP   91s

Community, discussion, contribution, and support

You can reach the maintainers of this project at:

Slack: #volcano-sh

Mailing List: https://groups.google.com/forum/#!forum/volcano-sh

volcano's People

Contributors

tommylike avatar haozi23 avatar lminzhw avatar hzxuzhonghu avatar wangyuqing4 avatar wackxu avatar asifdxtreme avatar kevin-wangzefeng avatar volcano-sh-bot avatar k82cn avatar sivanzcw avatar onlycc avatar quinton-hoole avatar dingtsh1 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.