dreamlab / vmshepherd

VmShepherd is a lightweight, scalable, standalone asynchronous app to manage clusters

VmShepherd

_ _

Introduction

VmShepherd is an application that helps you maintain groups (clusters) of virtual machines. It keeps defined parameters (like count, image, etc.) by checking state in an IaaS layer, and tests underlying services with a specified health check.

Architecture

VmShepherd is designed to be easily extensible with plugins. Empowered by python3 and its asyncio module to facilitate scaling. The diagram below shows the base components of the app.

• Preset Manager is responsible for fetching cluster spec/definition (preset). Built-in presets: DirectoryDriver and GitRepoDriver
• Runtime Manger exposes functionality of locking preset, and holds intermediate states. Currently available: InMemoryDriver, PostgresDriver, ZookeeperDriver.
• IaaS is a glue (api wrapper) to IaaS provider, OpenStackDriver is the first implemented.
• Healthcheck allows to check service's state, not only for existence of a virtual machine. HttpHealthcheckDriver is built-in.

For more information, please take look at the documentation.

Installation

Application requires python 3.6 or later. The latest stable version is available on Pypi.

pip install vmshepherd

Another way of installation for a VmShepherd is a docker. You can easily download latest version of our application from a docker hub.

docker pull dreamlabcloud/vmshepherd

Before you run a application, you need to prepare configuration files according to this rules.

When you create a configuration file, you can deploy a VmShepherd like that:

docker run -v $PATH_TO_CONFIG_DIRECTORY/:/home/shepherd -p 8888:8888 -it vmshepherd -c config/settings.example.yaml

• Where PATH_TO_CONFIG_DIRECTORY is a localisation of a configuration files on your host
• -c config/settings.example.yaml is a list of arguments passed to a VmShepherd in container

Example:

➜  VmShepherd/docker ✗ sudo docker run -v $(realpath ../)/:/home/shepherd -p   8888:8888 -it vmshepherd -c config/settings.example.yaml
INFO:root:Starting server, listening on 8888.
INFO:root:VmShepherd start cycle...
INFO:root:VMs Status: 1 expected, 0 in iaas, 0 running, 0 nearby shutdown, 0 pending, 0   after time shutdown, 0 terminated, 0 error, 0 unknown, 1 missing
INFO:root:VMs Status update: 0 terminated, 0 terminated by healthcheck, 1 created, 0 failed healthcheck

We also provide a Dockerfile which can be used during a development:

docker build -t vmshepherd .
docker run -it  -p 8888:8888 -p 8000:8000 vmshepherd run

Usage

After installation you need to create a main configuration file (check examples in config/ directory in this repo).

Run:

vmshepherd -c CONFIGFILE

Contributing to VmShepherd

Thank you for your interest in contributing to VmShepherd. Like always there are many ways to contribute, and we appreciate all of them.

Pull requests and issues are the primary mechanism we use to change VmShepherd. Github itself has a great documentation about using Pull Requests. We use the "fork and pull" model, where contributors push changes to their personal fork and create pull requests to bring those changes into the source repository.

If you want to find something to work on, please check issues in our roadmap.

Check out the documetation http://doc.dreamlab.pl/VmShepherd/development/index.html.

TL;DR

Pull requests will need:

• Tests
• Documentation
• A logical series of well written commits

License

Apache License 2.0

vmshepherd's Issues

Complete documentation

Fix or get rid of pipenv

Currently we are not using pipenv as it should be. There are some problems (slows down test start, tox virtualenv vs pipenv's one) without any gain.

OpenStack driver scroll fetch

Data from openstack (list of servers, flavors) should be fetched witch scroll (by N items, paginated)

Use pipenv or poetry or ...

Currently usage of pipenv has been dropped (#10 ), but ultimately it would be great to revisit our needs and take advantages of such a solution (like ease of setting up env, pipfile, ...)

EC2 IaaS driver

Keep preset state

Keep in memory presets' state from last cycle. UI/API should use that state. Currently we generate too much traffic to controllers.

Daemonize

Affinity groups usage

We need to have a affinity groups usage

Flexible logging configuration

Enhance UI

Current approach is not bad, but it would be good to ugrade bootstrap, reformat preset's parameters, add sort/colors to vms tree.

Auto scaling mechanism

Would be nice to implement auto-scaling abstraction (autoscaler driver?), that supports ec2 autoscaler groups and so on..

Current state - API and dashboard

Implement mechanism to refresh vms, runtime state per X time. #4 introduced some some drawbacks, basically it's possible to get out-of-date data

Policy abstraction

Logic that manage preset could be simplified, decoupled in policies like:

• Maintain
• TerminateUnhealthy
• Timedshutdown

RW dashboard with auth

Requirements

It's necessary to add minimal requirements of the app in the documentation/README.

For example:

• python version (i.e. documentation generation fails with Python 3.5.1)
• OS (I guess it's debian based app)