contiv-experimental / volplugin

**EXPERIMENTAL** Contiv Storage: Policy backed Clustered Storage (via Ceph or NFS) for Docker

volplugin: cluster-wide volume management for container ecosystems

Note: Most of this documentation is about the open source project. If you came to try Contiv, read our documentation.

volplugin controls Ceph RBD or NFS devices, in a way that makes them easy to use for devs with docker, and flexible to configure for ops. Reference your volumes with docker from anywhere your storage is available, and they are located and mounted. Works great with Compose and Swarm, now Mesos too!

Our profiles system makes instantiating lots of similar class volumes a snap, allowing for a variety of use cases:

• Give your dev teams full-stack dev environments (complete with state) that arrive on demand. They can configure them.
• Scale your stateful containers in a snap with our snapshot facilities, just volcli volume snapshot copy and refer to the volume immediately. Anywhere. (Ceph only)
• Container crashed? Host died? volplugin's got you. Just re-init your container on another host with the same volume name.

volplugin currently only supports Docker volume plugins. First class scheduler support for: Kubernetes and Mesos will be available before the first stable release.

• On-the-fly image creation and (re)mount from any Ceph source, by referencing a policy and volume name.
• Manage many kinds of filesystems, including providing mkfs commands.
• Snapshot frequency and pruning. Also copy snapshots to new volumes!
• Ephemeral (removed on container teardown) volumes
• BPS limiting (via blkio cgroup)

volplugin is still alpha at the time of this writing; features and the API may be extremely volatile and it is not suggested that you use this in production.

Try it out

This will start the suite of volplugin tools in containers from the contiv/volplugin image. It will do the work of configuring docker for you. Note that you must have a working ceph environment that volplugin can already use.

$ docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock contiv/volplugin-autorun

If you get an error like "mountpoint / is not a shared mount", set MountFlags=shared in your systemd unit file for docker. It will most likely be set to slave instead.

Development Instructions

Our Getting Started instructions should be the first thing you read. The prerequisites are absolutely necessary.

Please see our CONTRIBUTING document as well.

Please read the Makefile for most targets. If you make build you will get apiserver/volplugin/volcli installed on the guests, so make run-build if you want a go install'd version of these programs on your host. apiserver/volplugin do not run on anything but linux (you can use volcli, however, on other platforms).

make start will start the development environment. make stop stops, and make restart rebuilds it.

If you wish to run the tests, make test. The unit tests (make unit-test) live throughout the codebase as *_test files. The system tests / integration tests (make system-test) live in the systemtests directory. Note that make system-test will not successfully run on OSX due to dependencies on unavailable libraries.

volplugin's Issues

Question: How to deploy on CoreOS?

Are there any hints on how to run the volplugin on CoreOS?

We run a fully Dockerized Mesos and Ceph environment, and for using stateful containers, we'd like to use Docker volumes on Ceph. As far as I understand the installation of this plugin requires the some libraries to be present on the host system.

As CoreOS works a little different then CentOS/Debian/etc., is there a way to run the plugin within a container itself? If no, how would I otherwise be able to use it on CoreOS?

Fix /get/{policy}/{volume} REST API

It currently returns

{
  "Volume": {
    "Name": "policy1/postgres-vol",
    "Mountpoint": "/mnt/ceph/rbd/policy1.postgres-vol"
  },
  "Err": ""
}

Expect it to return same output as volcli volume get policy1/postgres-vol:

{
  "policy": "policy1",
  "name": "postgres-vol",
  "driver": {
    "pool": "rbd"
  },
  "create": {
    "size": "10MB",
    "filesystem": "ext4"
  },
  "runtime": {
    "snapshots": true,
    "snapshot": {
      "frequency": "30m",
      "keep": 20
    },
    "rate-limit": {
      "write-iops": 0,
      "read-iops": 0,
      "write-bps": 0,
      "read-bps": 0
    }
  },
  "backend": "ceph"
}

Runtime and Creation policies

Overview

Splitting the policy into runtime and create-time configuration policies seems like the best idea for the purposes of modifying the runtime configuration and keeping the create-time configuration static, something that has been a problem up to this point.

Property Map

Runtime Configuration

Consists of things that would be applied in a variant facility, like at mount time.

• rate limiting
• snapshots

Create Configuration

Consists of things that would need to be applied at create time and cannot be modified at runtime, such as size.

• size
• filesystem
• pool

Logic

Logic would look something like this:

• At create time, apply the runtime parameters (if necessary) to the created device
  • Create & Format the volume according to the create parameters
• At mount time, apply the runtime parameters. Establish a watching protocol for changes so they can be updated on the fly.
• volsupervisor will accordingly watch the snapshots parameters to act on them.

This is a big change and I believe I will take it myself.

volcli: use `tenant/volume` syntax instead of multiple args

volcli has many tools that use this notation: volcli command subcommand tenant volume, which I think would be more canonically represented as volcli command subcommand tenant/volume like it is used inside docker.

snapshots race volume remove

I noticed today that the process of taking a snapshot can race volcli volume remove when the snapshot delay is short enough so that they compete.

timeouts for rbd commands/ceph driver

Currently, if the rbd tool hangs, so does our suite of tools. incorporate a sideband timeout that appropriately errors out when this happens, so the administrator can fix the ceph installation (Which is probably already hosed)

Security, AAA, and policy management, and multitenancy

Overview

This is a bucket-list of ideas @jainvipin and I sketched out while thinking about this problem.

Basically we want to have a basic RBAC in the sense that each consumer has a role, and roles are used to gain access to things.

Certs are tied to roles. Certs are the unit of authentication. Superusers will have a secondary set of certs to access etcd directly. We are thinking about using notary or atlas to manage this portion.

As mentioned, there is a super-user group that has access to several commands, like:

• volume list-all
• tenant CRUD commands (see below), except get which will be role-scoped.

Tenants are policies which group other policies, and a role. Tenants live above the policy level and scope policies by role. Roles in this iteration will have no place in policies themselves (but it is expected this will change).

Tenant commands:

These are superuser commands:

• tenant upload (upload a tenant policy, similar to policy upload)
• tenant delete
• tenant list

Role-gated commands:

• tenant get

Other commands

All other commands with volmaster access will be role-gated, otherwise etcd certs will be used.

Address configuration for running volplugin suite in containers

Right now we don't test this, or support it, and we should! #63 and #60 mention this is a desirable feature and I think given prioritization this should be done.

Some dockerfiles to build pre-baked images for people would certainly be good too.

having issue running make run-build

make run-build
go get -u github.com/kr/godep
GOGC=1000 godep go install -v ./volcli/volcli/ ./volplugin/volplugin/ ./volmaster/volmaster/ ./volsupervisor/volsupervisor/
config/global.go:12:2: cannot find package "github.com/Sirupsen/logrus" in any of:
/usr/lib/golang/src/github.com/Sirupsen/logrus (from $GOROOT)
/home/rodrigo/newwork/src/github.com/contiv/volplugin/Godeps/_workspace/src/github.com/Sirupsen/logrus (from $GOPATH)
/home/rodrigo/newwork/src/github.com/Sirupsen/logrus
config/volume.go:13:2: cannot find package "github.com/alecthomas/units" in any of:
/usr/lib/golang/src/github.com/alecthomas/units (from $GOROOT)
/home/rodrigo/newwork/src/github.com/contiv/volplugin/Godeps/_workspace/src/github.com/alecthomas/units (from $GOPATH)
/home/rodrigo/newwork/src/github.com/alecthomas/units
volcli/volcli/cli.go:7:2: cannot find package "github.com/codegangsta/cli" in any of:
/usr/lib/golang/src/github.com/codegangsta/cli (from $GOROOT)
/home/rodrigo/newwork/src/github.com/contiv/volplugin/Godeps/_workspace/src/github.com/codegangsta/cli (from $GOPATH)
/home/rodrigo/newwork/src/github.com/codegangsta/cli
volcli/volcli.go:18:2: cannot find package "github.com/contiv/errored" in any of:
/usr/lib/golang/src/github.com/contiv/errored (from $GOROOT)
/home/rodrigo/newwork/src/github.com/contiv/volplugin/Godeps/_workspace/src/github.com/contiv/errored (from $GOPATH)
/home/rodrigo/newwork/src/github.com/contiv/errored
storage/backend/ceph/ceph.go:16:2: cannot find package "github.com/contiv/executor" in any of:
/usr/lib/golang/src/github.com/contiv/executor (from $GOROOT)
/home/rodrigo/newwork/src/github.com/contiv/volplugin/Godeps/_workspace/src/github.com/contiv/executor (from $GOPATH)
/home/rodrigo/newwork/src/github.com/contiv/executor
watch/watch.go:11:2: cannot find package "github.com/coreos/etcd/client" in any of:
/usr/lib/golang/src/github.com/coreos/etcd/client (from $GOROOT)
/home/rodrigo/newwork/src/github.com/contiv/volplugin/Godeps/_workspace/src/github.com/coreos/etcd/client (from $GOPATH)
/home/rodrigo/newwork/src/github.com/coreos/etcd/client
volcli/volcli.go:21:2: cannot find package "github.com/kr/pty" in any of:
/usr/lib/golang/src/github.com/kr/pty (from $GOROOT)
/home/rodrigo/newwork/src/github.com/contiv/volplugin/Godeps/_workspace/src/github.com/kr/pty (from $GOPATH)
/home/rodrigo/newwork/src/github.com/kr/pty
watch/watch.go:12:2: cannot find package "golang.org/x/net/context" in any of:
/usr/lib/golang/src/golang.org/x/net/context (from $GOROOT)
/home/rodrigo/newwork/src/github.com/contiv/volplugin/Godeps/_workspace/src/golang.org/x/net/context (from $GOPATH)
/home/rodrigo/newwork/src/golang.org/x/net/context
storage/backend/ceph/ceph.go:13:2: cannot find package "golang.org/x/sys/unix" in any of:
/usr/lib/golang/src/golang.org/x/sys/unix (from $GOROOT)
/home/rodrigo/newwork/src/github.com/contiv/volplugin/Godeps/_workspace/src/golang.org/x/sys/unix (from $GOPATH)
/home/rodrigo/newwork/src/golang.org/x/sys/unix
volplugin/handlers.go:15:2: cannot find package "github.com/docker/docker/pkg/plugins" in any of:
/usr/lib/golang/src/github.com/docker/docker/pkg/plugins (from $GOROOT)
/home/rodrigo/newwork/src/github.com/contiv/volplugin/Godeps/_workspace/src/github.com/docker/docker/pkg/plugins (from $GOPATH)
/home/rodrigo/newwork/src/github.com/docker/docker/pkg/plugins
volplugin/volplugin.go:21:2: cannot find package "github.com/gorilla/mux" in any of:
/usr/lib/golang/src/github.com/gorilla/mux (from $GOROOT)
/home/rodrigo/newwork/src/github.com/contiv/volplugin/Godeps/_workspace/src/github.com/gorilla/mux (from $GOPATH)
/home/rodrigo/newwork/src/github.com/gorilla/mux
godep: go exit status 1
make: *** [run-build] Error 1

Stop using rbd command

I want to use this plugin on coreos but rbd isn't available.

I wonder why not using https://github.com/ceph/go-ceph/tree/master/rbd for any rbd operation? It seems to be more stable (thanks to godep) and with less dependency on an external tool who can change from different versions.

volume <-> pool relationship leaks volumes

The current volume / pool relationship is flawed; it currently relies on volumes being the source of truth for a volume's existence, and the pool name / volume name to be present as a combination for mount handling.

This is a relational nightmare and is the result of thousands of cuts to this part of the code. What needs to happen is the naming of volumes needs to change to include the tenant name as well.

`make start` should also `make run`

/cc @jainvipin

Storage Backends

Some ideas (please add your own!):

• NFS
• Amazon EBS
• GCE storage

Backend support for Consul

Hey there,

I am carefully crawl towards CEPH backed volumes after I played around with the new networking:
http://qnib.org/2015/11/29/multi-host-slurm/

I would like to expand the post with ceph-backed volumes. For now you only support etcd if I get the config/config.go file right... Not [yet] a way to plug in Consul. No big deal, I could spin up an etcd-instance, no problem.

Just so that I know. Maybe others are interested in this as well.

Cheers, keep up the great work
Christian

latest volcli breaks contivctl

Reference: http://contiv.ngrok.io/job/Contivctl_Release/25/

Improve error messages in volcli

Several error messages aren't very clear or incomplete, so we need to flesh this out some.

Volsupervisor to take a signal to snapshot remotely

volsupervisor needs a HTTP REST endpoint to service a signal from applications or database containers which wish to sync their filesystems before snapshotting for a consistent snapshot; this signal would create a manual snapshot that would still get cleaned up in a normal keep scenario.

The signaling will be toggle-able through the policy by adding a signal: true property in the snapshot runtime configuration. It is orthogonal to the standard snaps (which should now accept a frequency of null to turn them off, but still allow for keep pruning)

It would also be nice if we had a tool that accomplished the client side of this operation, e.g., "run a command then fire the snapshot, then run another command". Wrapping this will play well into the security story later.

use rbd locks

We don't right now (we handle this in etcd), but we should do both for safety.

Retrieve mount points from /proc/self/mountinfo

This will enable better volplugin recovery on host restart.

develop a null volume driver

This driver shall be primarily used for testing and benchmarking voplugin functionality in isolation.

ephemeral volumes do not get cleaned up on host disappearance

Right now, if the host disappears and docker loses state of the volume, we leak a volume that was intended to be deleted upon container/volume teardown inside of docker.

To resolve this, I propose we implement a scheduled jobs daemon that handles this and snapshots, and of course, can be used for more later. This volsupervisor could live on one host with a connection to the volmaster and etcd, and can be asynchronously managed with the rest of the cluster due to its tertiary purpose.

auto generation of bash auto completion for volcli

It will be good to tie bash completion scripts to the go source code as discussed here #126 (comment)

Snapshot management

Overview

Snapshots in volplugin suck. Let's make them better.

What's good

• Takes snapshots
• Does them on an interval
• Keeps X of the latest ones

What's not

(checklist is for separate PRs that could solve this)

• Periodic snapshots that happen on tiers
• Toggle snapshotting during operation (after the policy's been set for the volume)
• volume copy <tenant> <orig> <snap> <dest>:
  • copies a snapshot to a new volume
  • or if the snapshot is omitted, makes an intermediate snapshot of original volume, and uses that snapshot to make the destination volume.
  • tenant is provided to be in concert with #164, does not need to exist until these changes have arrived.
• Snapshots currently race volume remove: #156
• Snapshot signalling so containers can perform operations before triggering the snapshot.

volcli improvement: detect etcd missing keys and respond with nicer messages

Currently this would involve changes to the volmaster, config library and volcli. If the volmaster was smart enough to return 404 on etcd "key not found" failures, it would be a much nicer user experience, and easier to test in the long term I think. It could also exit a different value if the etcd key is missing.

Can't create volume

I've got a straightforward test setup going. With rbd I can create, map and mount volumes.

Running:

./volmaster --etcd http://172.16.31.215:24 --debug &
./volplugin --debug &
docker volume create -d volplugin --name='tenant1/test3'

I get

#( from volmaster)
DEBU[0109] Dispatching /create with {"tenant":"tenant1","volume":"test3","opts":{}}
DEBU[0110] mapped volume "tenant1.test3" as "/dev/rbd1"
WARN[0110] Returning HTTP error handling plugin negotiation: Creating volume exit status 1

#( from volplugin) 
DEBU[0097] Dispatching Create with {"Name":"tenant1/test3","Opts":{}}
WARN[0098] Returning HTTP error handling plugin negotiation: Could not determine tenant configuration Status was not 200: was 500: "Creating volume exit status 1"

#(from docker)
Error response from daemon: Plugin Error: VolumeDriver.Create, {"Mountpoint":"","Err":"Could not determine tenant configuration Status was not 200: was 500: \"Creating volume exit status 1\""}

The end result is that the rbd volume is created, mappped, and the ext4 fs is created on it, but volcli volume list-all doesnt show it. This is what's in etcd:

/volplugin/volumes
/volplugin/volumes/tenant1
/volplugin/users
/volplugin/users/tenant1
/volplugin/users/tenant1/test3
/volplugin/tenants
/volplugin/tenants/tenant1

I'm not sure what other debugging I can do on this. It's not clear to me what step of the create is failing.

Collaborate

Hi there.

We are about to open source our Ceph RDB driver plugin and wondered if you all would like to collaborate on this to make one good one?

Let me know and thanks!

Refactor: volmaster and volsupervisor need to take daemon structs as self

Right now we pass these around the daemon object quite a bit, and in volsupervisor's case, a lot of arguments that travel down the stack. We should organize these into structs the methods live on, so that they can re-use the same data without having ugly call sites.

How is the container mount point propagated to docker daemon?

This is more of an observation than an issue, so feel free to close it.

I am trying to run the volplugin daemons in containers on a system that has ceph installed in containers.

I am sure I am missing something, but I thought that docker plugins were designed to be able to run as in containers. So this is an experiment.

I run some containers as follows:

docker run -d --name volmaster \
 --net=host --privileged -e KV_IP=10.1.0.11 \
 vol \
 volmaster --debug --listen 10.1.0.11:9005 --etcd http://10.1.0.11:2379

docker run -d --name volsuper \
 --net=host --privileged -e KV_IP=10.1.0.11 \
 vol \
 volsupervisor --etcd http://10.1.0.11:2379

docker run -d --name volplug \
 --net=host --privileged -e KV_IP=10.1.0.11 \
 -v /var/run/docker:/var/run/docker \
 -v /sys:/sys -v /dev:/dev \
 volplug \
 volplugin --debug --master 10.1.0.11:9005

In the volplug container I can list my rbd images and map/unmap them fine with rbd commands.

docker volume create -d volplugin --name=tenant1/test works fine.

However docker run -it --rm -v tenant1/test:/mnt ubuntu bash will generate an error from docker:

Timestamp: 2015-12-01 16:14:49.745337292 -0500 EST
Code: System error

Message: stat /mnt/ceph/rbd/tenant1.test: no such file or directory

Frames:

---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@40

---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@57

---
2: StartInitialization
Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
File: factory_linux.go@242

---
3: initializer
Package: github.com/docker/docker/daemon/execdriver/native
File: init.go@35

---
4: Init
Package: github.com/docker/docker/pkg/reexec
File: reexec.go@26

---
5: main
Package: main
File: docker.go@18

---
6: main
Package: runtime
File: proc.go@63

---
7: goexit
Package: runtime
File: asm_amd64.s@2232
Error response from daemon: Cannot start container 18abc76c63d7b2661b6423e553c5c03d03bca6874681cd1564fc225b7e3a923e: [8] System error: stat /mnt/ceph/rbd/tenant1.test: no such file or directory

I am guessing that I am not propagating the mounted filesystem correctly to the system docker. I have put a check in the storage/ceph/ceph.go code and the filesystem is mounted and readable in the volplug container. Everything then gets cleaned up after the error.

I have MountFlags=slave set for the docker daemon.

Should I share the mount point /mnt/ceph?

requestVolumeConfig with tenant/volume or tenant/name?

I am just testing this out and I get an error when creating a volume and attempting to mount it into a container.

Here are some logs from volplugin:

time="2015-11-30T19:15:48Z" level=debug msg="Dispatching Path with {\"Name\":\"tenant1/foo\"}" 
time="2015-11-30T19:15:48Z" level=info msg="Returning mount path to docker for volume: \"tenant1/foo\"" 
time="2015-11-30T19:15:48Z" level=warning msg="Returning HTTP error handling plugin negotiation: Requesting tenant configuration Status was not 200: was 404: \"\"" 
time="2015-11-30T19:15:48Z" level=debug msg="Dispatching Mount with {\"Name\":\"tenant1/foo\"}" 
time="2015-11-30T19:15:48Z" level=info msg="Mounting volume \"tenant1/foo\"" 
time="2015-11-30T19:15:48Z" level=warning msg="Returning HTTP error handling plugin negotiation: Could not determine tenant configuration Status was not 200: was 404: \"\"" 

The logs from volmaster shows that the it get the requests:

time="2015-11-30T19:15:48Z" level=debug msg="Dispatching /request with {\"volume\":\"tenant1/foo\",\"tenant\":\"tenant1\"}" 
time="2015-11-30T19:15:48Z" level=debug msg="Dispatching /request with {\"volume\":\"foo\",\"tenant\":\"tenant1\"}" 

but the volume is different.

should https://github.com/contiv/volplugin/blob/master/volplugin/handlers.go#L132

be volConfig, err := requestVolumeConfig(master, uc.Tenant, uc.Name)?

if mkfs fails, use lock is acquired and no volume is written

write the volume after rbd create instead of mkfs.

some parameters should not stay with the volume

I wonder if IOPS in particular should be handled as a part of the volume policy, or as a parameter on mount. Currently it does the former. I think it might be best if it did the latter. I'm not sure what the best solution is right now.

NFS Backend

• Add NFS mount support
• Add lock-free operation (toggle per volume w/ flag or policy) for mounts so NFS can be fully exploited (moved to #259)
• Split runtime, create time, and mount time parameters to use separate backends (this should be tracked in a separate ticket)
• "Dummy" filer for testing advanced filer features (this too)
• Initial filer support for snapshots, size quotas and IOPS -- not quite sure on the hardware target yet. (moved to #260)
• Support for graceful NFS failover when a mount goes stale (moved to #261)

change certain volume parameters after creation

After creating a volume, parameters like read/write iops should be tweakable. Currently, the system is designed to do all the work up front and ignore future attempts to modify the parameters are ignored.

What we need to do is identify modifiable parameters (e.g., iops or snapshot schedule) and partition them from the unmodifiable ones (e.g., volume group or pool), and provide some kind of gate that allows the modifiable ones to dispatch handlers which then correct anythign that needs to be corrected, overlay the new configuration over the old one, and push the corrected result safely. This will need to happen on the volmaster and propagate down to the volplugin, so we need some kind of pub/sub functionality I think.

It will also need to acquire a use lock for the transaction processing time.

Monitoring integration

The info package allows us to export some basic information about the health of the services. We can probably use this to define thresholds within the services themselves to take action based on how healthy they are; e.g. a volplugin who's failing a lot of mounts, or has an obnoxious number of goroutines, may choose to restart itself or refuse mounting new devices until serviced.

It should also be able to signal a REST endpoint with any health updates.

if volplugin map/mount fails, rbd lock is not freed in some situations

if volcli volume remove fails, use lock is still held

probably just needs a defer in the right spot.

Metrics integration

The volmaster, volsupervisor, and volplugin should all expose metrics endpoints over their transports, e.g. volplugin serves over the unix socket and volmaster can serve over http. volsupervisor should probably publish over a unix socket as well to avoid having to expose a network endpoint.

Metrics to expose:

• Total # of Mounts over run time
• Total # of Active Mounts
• Total space used
• Per-volume space information
• Per-volume IO information
• all the statistics from our info package

volume mounting to more than one container

I made a volume with volplugin. Saw that i was able to mount with it in different hosts while having volplugin running. However, if i tried mounting the volume with more than one container, it either tells me that there is a lock or sometimes it seems to mess things up and im not able to mount to the volume anymore.

Docker umount issue

Tracking this here: moby/moby#20939

rate-limit volume format

apply IOPS to new volumes on volmasters for formatting

debugging information & version information

volplugin needs a feature which allows us to retrieve debugging information from a running system and display it in the CLI.

The following bits of information would be the minimum set of fields to show:
the number of file descriptors
volplugin version
ceph version (?)
architecture
kernel version

Re-upload of policy orphans volumes

reproduction:

volcli policy upload policy1 < /testdata/intent1.json
volcli volume create policy1/test
volcli policy upload policy1 < /testdata/fastsnap.json
volcli volume create policy1/test2
volcli volume list-all
<output>
policy1/test2
</output>

The resulting policy1/test is still in existence, but its etcd record is gone, basically orphaning it.

This should be an easy patch so I would ask that some others on the team could take it?

define and restrict a set of metacharacters in volume names

Right now / and . are special characters. There's bound to be a few more as we implement data stores. However, we do not ensure that the volume requested to be created does not have these characters in them already. Since we do a lot of transliteration of these two (ceph does not accept / in volume names) it's important we do not kill ourselves in the process.

This should be easy to do.

swarm `docker run` does not clean up volume on rm

it may just be that an upgrade is required.

docker-archive/classicswarm#1189 looks like the relevant issue here.

Implement RBD locking

Questions arising from #78 seemed to indicate that implementing mandatory rbd locking would solve a few problems acquiring locks in circumstances where network partitions were present.

Implementing this directly in the storage layer is the way to go. It should not be part of the driver interface, but instead be an internal component to the ceph driver.

support TTL-backed operations in mount, use locks

Use locks and mount operations both need to expire when the host dies, so that other hosts can reschedule the job and get the volumes they need.

Currently we have no support for this; the volumes will linger in etcd until someone manually clears them., and cannot be mounted anywhere else by volplugin. This is unacceptable.

Solution: implement TTL backed publishing of records to etcd. This will allow a failing volplugin to expire gracefully.

Requirements:

• TTL support must be refreshed periodically. Goroutines will accomplish this task trivially. This patch is ready and will be delivered at the end of this project.
• In the case we restart volplugin without clearing mounts or restarting the system, that state will be lost, so we must re-establish it. #73 and #57 accomplish this part of the equation by scanning both ceph and existing mounts to establish a unified set of mount points to check against the etcd store. This will allow us to validate that nobody else has attempted to acquire the lock in the meantime.
• We should also have a "maintenance mode" which performs no operations but refreshing etcd and removing mounts (no new mounts). This will allow the mounts to be held still by the containers, and locks can be managed safely between machines. This will be sent in an independent pull request.
• Updating etcd should go through the volmaster, as the volplugin may not have access to it. This patch is ready and will be delivered at the end of this project.
• Additionally, some rewiring of data structures in the use lock system may be required.

use a proper units package for intent size

something that supports 20G or 14.4k.

Reclamation policies

A policy system for groups with large numbers of volumes to garbage collect volumes after they have been stale for a certain period of time.

Probably should be applied at the policy level. Volsupervisor can take care of the dirty work.

Fix downloading the ceph stable repository key

This breaks VM setup about 50% of the time. Use a static file in the ansible subtree instead.

Global configuration in etcd

This would provide top-level configuration the volmaster, volsupervisor, and volplugin share and read (and refresh) from accordingly.

The system should be designed to:

• Handle dynamic updates (watch points and re-registration)
• volcli commands, e.g. volcli config upload
• handle at least these two fields for now:
  • TTL
  • Debug

See comments in #78 for more information