kube-vcloud-flexvolume

VMware vCloud Director flexVolume driver for Kubernetes.

Status

Successfully run this driver on production Kubernetes cluster for over half a year without any loss of data. The current stable version is: 2.3.0.

WARNING: Versions prior to 2.2.1rc1 have a problem with unstable disk paths which under some circumstances could cause data loss. After upgrade from affected versions make sure udev rules have been properly converted to the new format using this script.

Caveats

Due to how vCloud works if you want to simultaneously attach/detach disks to/from same VM you should implement a global lock inside attach/detach commands. ~~Check feature/etcd branch for experimental implementation using etcd key-value store.~~ (Merged via pull request #5)
When Kubernetes node is marked unschedulable (with kubectl drain) operationExecutor on a new node calls AttachVolume before DetachVolume is called on the old one. We periodically poll the volume to find out if is still attached, but vCloud deletes relation before asynchronous detach:disk task was finished. This sometimes can result in throwing an exception "Could not attach volume '%s' to node '%s'" and repeating the attemp by Kubelet process.
Using busType:busSubType combination other than SCSI:VirtualSCSI can lead to unexpected behavior. For example you can attach more than one disk of default type (SCSI:lsilogic), but only the first one will be detected by Linux kernel.
When something goes wrong during disk attaching and the driver throws an exception the udev rules required for restoring symlinks after reboot might not be generated. This can result in similar behaviour to one described in this issue. The code tries to minimize the chances of this happening. If the problem occurs, please fill the bug report.

Description

vcloud-flexvolume provides a storage driver using vCloud's Independent Disk feature. The Independent Disk provides persistent disk storage which can be attached to instances running in vCloud Director environment.

You can read more about Independent Disks here.

Installation

Make sure kubelet is running with --enable-controller-attach-detach=false
Create the directory /usr/libexec/kubernetes/kubelet-plugins/volume/exec/answear.com~vcloud
Install wrapper scripts/vcloud as /usr/libexec/kubernetes/kubelet-plugins/volume/exec/answear.com~vcloud/vcloud
Create the directory /opt/vcloud-flexvolume/etc
Install configuration file config/config.yaml.example as /opt/vcloud-flexvolume/etc/config.yaml and set parameters.

Install packages:

python3
python3-pip
python3-setuptools
python3-wheel
python3-flufl.enum
python3-lxml
python3-yaml
python3-pygments
python3-pyudev

Install the driver itself:

git checkout 2.3.0
python3 setup.py build
sudo python3 setup.py install

pip3 install --process-dependency-links git+https://github.com/answear/[email protected]

Restart kubelet process.

Create a Kubernetes Pod such as:

cat examples/nginx.yaml | kubectl apply -f -

The driver will create an independent disk with name "testdisk" and size 1Gi under storage profile "T1". The volume will also be mounted as /data inside the container.

Upgrading

Install the newest driver version using git or pip.
Apply any changes in example config file to your local copy.

Options

Following options are required:

volumeName - Name of the independent disk volume.
size - Size to allocate for the new independent disk volume. Accepts any value in human-readable format. (e.g. 100Mi, 1Gi)

Optional options may be passed:

busType - Disk bus type expressed as a string. One of: 5 - IDE, 6 - SCSI (default), 20 - SATA.
busSubType - Disk bus subtype expressed as a string. One of: "" (busType=5), buslogic (busType=6), lsilogic (busType=6), lsilogicsas (busType=6), VirtualSCSI (busType=6), vmware.sata.ahci (busType=20).
storage - Name of the storage pool.
mountoptions - Additional comma-separated options passed to mount. (e.g. noatime, relatime, nobarrier)

Driver invocation

Init:

>>> vcloud-flexvolume init
<<< {"status": "Success", "capabilities": {"attach": true}}

Volume is attached:

>>> vcloud-flexvolume isattached '{"kubernetes.io/fsType":"ext4","kubernetes.io/pvOrVolumeName":"testdisk","kubernetes.io/readwrite":"rw","mountoptions":"relatime,nobarrier","size":"1Gi","storage":"T1","busType":6,"busSubType":"VirtualSCSI","volumeName":"testdisk"}' nodename
<<< {"status": "Success", "attached": false}

Attach:

>>> vcloud-flexvolume attach '{"kubernetes.io/fsType":"ext4","kubernetes.io/pvOrVolumeName":"testdisk","kubernetes.io/readwrite":"rw","mountoptions":"relatime,nobarrier","size":"1Gi","storage":"T1","busType":6,"busSubType":"VirtualSCSI","volumeName":"testdisk"}' nodename
<<< {"status": "Success", "device": "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0-part1"}

The driver detects (using udev events) the name of the device under which it was registered by the Linux kernel and automatically creates symlink /dev/block/<URN> pointing to ../<device>. URN is a unique volume ID generated by vCloud Director.

Wait for attach:

>>> vcloud-flexvolume waitforattach /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0-part1 '{"kubernetes.io/fsType":"ext4","kubernetes.io/pvOrVolumeName":"testdisk","kubernetes.io/readwrite":"rw","mountoptions":"relatime,nobarrier","size":"1Gi","storage":"T1","busType":6,"busSubType":"VirtualSCSI","volumeName":"testdisk"}'
<<< {"status": "Success", "device": "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0-part1"}

Mount device:

>>> vcloud-flexvolume mountdevice /mnt/testdisk /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0-part1 '{"kubernetes.io/fsType":"ext4","kubernetes.io/pvOrVolumeName":"testdisk","kubernetes.io/readwrite":"rw","mountoptions":"relatime,nobarrier","size":"1Gi","storage":"T1","busType":6,"busSubType":"VirtualSCSI","volumeName":"testdisk"}'
<<< {"status": "Success"}

Unmount device:

>>> vcloud-flexvolume unmountdevice /mnt/testdisk
<<< {"status": "Success"}

Detach:

>>> vcloud-flexvolume detach testdisk nodename
<<< {"status": "Success"}

TODO

Write some tests.
~~Functions in flexvolume/mount.py should raise Exceptions just like the ones in attach.py.~~
Reuse vCloud API session token between invocations.
Validate input JSON with JSON Schema.

neotkm / kube-vcloud-flexvolume Goto Github PK