================================================================
Virtualization technology could provide a unique benefit to protect any legacy application systems from hardware failures. The reliability of virtual machines running on virtualized servers is not only threatened by hardware failures beneath the whole virtual infrastructure, but also nosy hypervisors that essentially support virtual machines cannot be trusted.
We develop the opensource tool, Cuju, which is a virtualization based fault tolerance technique with epoch-based synchronization. There are several performance optimization technologies are applied by Cuju, including a non-stop/pipelined, continuously migration, dirty tracking for guest virtual memory/virtual device status, and eliminate data transfer between QEMU and KVM.
Cuju shows that these optimizations have greatly saved the processor usage, synchronization bandwidth and have significantly improved VM network throughput and latency at the same time.
For more information see: https://cuju-ft.github.io/cuju-web/home.html
- Assume you have already builded Primary, Backup and NFS node.
A recommended topology below:
- If you only have one or two machine, you can reference this setting.
Another recommended topology below:
- Open the Intel virtualization support (VT-x) in your bios.
- Install OS in all nodes: (pick 1 of 2)
- Install related packages in all nodes
$ sudo apt-get update
ubuntu-16:
$ sudo apt-get install ssh vim gcc make gdb fakeroot build-essential \
kernel-package libncurses5 libncurses5-dev zlib1g-dev \
libglib2.0-dev qemu xorg bridge-utils openvpn vncviewer \
libssl-dev libpixman-1-dev nfs-common git
ubuntu-18:
$ sudo apt-get install ssh vim gcc make gdb fakeroot build-essential \
kernel-package libncurses5 libncurses5-dev zlib1g-dev \
libglib2.0-dev qemu xorg bridge-utils openvpn libelf-dev \
libssl-dev libpixman-1-dev nfs-common git tigervnc-viewer
- Set up the bridge and network environment
- You can follow our recommended topology to set up the network environment
- The example of network interfaces set up below (edit your
/etc/network/interfaces):
- NFS node
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 192.168.11.1
netmask 255.255.255.0
gateway 192.168.11.254
dns-nameservers 8.8.8.8
eth0 is your physical NIC name, please modify it according to your actual NIC name
- Primary node
auto lo
iface lo inet loopback
auto br0
iface br0 inet static
bridge_ports eth0
bridge_maxwait 0
address 192.168.11.2
netmask 255.255.255.0
gateway 192.168.11.254
dns-nameservers 8.8.8.8
auto eth0
iface eth0 inet static
address 0.0.0.0
auto eth1
iface eth1 inet static
address 192.168.111.1
netmask 255.255.255.0
- Backup node
auto lo
iface lo inet loopback
auto br0
iface br0 inet static
bridge_ports eth0
bridge_maxwait 0
address 192.168.11.3
netmask 255.255.255.0
gateway 192.168.11.254
dns-nameservers 8.8.8.8
auto eth0
iface eth0 inet static
address 0.0.0.0
auto eth1
iface eth1 inet static
address 192.168.111.2
netmask 255.255.255.0
-
Build the high-speed connections (ex. 10G NIC) with Primary and Backup nodes by the
eth1 -
After editing these network interfaces, type
/etc/init.d/networking restartorreboot
- Install the NFS service (Network FileSystem) in NFS node; then create a NFS folder placing the VM image
$ sudo apt-get install nfs-kernel-server
- Insert this line in
/etc/exportsto add your NFS folder:
/home/[your username]/nfsfolder *(rw,no_root_squash,no_subtree_check)
-
After editing
/etc/exports, type/etc/init.d/nfs-kernel-server restartorreboot -
Go to your nfs folder, then download Cuju and build a VM image file (or download our Ubuntu-16.04 VM image file, the
account/passwordisroot/root), they will be synced with Primary and Backup node.
- Mount the NFS folder
$ sudo mkdir /mnt/nfs
$ sudo mount -t nfs 192.168.11.1:/home/[your username]/nfsfolder /mnt/nfs
- Install the appropriate version of the kernel for Cuju (only on ubuntu-18)
$ sudo apt-get install linux-image-4.15.0-29-generic
$ sudo apt-get install linux-headers-4.15.0-29-generic
- Set appropriate version of the kernel for Cuju and reboot (only on ubuntu-18)
$ sudo vim /etc/default/grub
#GRUB_DEFAULT=0
GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 4.15.0-29-generic"
$ sudo update-grub
Reboot after the above modification
- Clone Cuju on your NFS folder from Github
$ cd /mnt/nfs
$ git clone https://github.com/Cuju-ft/Cuju.git
- Configure & Compile Cuju-ft
$ cd Cuju
$ ./configure --enable-cuju --enable-kvm --disable-pie --disable-xen --target-list=x86_64-softmmu
$ make clean
$ make -j8
- Configure, Compile & insmod Cuju-kvm module
*1*2
$ cd Cuju/kvm
$ ./configure
$ make clean
$ make -j8
$ ./reinsmodkvm.sh
P.S.
*1If you meeterror: incompatible type for argument 5 of '__get_user_pages_unlocked', you can use this patch:$ cd Cuju $ patch -p1 < ./patch/__get_user_pages_unlocked.patch
*2If you meeterror: implicit declaration of function 'use_eager_fpu' [-werror=implicit-function-declaration], you can use this patch:$ cd Cuju $ patch -p1 < ./patch/use_eager_fpu.patch
- Before launching your VM, you should update kvm module in Primary and Backup nodes:
$ cd /mnt/nfs/Cuju/kvm
$ ./reinsmodkvm.sh
- Boot VM (on Primary Host, /mnt/nfs/Cuju)
runvm.sh
sudo ./x86_64-softmmu/qemu-system-x86_64 \
-drive if=none,id=drive0,cache=none,format=raw,file=/mnt/nfs/Ubuntu20G-1604.img \
-device virtio-blk,drive=drive0 \
-m 1G -enable-kvm \
-net tap,ifname=tap0 -net nic,model=virtio,vlan=0,macaddr=ae:ae:00:00:00:25 \
-cpu host \
-vga std -chardev socket,id=mon,path=/home/[your username]/vm1.monitor,server,nowait -mon chardev=mon,id=monitor,mode=readline
You need to change the guest image path (file=/mnt/nfs/Ubuntu20G-1604.img) and monitor path (path=/home/[your username]/vm1.monitor) for your environment
- Use VNC to see the console
$ vncviewer :5900 &
The default account/password is root/root if you use we provide guest image
- Start Receiver (on Backup Host, /mnt/nfs/Cuju)
recv.sh
sudo x86_64-softmmu/qemu-system-x86_64 \
-drive if=none,id=drive0,cache=none,format=raw,file=/mnt/nfs/Ubuntu20G-1604.img \
-device virtio-blk,drive=drive0 \
-m 1G -enable-kvm \
-net tap,ifname=tap1 -net nic,model=virtio,vlan=0,macaddr=ae:ae:00:00:00:25 \
-vga std -chardev socket,id=mon,path=/home/[your username]/vm1r.monitor,server,nowait -mon chardev=mon,id=monitor,mode=readline \
-cpu host \
-incoming tcp:0:4441,ft_mode
- You need to follow Boot VM script to change the related parameter or you can use following script to replace Receiver start script (if your VM start script is runvm.sh)
recv.sh
sed -e 's/mode=readline/mode=readline -incoming tcp\:0\:4441,ft_mode/g' -e 's/vm1.monitor/vm1r.monitor/g' -e 's/tap0/tap1/g' ./runvm.sh > tmp.sh
chmod +x ./tmp.sh
./tmp.sh
- After VM boot and Receiver ready, you can execute following script to enter FT mode
ftmode.sh
ubuntu-16:
sudo echo "migrate -c tcp:192.168.111.2:4441" | sudo nc -U /home/[your username]/vm1.monitor
ubuntu-18:
sudo echo "migrate -c tcp:192.168.111.2:4441" | sudo nc -w 1 -U /home/[your username]/vm1.monitor
You need to change the ip address and port (tcp:192.168.111.2:4441) for your environment, this is Backup Host's IP
And change the monitor path (/home/[your username]/vm1.monitor) for your environment
-
If you successfully start Cuju, you will see the following message show on Primary side:
-
If you want to test failover You can
killorctrl-cVM on the Primary Host -
You will need new session with vncviewer:
- If you have Primary Host and Backup Host, execute on Backup Host:
$ vncviewer :5900 & - If you only have Primary Host with two VM:
$ vncviewer :5901 &
- If you have Primary Host and Backup Host, execute on Backup Host:
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent