linuxkit / virtsock

Go bindings for virtio and Hyper-V sockets

This repository contains Go bindings and sample code for Hyper-V sockets and virtio sockets(VSOCK).

Organisation

• pkg/hvsock: Go binding for Hyper-V sockets
• pkg/vsock: Go binding for virtio VSOCK
• cmd/sock_stress: A stress test program for virtsock
• cmd/vsudd: A unix domain socket to virtsock proxy (used in Docker for Mac/Windows)
• scripts: Miscellaneous scripts
• c: Sample C code (including benchmarks and stress tests)
• data: Data from benchmarks

Building

By default the Go sample code is build in a container. Simply type make.

If you want to build binaries on a local system use make build-binaries.

Testing

There are several examples and tests written both in Go and in C. The C code is Hyper-V sockets specific while the Go code also works with virtio sockets and HyperKit. The respective READMEs contain instructions on how to run the tests, but the simplest way is to use LinuxKit.

Assuming you have LinuxKit installed, the make target make linuxkit will build a custom Linux image which can be booted on HyperKit or on Windows. The custom Linux image contains the test binaries.

macOS

Boot the Linux VM:

linuxkit run hvtest

This should create a directory called ./hvtest-state.

Run the server in the VM and client on the host:

linux$ sock_stress -s vsock -v 1
macos$ ./bin/sock_stress.darwin -c vsock://3 -m hyperkit:./hvtest-state -v 1

Run the server on the host and the client inside the VM:

macos$ ./bin/sock_stress.darwin -s vsock -m hyperkit:./hvtest-state -v 1
linux$ sock_stress -c vsock://2 -v 1

Windows

On Windows we currently only support the server to be run inside the VM and the host connecting to it. In the future we will support running the server on the host as well.

For Linux guests on Windows there are two different implmentations, one in the LinuxKit 4.9.x kernels and one in 4.14.x upstream kernels. They require different protocols to be used. The sock_stress and vsudd programs automatically detect which version to use.

Boot the Linux VM (from an elevated powershell):

linuxkit run -name hvtest hvtest-efi.iso

Run the server in the VM and client on the host:

linux$ sock_stress -v 1 -s hvsock
win$ sock_stress -v 1 -c hvsock://<VM ID>

(where <VM ID> is from the output of: (get-vm hvtest).Id)

Run the server on the host and the client inside the VM:

win$ sock_stress -v 1 -s hvsock
linux$ sock_stress -v 1 -c hvsock://parent

Note: This may fail on the client with receiving unexpected EOFs (see below).

Known limitations

• hvsock: When running the server on the host with a client in a Linux VM, it looks like unidirectional shutdown() is not working properly. There appears to be a race of sort.

• hvsock: Hyper-V socket implementations prior to Windows build 10586 (aka 1511, aka Threshold 2) was buggy. There may even be issues with build prior to build 14393 (aka 1607, aka Redstone 1).

• hvsock: Earlier versions of this code supported the older Windows builds, but support has now been removed. If you require the older version, please use the end_10586_tag.

• vsock: There is general host side implementation as the interface is hypervisor specific. The vsock package includes some support for connecting with the VSOCK implementation in Hyperkit, but there is no implementation for, e.g. qemu.

virtsock's Issues

go: client on Linux get EOF on read

When running the sock_stress tests with the server on the host (sock_stress -v 1 -s hvsock) and the client in the Linux VM sock_stress -v 1 -c hvsock://parent the client frequently gets errors on receive, like this

2018/02/07 13:30:41 [00086] Failed to receive: EOF
2018/02/07 13:30:41 [00086] Failed to receive: EOF
2018/02/07 13:30:41 [00086] TX/RX:      49349 bytes in    36.6093 ms
2018/02/07 13:30:41 [00086] Checksums don't match
2018/02/07 13:30:41 [00087] Failed to receive: EOF
2018/02/07 13:30:41 [00087] Failed to receive: EOF

The server does not report any errors and the test works fine in the other direction.

It seems timing sensitive as some connections work fine, even with a similar amount of data transmitted.

The server (on the host) is basically performing:

con := Accept()
io.Copy(con, con)
con.Close()

The client in one go-routine is sending random data and when all data is sent calls (CloseWrite()). In another go routine is it repeatably calling io.ReadAll() of fixed sized chunks (min of buffer or data left to read). It's the io.ReadAll() which sees the EOF before all data is read.

arm64 support

It seems that only GOARCH=amd64 is specified in the Makefile, and I'm wondering if there's a timeline to also build with GOARCH=arm64, or what will be the blockers to make it work for arm64. Thank you!

File descriptor leak in Dial function on Linux

Code in repo:

virtsock/pkg/vsock/vsock_linux.go

Lines 30 to 47 in f8cee7d

	func Dial(cid, port uint32) (Conn, error) {
	fd, err := syscall.Socket(unix.AF_VSOCK, syscall.SOCK_STREAM|syscall.SOCK_CLOEXEC, 0)
	if err != nil {
	return nil, errors.Wrap(err, "Failed to create AF_VSOCK socket")
	}
	sa := &unix.SockaddrVM{CID: cid, Port: port}
	// Retry connect in a loop if EINTR is encountered.
	for {
	if err := unix.Connect(fd, sa); err != nil {
	if errno, ok := err.(syscall.Errno); ok && errno == syscall.EINTR {
	continue
	}
	return nil, errors.Wrapf(err, "failed connect() to %08x.%08x", cid, port)
	}
	break
	}
	return newVsockConn(uintptr(fd), nil, &Addr{cid, port}), nil
	}

The leak is on line 42, when we return from a function with error without closing a file descriptor.

Please clarify which part of the code is covered by BSD & MIT license

The LICENSE file now mentions the affected files but it is not clear which parts are licensed differently:
...
Some of the code in ./go/hvsock.go, ./go/hvsock_windows.go, and
zsyscall_windows.go is covered by the following licenses:
...

Remove support for Hyper-V socket pre Windows build 14393

The hvsock package has a bunch of workaround for windows builds earlier than 14393 (1607 Anniversary Update). Specifically this is a small protocol implemented on top of the raw socket interface to emulate shutdown() (CloseRead()/CloseWrite() in Go).

Since the downstream users of the hvsock package no longer support earlier Windows versions we can remove this work around.

Checkout Go support for AF_VSOCK

See golang/sys@d67a327

Implement `accept()`/`Listen()` for windows

Due to a bug in earlier builds of windows hvsock did not provide support for Listen() on the Windows host. Since downstream users of hvsock no longer support the earlier versions, we should now implement Listen()