hyperhq / hyperstart Goto Github PK

View Code? Open in Web Editor NEW

135.0 135.0 63.0 115.98 MB

The tiny Init service for HyperContainer

Home Page: https://www.hypercontainer.io

License: Apache License 2.0

Shell 1.63% C 96.88% Makefile 0.50% M4 0.98%

containers hyper hypervisor

hyperstart's People

Contributors

Stargazers

Watchers

Forkers

bergwolf jansonzhou hustcat gao-feng jefby stigkj laijs gnawux jimmyxian weizhang555 mba811 liusdu ptptptptptpt xlgao-zju yaozengzeng husttsq hukeping wrouesnel dlespiau devimc amshinde clearlinux xiaods gaocegege jinfenw sboeuf marcosnils cybertan crazykev clearcontainers dvoytik duzhanyuan harche kyasuda2003 weichen81 mendickxiao alexxnica kryndex vincentpanqi sohu0106 jackzhanghk codercom yyyeerbo jwessel starnop teawater bluebreezecf fangwenqi carletes whalesalad google38438 bbbbb38438 frankfanslc pkafma-aon pgdatahome

hyperstart's Issues

add hostname into /etc/hostname

and fix the access mode of the /etc/hostname.
(it is now 0755, but expected 0644)

clean up hyper_create() and hyper_create_file().

bind mount the /dev/console

opencontainers/runtime-spec#518

change the type of /dev from devtmpfs to tmpfs

hyperstart uses devtmpfs for containers' /dev, but devtmpfs is the singleton in the whole system,
it is shared with all the containers and the rest of the system. we can convert it to tmpfs as the same as libcontainer.

Environment variables can't be found

This issue is filed by @feiskyer , the original bug link is hyperhq/hyperd#71

I created a pod using following spec:

{
    "containers": [
        {
            "envs": [
                {
                    "env": "SERVICE_HOST",
                    "value": "10.254.0.1"
                },
                {
                    "env": "SERVICE_PORT",
                    "value": "443"
                }
            ],
            "image": "nginx",
            "name": "nginx",
            "ports": [
                {
                    "containerPort": 80,
                    "protocol": "TCP"
                }
            ],
            "tty": false,
        }
    ],
    "resource": {
        "memory": 192,
        "vcpu": 1
    },
    "tty": true,
    "type": "pod",
}

But there is no SERVICE_HOST or SERVICE_PORT found in environment variables:

# hyper exec pod-TToRTANXBb env
HOME=/
TERM=linux
PATH=/bin:/sbin/:/usr/bin/:/usr/sbin/
PWD=/

add log level

hyperstart opened fds are leak into container.

/ # ls -l /proc/$$/fd
total 0
lrwx------    1 root     root            64 Aug 29 08:36 0 -> /dev/pts/0
lrwx------    1 root     root            64 Aug 29 08:36 1 -> /dev/pts/0
lrwx------    1 root     root            64 Aug 29 08:36 10 -> /dev/tty
lrwx------    1 root     root            64 Aug 29 08:36 2 -> /dev/pts/0
lrwx------    1 root     root            64 Aug 29 08:36 6 -> socket:[701]

socket:[701] is sure to be a leaking fd. it might be netlink fd. @gao-feng
/dev/tty is unsure what is it. It might be opened by busybox.

[root@sbox runv]# docker exec b3ce4aaa72c0 ls -l '/proc/self/fd'
total 0
lr-x------    1 root     root            64 Aug 29 08:42 0 -> pipe:[874]
l-wx------    1 root     root            64 Aug 29 08:42 1 -> pipe:[875]
l-wx------    1 root     root            64 Aug 29 08:42 2 -> pipe:[876]
ls: /proc/self/fd/3: cannot read link: No such file or directory
lr-x------    1 root     root            64 Aug 29 08:42 3
lrwx------    1 root     root            64 Aug 29 08:42 6 -> socket:[701]

what is the fd/3 ?

pod_init process's rootfs should be setup to an empty dir

otherwise container can access the vm-rootfs via /proc/1/root/ or /proc/1/cwd/

hyperstart stops on HUP event

Test log at http://ci.hypercontainer.io:8080/job/hyperd-auto/436/consoleFull

hyperstart got HUP event resulting container to stop. The source of HUP event needs further investigation.

stream protocol between runv and hyperstart

Current streams are multiplexed and sent/received via the serial port (named: "sh.hyper.channel.1")
And the format for multiplexing is:

/*
 * stream message format
 * | stream sequence | length  | payload (length-12)     |
 * | . . . . . . . . | . . . . | . . . . . . . . . . . . |
 * 0                 8         12                        length
 */
#define STREAM_HEADER_SIZE              12
#define STREAM_HEADER_LENGTH_OFFSET     8

Both stream sequence and length are encoding in bigendian.
And length=0 indicates the command of close the stream (only one direction)
(and there is an additional ugly data is sent from hyperstart to runv for the exitcode,
this part of the protocol will be removed soon(scheduled after the big refactor(hyperhq/runv#295)))

The major problem is that payload will be discarded in hyperstart if the buffer if full,
and stream service is blocked in runv if the buffer if full.

We need change the protocol after the big refactor(hyperhq/runv#295) as:

we assume serial port doesn't discard any data, and data is received as the same order as it was sent.
length(decoded) doesn't include the length of the header
the length of the payload must less than (1 << 30) (the practical length < 4096)
when A received length(decoded) == 0x80000000 | num. ACK-COMMAND: it means the opposite(B) side had just received and consumed num bytes of data(ack for the earlier message). A should record how much the data that B had received. A shouldn't send any more data to B util A get ALL the ack.
when A received length(decoded) == 0xC0000000. CLOSE-COMMAND: the B side close the stream, it meems the upstream fd in B is closed, the B will not send any message with payload, A should close the corresponding downstream fd if needed.
when A received length(decoded) == 0xC0000001. REQUEST-COMMAND: the B side request data, A should send data to B from the first unacked data. (B had discarded the unacked payload).

how to build a local kernel

I am using runv and need to use my own kernel for the vm rather than the default one of hyperstart.
Then I tried to build my own kernel using hyperstart.

run "make kernel-local" under hyperstart/build, then based on Dockerfile.x86_64, a kernel of 4.12.4 will be created
run "make initrd-local" under hyperstart/build to create the initrd
copy kernel and initrd to /var/lib/hyper

Is this the right way to build local kernel?

all containers need to share the same /dev/shm

all containers need to share the same /dev/shm, the same tmpfs mount with bindmount for sharing.
since all containers share the same ipc ns in the sandbox.

More investigation is needed.

ref: setupIpcDirs() in moby/daemon/container_operations_unix.go

process id and console-resize

rename process->id to process->container
use process->id as process id, also modify the parser. (runv site can be changed later if possible)
console-resize via process id, not stream id(seq)

prevent the container creates processes endless when hyperstart destroy the container

if the container creates the processes very frequent and quickly. the looping killer(#96) can not kill all the processes.
Is there any proposal to solve it? (it is not bug, it is user's responsibility if the container can't end, but it will be better if hyperstart can help).

(in my test, it loops util the test script stopped by OOM, if the script does fork processes carefully, the container will not end.)

see #96

Additional groups handling broken when working with runv and docker

In exec.c:265:

	// append additional groups to supplementary groups
	int i;
	reallocgroups = realloc(groups, sizeof(gid_t) * (ngroups + exec->nr_additional_groups));
	if (reallocgroups == NULL)
		goto fail;
	groups = reallocgroups;
	for (i = 0; i < exec->nr_additional_groups; i++) {
		fprintf(stdout, "try to find the group: %s\n", exec->additional_groups[i]);
		struct group *gr = hyper_getgrnam(exec->additional_groups[i]);
		if (gr == NULL) {
			perror("can't find additional group");
			goto fail;
		}
		groups[ngroups] = gr->gr_gid;
		ngroups++;
	}

This stanza of code is broken when using runv with docker - the problem is docker passes in a huge stanza of additional groups by default from the user running it (docker 1.12.1 at least):

I0106 16:43:13.550083   25707 vm_console.go:46] SB[vm-rDTiYKRrYK] [CNL] call hyper_new_container, json {"id":"1fa52cef9d6a715e905ae2c07e178f31ca87671a0c706f690a1f973717b2286e","rootfs":"rootfs","image":"1fa52cef9d6a715e905ae2c07e178f31ca87671a0c706f690a1f973717b2286e","process":{"id":"init","additionalGroups":["0","1","2","3","4","6","10","11","20","26","27"],"terminal":true,"stdio":1,"args":["sleep","10"],"envs":[{"env":"PATH","value":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"},{"env":"HOSTNAME","value":"1fa52cef9d6a"},{"env":"TERM","value":"xterm"}],"workdir":"/"},"restartPolicy":"never","initialize":false}

Obviously none of these groups exist in the client container, so the lookup fails and the group startup fails.

Probably the way this needs to be handled is do the lookup, and if it fails, try a string to int conversion, and if that succeeds use the integer number without the lookup. If both succeed for some reason, then fail.

There's a secondary issue which is a failure in this part of the code does not cleanly message back to the runv containerd daemon, so docker gets stuck waiting for a container to start without failing.

Add script to build the kernel

Add script to build the kernel with the config in a hypercontainer or native environment.

A Dockerfile is also required for defining the hypercontainer's image.

apt-get update failed on overlay storage driver

Step 6 : RUN apt-get update
---> Running in 33968afa2cb0
Ign http://archive.ubuntu.com trusty InRelease
Get:1 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB]
E: Unable to determine file size for fd 7 - fstat (2: No such file or directory)

don't over mount on /etc/hostname

don't over mount on /etc/hostname when run container for docker, since docker provides and is keeping updating this file.

Problem1: how to distinguish whether /etc/hostname ?
Problem2: hyperstart should be also notified when hostname need to be updated. hyperstart can also update this /etc/hostname. (So, is this issue a real problem?)

kernel: enable cgroup and netfilter

[Reference] vsock NFS support

NFS via vsock is under developing, and is promising for the future FS sharing of hyper, because:

NFS has much better performance/feature set than 9p
vsock doesn't need to add a network interface to guest, and then doesn't setup a network connection between host and guest, i.e. better isolation than open a network connection to every guest.
and it may also substitute the virt-serial for communication

Some blocker of adopting vsock NFS

the tool set is under developing:
the xen doesn't support vsock yet, i mean xenbus based transport, once it is supported xenbus, we can use it to provide FS sharing in PV mode.

Some references:

The initial patch set: https://lwn.net/Articles/647516/
Slides: http://events.linuxfoundation.org/sites/events/files/slides/stefanha-kvm-forum-2015.pdf
qemu wiki: http://qemu-project.org/Features/VirtioVsock

Related repos:

kernel, vsock-nfs: https://github.com/stefanha/linux/tree/vsock-nfs
user space tools: https://github.com/stefanha/nfs-utils
qemu vsock support: https://github.com/stefanha/qemu/tree/vsock

Support for stopping container only

I'll write some code to make hyperstart support such situation that
there are more than one container in the pod and I want to stop one of these containers and the pod is still running.

clang3.9.1 util.c - comparison of unsigned expression < 0 is always false

Environment

Fedora 26 x86_64
clang 3.9.1
git faef420

Error

/usr/bin/clang -DHAVE_CONFIG_H -I. -I..    -Wall -Werror -DVERSIONCOMMIT=\""0.8.1, commit: `git describe --dirty --always --tags 2> /dev/null || true`"\" -g -O2 -MT util.o -MD -MP -MF $depbase.Tpo -c -o util.o util.c &&\
mv -f $depbase.Tpo $depbase.Po
util.c:251:9: error: comparison of unsigned expression < 0 is always false [-Werror,-Wtautological-compare]
                if (l < 0) {
                    ~ ^ ~
1 error generated.
make[2]: *** [Makefile:362: util.o] Error 1

gcc7 util.c - ‘%s’ directive writing up to 255 bytes into a region of size 232

Environment

Fedora 26 x86_64
gcc 7.1.1
git hash faef420

Error

gcc -DHAVE_CONFIG_H -I. -I..    -Wall -Werror -DVERSIONCOMMIT=\""0.8.1, commit: `git describe --dirty --always --tags 2> /dev/null || true`"\" -g -O2 -MT util.o -MD -MP -MF $depbase.Tpo -c -o util.o util.c &&\
mv -f $depbase.Tpo $depbase.Po
util.c: In function ‘online_cpu’:
util.c:514:42: error: ‘%s’ directive writing up to 255 bytes into a region of size 232 [-Werror=format-overflow=]
   sprintf(path, "/sys/devices/system/cpu/%s/online", entry->d_name);
                                          ^~
util.c:514:3: note: ‘sprintf’ output between 32 and 287 bytes into a destination of size 256
   sprintf(path, "/sys/devices/system/cpu/%s/online", entry->d_name);
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
util.c: In function ‘online_memory’:
util.c:550:45: error: ‘%s’ directive writing up to 255 bytes into a region of size 229 [-Werror=format-overflow=]
   sprintf(path, "/sys/devices/system/memory/%s/online", entry->d_name);
                                             ^~
util.c:550:3: note: ‘sprintf’ output between 35 and 290 bytes into a destination of size 256
   sprintf(path, "/sys/devices/system/memory/%s/online", entry->d_name);
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Configure multiple interfaces

The SETUPINTERFACE command parses the json for a single interface and configures it. It would be nice to have a command (setupmultiinterface/setupinterfaces?) to configure multiple interfaces or modify the existing command to support multiple interfaces. Helps reduce the round trips in case of multiple interfaces. I see that the SETUPROUTE command, currently supports multiple routes.

cannot find vsock device

cannot find vsock device
scan /sys/class/virtio-ports/ failed: No such file or directory
cannot find ctl channel
fail to setup hyper serial channel

yum failed even with yum-plugin-ovl

looks this is not the same with #74 , and I have already applied #86 and #87

[root@centos-rpm-build SPECS]# yum install strace
Loaded plugins: fastestmirror, ovl
base                                                     | 3.6 kB     00:00
extras                                                   | 3.4 kB     00:00
updates                                                  | 3.4 kB     00:00
Loading mirror speeds from cached hostfile
 * base: mirror.stjschools.org
 * extras: mirror.keystealth.org
 * updates: mirror.hmc.edu
Resolving Dependencies
--> Running transaction check
---> Package strace.x86_64 0:4.8-11.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package          Arch             Version                 Repository      Size
================================================================================
Installing:
 strace           x86_64           4.8-11.el7              base           265 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 265 k
Installed size: 950 k
Is this ok [y/d/N]: y
Downloading packages:
strace-4.8-11.el7.x86_64.rpm                               | 265 kB   00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Installing : strace-4.8-11.el7.x86_64                                     1/1


Rpmdb checksum is invalid: dCDPT(pkg checksums): strace.x86_64 0:4.8-11.el7 - u

There is a warning: Warning: RPMDB altered outside of yum., and failed at last

[discuss] convert events(src/event.c src/event.h) to libevent

libevent is much mature and easy to use.

event read handling refactor

move the two block of common code(the two loops) in hyper_event_read() into nonblock_read() (don't contain the ack-code)
use different read-handlers for the ctl & tty channel and remove ops->len_offset
remove ops->ack, directly do it in the read-handlers
remove event->rbuf, use static buf in the read-handler functions(or static pointer if the buffer needs to be extend)
remove ops->handle directly do/call it in the read-handlers

/dev/net/tun device is missing from hyperstart

hyperstart doesn't setup the /dev/net/tun interface inside containers, which means using tun devices does not work unless it is explicitely setup in the image:

mkdir /dev/net
mknod /dev/net/tun c 10 200

solves the problem, but it seems like hyperstart should probably handle this?

fd event simplify

remove rbuf, put them on static variable or container structure(access it via container_of()->wbuf)
remove wbuf, put them on static variable or container structure(access it via container_of()->wbuf)
add field for event interesting, and manipulate the events based on event interesting

http://www.visualops.io/还维护么

麻烦问一下，现在http://www.visualops.io/还维护么，注册总是返回401。谢谢！

container init was failed to kill if signal comes before container init executed

Let me describe the detail:

(1) init new container
        |
       \/
(2) fork() ----->(3b) fork again for namespace ----> (B) init stdio & exec process
        |
       \/
(3a) return success
        |
       \/
(A) next command (signal container with SIGKILL)

Some failed logs:

The (A) was shown (signal and return normally) in log before (B) (hyper_install_process_stdio ), and the container process was not killed.

Detailed logs:

fork and success:

SB[vm-QGyxKjgyXa] [CNL] hyper send type 8
SB[vm-QGyxKjgyXa] [CNL] prerequisite process pid 332
SB[vm-QGyxKjgyXa] [CNL] create child process pid=333 in the sandbox
SB[vm-QGyxKjgyXa] [CNL] hyper send type 333
SB[vm-QGyxKjgyXa] [CNL] hyper_init_event exec stdin event 0x867eb8, ops 0x61d5c0, fd 7
SB[vm-QGyxKjgyXa] [CNL] hyper_add_event add event fd 7, 0x61d5c0
SB[vm-QGyxKjgyXa] [CNL] hyper_init_event exec stdout event 0x867ef8, ops 0x61d5a0, fd 8
SB[vm-QGyxKjgyXa] [CNL] hyper_add_event add event fd 8, 0x61d5a0
SB[vm-QGyxKjgyXa] [CNL] hyper_init_event exec stderr event 0x867f38, ops 0x61d580, fd 11
SB[vm-QGyxKjgyXa] [CNL] hyper_add_event add event fd 11, 0x61d580
SB[vm-QGyxKjgyXa] [CNL] hyper_run_process process pid 333
SB[vm-QGyxKjgyXa] [CNL] hyper ctl append type 9, len 0
SB[vm-QGyxKjgyXa] [CNL] hyper_handle_event event EPOLLOUT, he 0x61d648, fd 3, 0x61d4c0

then, kill and success:

SB[vm-QGyxKjgyXa] [CNL] hyper_handle_event event EPOLLIN, he 0x61d648, fd 3, 0x61d4c0
SB[vm-QGyxKjgyXa] [CNL] hyper ctl append type 14, len 4
SB[vm-QGyxKjgyXa] [CNL] hyper_modify_event modify event fd 3, 0x61d648, event 8197
SB[vm-QGyxKjgyXa] [CNL] hyper_ctlfd_read: get length 116
SB[vm-QGyxKjgyXa] [CNL] hyper ctl append type 14, len 4
SB[vm-QGyxKjgyXa] [CNL] hyper_ctlmsg_handle, type 24, len 116
SB[vm-QGyxKjgyXa] [CNL] hyper ctl append type 9, len 0
SB[vm-QGyxKjgyXa] [CNL] hyper_handle_event event EPOLLOUT, he 0x61d648, fd 3, 0x61d4c0

type 24 is SIGNALPROCESS, and the command payload is

{"container":"d9f81367f4ad974fb2ba3f60f38917602d28914a15764f01c117be082aad9e39","process":"init","signal":9}

then, initialize container process:

SB[vm-QGyxKjgyXa] [CNL] pid 332 exit normally, status 0
SB[vm-QGyxKjgyXa] [CNL] hyper_install_process_stdio

after test timeout, the daemon and pod was killed (again):

(more than 9 minutes later, timeout)

SB[vm-QGyxKjgyXa] [CNL] hyper ctl append type 14, len 4
SB[vm-QGyxKjgyXa] [CNL] hyper_modify_event modify event fd 3, 0x61d648, event 8197
SB[vm-QGyxKjgyXa] [CNL] hyper_ctlfd_read: get length 117
SB[vm-QGyxKjgyXa] [CNL] hyper ctl append type 14, len 4
SB[vm-QGyxKjgyXa] [CNL] hyper_ctlmsg_handle, type 24, len 117
SB[vm-QGyxKjgyXa] [CNL] hyper ctl append type 9, len 0
SB[vm-QGyxKjgyXa] [CNL] hyper_handle_event event EPOLLOUT, he 0x61d648, fd 3, 0x61d4c0
SB[vm-QGyxKjgyXa] [CNL] hyper_handle_event event EPOLLHUP or EPOLLERR, he 0x867f38, fd 11, 10
SB[vm-QGyxKjgyXa] [CNL] stderr_hup, seq 1, id init
SB[vm-QGyxKjgyXa] [CNL] still have 3 user of exec
SB[vm-QGyxKjgyXa] [CNL] hyper_handle_event event EPOLLHUP or EPOLLERR, he 0x867ef8, fd 8, 10
SB[vm-QGyxKjgyXa] [CNL] stdout_hup, seq 1, id init
SB[vm-QGyxKjgyXa] [CNL] still have 2 user of exec
SB[vm-QGyxKjgyXa] [CNL] hyper_handle_event event EPOLLHUP or EPOLLERR, he 0x867eb8, fd 7, 8
SB[vm-QGyxKjgyXa] [CNL] stdin_hup, seq 1, id init
SB[vm-QGyxKjgyXa] [CNL] still have 1 user of exec
SB[vm-QGyxKjgyXa] [CNL] pid 333 exit by signal, status 15
SB[vm-QGyxKjgyXa] [CNL] hyper_handle_exec_exit exec exit pid 333, seq 1, container d9f81367f4ad974fb2ba3f60f38917602d28914a15764f01c117be082aad9e39
SB[vm-QGyxKjgyXa] [CNL] container init process 333
SB[vm-QGyxKjgyXa] [CNL] last user of exec exit, release

complete logs:

Failed: http://ci.hypercontainer.io:8080/job/hyperd-auto/288/consoleFull
Success: http://ci.hypercontainer.io:8080/job/hyperd-auto/295/consoleFull

add rlimits

hyperhq/runv@04cbac7

guest kernel crashed when hyperstart handle SETUPROUTE without network

While hyperstart try to run command to setup route, the guest kernel will crash:
hyper_ctlmsg_handle SETUPROUTE
init[1]: unhandled level 2 translation fault (11) at 0x00000000, esr 0x92000006
pgd = ffffffc005767000
[00000000] *pgd=0000000085769003, *pud=0000000085769003
, *pmd=0000000000000000

CPU: 0 PID: 1 Comm: init Not tainted 4.9.36 #3
Hardware name: linux,dummy-virt (DT)
task: ffffffc00744ad00 task.stack: ffffffc00744c000
PC is at 0x406ba8
LR is at 0x4079f0
pc : [<0000000000406ba8>] lr : [<00000000004079f0>] pstate: 60000000
sp : 0000007ffec98cb0
x29: 0000007ffec98cb0 x28: 0000000000000000
x27: 000000000042c000 x26: 000000002f2131c0
x25: 0000000000000015 x24: 0000000000416000
x23: 000000000042c000 x22: 0000007ffec99170
x21: 000000000042c000 x20: 000000000042c000
x19: 0000000000000000 x18: 0000000000000001
x17: 0000007f95058988 x16: 000000000042c2a8
x15: 0000000000000001 x14: 0000000000000003
x13: 0000000000417b58 x12: 00000000ffffffff
x11: 000000000000000a x10: 0000000000000000
x9 : 0000000000000001 x8 : 00000000ffffffff
x7 : 0000000000000002 x6 : 000000002f2131d0
x5 : 0000000000000001 x4 : 0000000000000001
x3 : 0000000000000000 x2 : 000000000042c448
x1 : 000000000042c640 x0 : 0000000000000000

Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

CPU: 0 PID: 1 Comm: init Not tainted 4.9.36 #3
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x198
[] show_stack+0x14/0x20
[] dump_stack+0x94/0xb8
[] panic+0x110/0x258
[] complete_and_exit+0x0/0x20
[] do_group_exit+0x38/0xa8
[] get_signal+0x234/0x4d0
[] do_signal+0x354/0x508
[] do_notify_resume+0x90/0xb0
[] work_pending+0x8/0x10
Kernel Offset: disabled
Memory Limit: none
---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

E0904 15:08:58.579042 2185 network.go:184] rpc error: code = Internal desc = transport is closing
E0904 15:09:58.939408 2185 vm_states.go:246] SB[vm-hBJcxNbNru] Shutting down because of an exception: %!(EXTRA string=Destroy pod failed: &grpc.rpcError{code:0xe, desc:"grpc: the connection is unavailable"})
Create new container failed: rpc error: code = Internal desc = transport is closing

We have created a PR# #342 to fix this issue.

use defined instant of libc.so for initrd

don't use the host /lib64/libc.so

the build system should ensure every one creates the initrd with the same libs.

9p rootfs become read-only if restart a container

During testing hyperhq/hyperd#585 (fix hyperhq/hyperd#539), hyperstart #290 failed when restart container, while the restart works with #287 .

#287 result:

➜ sudo ./hyperctl list container
Container ID                                                       Name                 POD ID               Status
2c8abf13b126d71e8fdfdf21d117633178f6f2635f4d884f3179c3e7e306a54c   busybox-9326359365   busybox-9326359365   succeeded
➜ sudo ./hyperctl start -c busybox-9326359365
Successfully started container busybox-9326359365
➜ sudo ./hyperctl attach busybox-9326359365

/ # ls
bin   dev   etc   home  lib   proc  root  sys   tmp   usr   var
/ # exit
➜

#290 result:

➜ sudo ./hyperctl list container
Container ID                                                       Name                 POD ID               Status
3ccb3256b304f77602b159cdb75dd748e9675d8e2fc6526c94ba31296888fd01   busybox-7322297518   busybox-7322297518   succeeded
➜ sudo ./hyperctl start -c busybox-7322297518
./hyperctl ERROR: Error from daemon's response: Create new container failed: Error:
255 ➜

console logs:

I0417 23:24:00.703250 [CNL] hyper_handle_event event EPOLLIN, he 0x61b5a8, fd 3, 0x61b460
I0417 23:24:00.704053 [CNL] hyper ctl append type 14, len 4
I0417 23:24:00.705311 [CNL] hyper_modify_event modify event fd 3, 0x61b5a8, event 8197
I0417 23:24:00.706092 [CNL] hyper_ctlfd_read: get length 482
I0417 23:24:00.706898 [CNL] hyper ctl append type 14, len 4
I0417 23:24:00.707836 [CNL] hyper_ctlmsg_handle, type 17, len 482
I0417 23:24:00.716334 [CNL] call hyper_new_container, json {"id":"3ccb3256b304f77602b159cdb75dd748e9675d8e2fc6526c94ba31296888fd01","rootfs":"rootfs","image":"/3bb3a3600dd818aec2c01ba2ffc474442ab43c3c30fff69fb5e7a63cf73a4a3f","fsmap":[{"source":"JbzXCiRQfJ","path":"/etc/hosts","readOnly":false,"dockerVolume":false}],"process":{"id":"init","terminal":true,"stdio":2,"args":["sh"],"envs":[{"env":"PATH","value":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"}],"workdir":"/"},"restartPolicy":"never","initialize":true}, len 474
I0417 23:24:00.717478 [CNL] hyper send mntns referenced event: normal
I0417 23:24:00.718539 [CNL] create child process pid=338 in the sandbox
I0417 23:24:00.720520 [CNL] src directory /tmp/hyper/shared//3bb3a3600dd818aec2c01ba2ffc474442ab43c3c30fff69fb5e7a63cf73a4a3f/
I0417 23:24:00.723649 [CNL] root directory for container is /tmp/hyper/3ccb3256b304f77602b159cdb75dd748e9675d8e2fc6526c94ba31296888fd01/root//rootfs, init task sh
I0417 23:24:00.724620 [CNL] recreate file ./etc/hosts
I0417 23:24:00.725966 [CNL] recreate file ./etc/hostname
I0417 23:24:00.727686 [CNL] recreate symlink ./etc/mtab to /proc/mounts
I0417 23:24:00.729321 [CNL] container sets up init layer failed
I0417 23:24:00.730327 [CNL] hyper send container inited event: error
I0417 23:24:00.731311 [CNL] wait for setup container rootfs failed
I0417 23:24:00.733253 [CNL] create child process pid=340 in the sandbox
I0417 23:24:00.734420 [CNL] fail to enter container ns: Bad file descriptor
I0417 23:24:00.735414 [CNL] hyper send enter container ns event: error
I0417 23:24:00.736197 [CNL] hyper ctl append type 10, len 0
I0417 23:24:00.737698 [CNL] hyper_handle_event event EPOLLOUT, he 0x61b5a8, fd 3, 0x61b460

convet jsmn to parson gradually

refer: https://github.com/kgabis/parson
example: #123
apis: https://github.com/kgabis/parson/blob/master/parson.h

JSON_Value *value = hyper_json_parse(json, length);              #parse, return root value
json_object_get_string(json_object(value), "container");            # retrieve a filed
(int)json_object_get_number(json_object(value), "signal");
json_object_dotget_string(json_object(value), "container.process.workdir"); # retrieve a filed via dot
json_value_free(value);                         # free

JSON_Value  * json_object_get_value  (const JSON_Object *object, const char *name);
const char  * json_object_get_string (const JSON_Object *object, const char *name);
JSON_Object * json_object_get_object (const JSON_Object *object, const char *name);
JSON_Array  * json_object_get_array  (const JSON_Object *object, const char *name);
double        json_object_get_number (const JSON_Object *object, const char *name); /* returns 0 on fail */
int           json_object_get_boolean(const JSON_Object *object, const char *name); /* returns -1 on fail */

/* dotget functions enable addressing values with dot notation in nested objects,
 just like in structs or c++/java/c# objects (e.g. objectA.objectB.value).
 Because valid names in JSON can contain dots, some values may be inaccessible
 this way. */
JSON_Value  * json_object_dotget_value  (const JSON_Object *object, const char *name);
const char  * json_object_dotget_string (const JSON_Object *object, const char *name);
JSON_Object * json_object_dotget_object (const JSON_Object *object, const char *name);
JSON_Array  * json_object_dotget_array  (const JSON_Object *object, const char *name);
double        json_object_dotget_number (const JSON_Object *object, const char *name); /* returns 0 on fail */
int           json_object_dotget_boolean(const JSON_Object *object, const char *name); /* returns -1 on fail */

JSON_Value  * json_array_get_value  (const JSON_Array *array, size_t index);
const char  * json_array_get_string (const JSON_Array *array, size_t index);
JSON_Object * json_array_get_object (const JSON_Array *array, size_t index);
JSON_Array  * json_array_get_array  (const JSON_Array *array, size_t index);
double        json_array_get_number (const JSON_Array *array, size_t index); /* returns 0 on fail */
int           json_array_get_boolean(const JSON_Array *array, size_t index); /* returns -1 on fail */
size_t        json_array_get_count  (const JSON_Array *array);

Can't create pod

[HYPER INFO  1105 01:43:^@41 00456 pod.go] [:298] Process the Containers section in POD SPEC
[HYPER INFO  1105 01:43:^@41 00456 pod.go] [:301] trying to init container kube_5e212388-8385-11e5-bb0e-063553ee1923_influxdb-grafana-ex8d0_kube-system-service-discovery
[HYPER INFO  1105 01:43:^@41 00456 tags.go] [:133] LookupImage Name is haproxy:latest
[HYPER INFO  1105 01:43:^@41 00456 server.go] [:1052] Calling GET /list
[HYPER INFO  1105 01:43:^@41 00456 server.go] [:190] List type is pod, specified pod: [], list auxiliary pod:
[HYPER INFO  1105 01:43:^@41 00456 job.go] [:78] +job list(pod, , )
[HYPER INFO  1105 01:43:^@41 00456 container.go] [:10] ready to get the container(4a0753465d925c5237e21a53c05dce7d1dbf09b2dba9d56f63f4c3f3810337df) info
[HYPER INFO  1105 01:43:^@41 00456 pod.go] [:301] trying to init container kube_5e212388-8385-11e5-bb0e-063553ee1923_influxdb-grafana-ex8d0_kube-system_influxdb.cb712a1b_47d8cbdf
[HYPER INFO  1105 01:43:^@41 00456 tags.go] [:133] LookupImage Name is kubernetes/heapster_influxdb:v0.5
[HYPER ERROR 1105 01:43:^@41 00456 pod.go] [:315] mkdir /var/lib/hyper/overlay/6211ea67e7be674977fa0826145ab159ffa4f00c0bf7fb8c3fb365275403c030-init/merged/dev/shm: invalid argument

Support docker in hyper container

I try to run docker in hyper container, but failed. The problems are:

Storage driver
No storage driver (dm, aufs, overlay, etc...) is supported by hyper kernel.
Bridge module

bash-4.1# /sbin/modprobe  -v bridge
install /sbin/modprobe --ignore-install bridge && /sbin/sysctl -q -w net.bridge.bridge-nf-call-arptables=0 net.bridge.bridge-nf-call-iptables=0 net.bridge.bridge-nf-call-ip6tables=0
FATAL: Module bridge not found.
FATAL: Error running install command for bridge
bash-4.1#

create pts with group=tty=5

see https://www.togaware.com/linux/survivor/Standard_Groups.html

tty	5	The terminal devices with names beginning with /dev/tty are group accessible to group tty. Programs such as write and wall need access to /dev/tty and they set their group id (sgid) to tty.

add process capabilities configuration

Although vm is strong isolation, but capabilities might be still need for some other cases...

setup HOME to /home/$USER if HOME is not set and USER confng is set.

Document the host ⟷ hyperstart protocol

It'd be nice to document the protocol between the host and hyperstart for projects wanted to explore hyperstart. In particular:

the CTL and I/O serial links
READY packet(s)
framing protocol to interleave multiple command streams & I/O streams onto the 2 serial links
list of commands on the CTL serial (http://github.com/hyperhq/runv/tree/master/hyperstart/api/json)
flow between the commands (eg. when STARTPOD should be sent, ...)
maybe talk about WITH_VBOX

Failed to change owner/group of file with overlay/aufs driver

[root@centos-0318811974 /]# chgrp adm /a
chgrp: changing group of '/a': Operation not permitted

found in centos and ubuntu image, overlay and aufs driver.

and

[root@h8s-single centos]# grep clear /etc/libvirt/qemu.conf
clear_emulator_capabilities = 0

Is the source for Hyperkernel around somewhere?

So this is the source for init, but I would also like to see the source for Hyperkernel. Is it possible to have the source for that as well? I'm mainly just curious about which build system was used for it - Buildroot? Yocto? Something custom? Thanks in advance!

failed to build

Hello all,
I'm newbie on this project.
and I got below error on build time.

$ make
make all-recursive
make[1]: Entering directory '/home/keyolk/.gvm/pkgsets/go1.6.2/container/src/github.com/hyperhq/hyperstart'
Making all in src
make[2]: Entering directory '/home/keyolk/.gvm/pkgsets/go1.6.2/container/src/github.com/hyperhq/hyperstart/src'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/keyolk/.gvm/pkgsets/go1.6.2/container/src/github.com/hyperhq/hyperstart/src'
Making all in build
make: relocation error: libm.so.6: symbol __get_cpu_features, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference
make[1]: *** [Makefile:352: all-recursive] Error 1
make[1]: Leaving directory '/home/keyolk/.gvm/pkgsets/go1.6.2/container/src/github.com/hyperhq/hyperstart'
make: *** [Makefile:293: all] Error 2

$ objdump -T /usr/lib/libc.so.6 | grep GLIBC_PRIVATE | grep cpu
00000000000f4300 g DF .text 0000000000000036 GLIBC_PRIVATE __clock_getcpuclockid

to solve this,
is there any something workaround ?

remove hyper_symlink() and hyper_unlink()

directly use symlink() and unlink().

hyper_create() also needs to be removed. see #204

How to build qboot image?

Is there any tutorial that could tell us how to build a qboot image?

hyperstart cannot create network adapter on ppc64le

The init binary executes /usr/sbin/rtas_errd to configure the VM network adapters in ppc64le. This comes from build/arch/ppc64le/binary/rtas.tar in the repository but the shared libraries it requires are not copied into lib64 in build/make-initrd.sh resulting in these errors in the vmconsole output:

I0417 14:24:20.944302   47891 watcher.go:74] vmconsole: executing cmd /usr/sbin/rtas_errd -d
I0417 14:24:20.945676   47891 watcher.go:74] vmconsole: create directory /tmp/hyper/shared
I0417 14:24:20.946671   47891 watcher.go:74] vmconsole: /usr/sbin/rtas_errd: error while loading shared libraries: librtas.so.2: cannot open shared object file: No such file or directory
I0417 14:24:20.946690   47891 watcher.go:74] vmconsole: create directory /tmp/hyper/shm
I0417 14:24:20.946824   47891 watcher.go:74] vmconsole: pod init pid 1707
I0417 14:24:20.947125   47891 watcher.go:74] vmconsole: /usr/sbin/rtas_errd -d cmd exit normally, status 127
I0417 14:24:20.947243   47891 watcher.go:74] vmconsole: cmd /usr/sbin/rtas_errd -d exit unexpectedly, status 32512
I0417 14:24:20.947359   47891 watcher.go:74] vmconsole: rtas start failed: No such file or directory

No pid-namespacing of containers sharing a pod

Hyperstart doesn't implement pid-namespacing when containers share a pod (i.e. when you have shared container network namespaces).

make-initrd.sh vbox failed

make will failed since 'make-initrd.sh' has more unexpectledy 'cd ../'.

[ray@localhost hyperstart]$ make
......
make[2]: Entering directory '/home/ray/go/src/github.com/hyperhq/hyperstart/build'
bash ./make-initrd.sh vbox
build initrd for vbox
28547 blocks
cp: cannot stat '../vbox/kernel': No such file or directory
cp: cannot stat '../hyper-initrd.img': No such file or directory
cp: cannot stat '../vbox/isolinux/isolinux.bin': No such file or directory
cp: cannot stat '../vbox/isolinux/ldlinux.c32': No such file or directory
I: -input-charset not specified, using utf-8 (detected in locale settings)
genisoimage: Uh oh, I cant find the boot image 'isolinux/isolinux.bin' !
make[2]: Leaving directory '/home/ray/go/src/github.com/hyperhq/hyperstart/build'
make[2]: Entering directory '/home/ray/go/src/github.com/hyperhq/hyperstart'
make[2]: Leaving directory '/home/ray/go/src/github.com/hyperhq/hyperstart'
make[1]: Leaving directory '/home/ray/go/src/github.com/hyperhq/hyperstart'