[feature request] support lxcfs in pouch to improve isolation about pouch

[CodeJuan]

create container

1. start pouchd with argument --enable-lxcfs
2. pouchd run lxcfs process
3. create container with parameter lxcfs
4. bind /var/lib/lxc and /proc

+---------------------+
|                     |        +------------+
|                     | 2 run  |            |
|                     +--------+  lxcfs     +-------------------------+
|         1           |        |            |                         |
|pouchd --enable+lxcfs|        +------------+                   +-----+-------------------+
|                     |                                         |                         |
|                     |                             +------------>/var/lib/lxc/lxcfs      |
|                     |                             |           |                         |
+-+-------------------+                             |     +------>/var/lib/lxc/lxcfs/proc |
  |                                                 |     |     |                         |
  |                                                 |     |     +-------------------------+
  |                                                 |     |
  |                                                 |     |
  |                                                 |     |
  |                                                 |     |
  |               +---------------------------+     |     |
  |  3            |                           |     |     |
  | create        | Container                 |     |     |
  | --lxcfs=True  |                           |     |     |
  +--------------->                           |     |     | 4 bind
                  |                           |     |     |
                  |                           |     |     |
                  |                           |     |     |
                  |    +----------------------+     |     |
                  |    | /var/lib/lxc+--------------+     |
                  |    |                      |           |
                  |    | /proc/meminfo        |           |
                  |    | /proc/cpuinfo+-------------------+
                  |    | ...                  |
                  |    | ...                  |
                  |    |                      |
                  +----+----------------------+

remount

1. when lxcfs stopped(or crashed) and restarted
2. run the remount script to remount /proc in container

+---------------------+  1
|                     | lxcfs  +------------+
|                     |restart |            |
|                     <--------+  lxcfs     +-------------------------+
|                     |        |            |                         |
|pouchd --enable+lxcfs|        +------------+                   +-----+-------------------+
|                     |                                         |                         |
|                     |                             +------------>/var/lib/lxc/lxcfs      |
|                     |                             |           |                         |
+-+-------------------+                             |     +------>/var/lib/lxc/lxcfs/proc |
  |                                                 |     |     |                         |
  |                                                 |     |     +-------------------------+
  |                                                 |     |
  |                                                 |     |
  |                                                 |     |
  |                                                 |     |
  |               +---------------------------+     |     |
  |               |                           |     |     |
  |   2           | Container                 |     |     |
  | remount       |                           |     |     |
  +--------------->                           |     |     |
                  |                           |     |     |
                  |                           |     |     |
                  |                           |     |     |
                  |    +----------------------+     |     |
                  |    | /var/lib/lxc+--------------+     |
                  |    |                      |           |
                  |    | /proc/meminfo        |           |
                  |    | /proc/cpuinfo+-------------------+
                  |    | ...                  |
                  |    | ...                  |
                  |    |                      |
                  +----+----------------------+

[CodeJuan]

I found a non-intrusive method to integrate lxcfs with Pouch.

1. Bind the path of lxcfs and the /proc to container

-v /var/lib/lxc/:/var/lib/lxc/:shared 
-v /var/lib/lxc/lxcfs/proc/uptime:/proc/uptime 
-v /var/lib/lxc/lxcfs/proc/swaps:/proc/swaps 
-v /var/lib/lxc/lxcfs/proc/stat:/proc/stat 
-v /var/lib/lxc/lxcfs/proc/diskstats:/proc/diskstats 
-v /var/lib/lxc/lxcfs/proc/meminfo:/proc/meminfo 
-v /var/lib/lxc/lxcfs/proc/cpuinfo:/proc/cpuinfo

2. If the lifecycle of lxcfs is managed by systemd, just add a remount script to lxcfs.service.ExecStartPost

# /etc/systemd/system/multi-user.target.wants/lxcfs.service

[Unit]
Description=FUSE filesystem for LXC
ConditionVirtualization=!container
Before=lxc.service
Documentation=man:lxcfs(1)

[Service]
ExecStart=/usr/bin/lxcfs /var/lib/lxc/lxcfs/
KillMode=process
Restart= always
ExecStopPost=-/bin/fusermount -u /var/lib/lxc/lxcfs
Delegate=yes

# add remount script
ExecStartPost=/usr/local/bin/container_remount_lxcfs.sh

[Install]
WantedBy=multi-user.target

3. When lxcfs restarted, systemd would run the remount script to remount /proc in container

[g0194776]

@senthilnathann @Billyxxu
We've solved the problem. You have to use 2 layers folder path to mount lxcfs directory into your container.

for example:

OLD: /usr/local/var/lib/lxcfs

NEW: /usr/local/var/lib/lxc/lxcfs

and...please mount path /usr/local/var/lib/lxc into your container. After lxcfs crashed and restarted, You can use bash command to remount it again.

[allencloud]

Yeah, thanks for your design. In your architecture, I found that you enabled this flag in pouchd. If that, I am afraid all containers will be influenced by lxcfs.
I am wondering if we could add the functionality that enable every single container by pouch create --enable-lxcfs or something like other?
@CodeJuan

[CodeJuan]

@allencloud Great idea. Please have a look at the updated diagram.

[CodeJuan]

According to the issue from lxcfs Anyone think about the problem of restart the lxcfs.service, when lxcfs stopped(or crashed) and restarted, it would be unmounted in the process and caused an exception in containerTransport endpoint is not connected. So we should monitor the lxcfs mount/umount event, and exec remount command in container.

[CodeJuan]

pseudocode of container_remount_lxcfs.sh

LXCFS="/var/lib/lxc/lxcfs"

containers=$(pouch ps | awk '{print $2}' | grep -v Name)
for container in $containers;do
        echo "remount $container"
        PID=$(pouch inspect $container | grep Pid | awk '{print $2}' | awk -F ',' '{print $1}' )
        for file in meminfo cpuinfo stat uptime swaps diskstats;do
                nsenter --target $PID --mount --  mount -B "$LXCFS/proc/$file" "/proc/$file"
        done
 done

[allencloud]

Set this in 0.1 milestone. And we are supposed to finish before Jan 10. @CodeJuan 😄

[senthilnathann]

Hi ,

Not able to remount the proc files inside the containers ,
whil executing below command ,

nsenter --target 8799 --mount -- mount -B /usr/local/var/lib/lxcfs/proc/meminfo /proc/meminfo

mount: mount /usr/local/var/lib/lxcfs/proc/meminfo on /proc/meminfo failed: Transport endpoint is not connected

Pls check and help

[fanhaozzu]

@senthilnathann me too,@CodeJuan help?

[fanhaozzu]

[senthilnathann]

@CodeJuan Pls help on this

[xigang]

@senthilnathann @fanhaozzu Please use this script:)
https://github.com/xigang/lxcfs-admission-webhook/blob/dev/script/container_remount_lxcfs.sh

[Billyxxu]

@senthilnathann @fanhaozzu Please use this script:)
https://github.com/xigang/lxcfs-admission-webhook/blob/dev/script/container_remount_lxcfs.sh

Hi, I used your script, but still couldn't solve the problem.Pls help

[Billyxxu]

@senthilnathann @Billyxxu
We've solved the problem. You have to use 2 layers folder path to mount lxcfs directory into your container.

for example:

OLD: /usr/local/var/lib/lxcfs

NEW: /usr/local/var/lib/lxc/lxcfs

and...please mount path /usr/local/var/lib/lxc into your container. After lxcfs crashed and restarted, You can use bash command to remount it again.

Thks for your help, I have solved the problem in the container according to your method. But is there a good solution in Kubernetes? I see that pouchcontainer handles this problem in the source code, but I use docker, and I have to change the source code as well?

[g0194776]

@Billyxxu
You can use admission hook to mount those of paths automatically, Here is my copy.

[Billyxxu]

@Billyxxu
You can use admission hook to mount those of paths automatically, Here is my copy.

good idea. thks