rpc faults && permission denied issues about docker-nfs-server

[ehough]

Got it to work. AppArmor is indeed not allowing the container to perform the required mounts. We can supply a custom AppArmor profile for the container to use. Here's how:

WARNING I am not very familiar with AppArmor and cannot make any claims of the security impact of these instructions. However, this should be quite safe.

1. Ensure you have apparmor-utils installed:

   sudo apt-get install apparmor-utils
   

2. Create a file on the Docker host with the following contents:

   #include <tunables/global>
   profile erichough-nfs flags=(attach_disconnected,mediate_deleted) {
     #include <abstractions/lxc/container-base>
     mount fstype=nfs*,
     mount fstype=rpc_pipefs,
   }
   

3. Load this profile into AppArmor:

   sudo apparmor_parser -r -W /path/to/file/from/previous/step
   

4. Add --security-opt apparmor=erichough-nfs to your docker run command. e.g.

   docker run                                \
     -v /path/to/exports.txt:/etc/exports:ro \
     -v /path/to/share:/nfs                  \
     --cap-add SYS_ADMIN                     \
     -p 2049:2049                            \
     --security-opt apparmor=erichough-nfs   \
     erichough/nfs-server
   
   

Give it a try and let me know? If it works for you, I'll add these instructions to the README. Thanks.

[ehough]

Thanks for the report. You're definitely not missing anything obvious; I'm also a little stumped.

mount: rpc_pipefs is write-protected, mounting read-only
mount: cannot mount rpc_pipefs read-only
could not open /proc/fs/nfs/exports for locking: errno 13 (Permission denied)

These filesystem permissions issues are clearly the cause. It seems like the mounts (required by nfsd) aren't allowed, even though --cap-add SYS_ADMIN should permit them.

I'll spin up an Ubuntu vm to try to reproduce this. In the meantime, would you:

1. Send along the output of docker info so I can ensure we have the same environment and
2. Try using --privileged instead of --cap-add SYS_ADMIN, just to see if it's a Docker permissions issue.

We should be able to figure this out.

[ehough]

@michaelrcarroll that's great to hear, thank you! I've added some docs for AppArmor, including a sample docker-compose.yml based on yours.

Closing this issue but please feel free to re-open or continue the discussion. Thanks, all!

[theriverman]

Tried --privileged, no luck.

docker info below:

Containers: 16
 Running: 15
 Paused: 0
 Stopped: 1
Images: 99
Server Version: 17.12.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: v0.13.0 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-23-generic
Operating System: Ubuntu 18.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.604GiB
Name: ourkid
ID: <lots_of_random_strings>
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

[ehough]

Thank you. Do you have AppArmor enabled by chance?

sudo aa-status

[theriverman]

It is loaded, indeed:

root@ourkid ~ # sudo aa-status
apparmor module is loaded.
22 profiles are loaded.
22 profiles are in enforce mode.
   /sbin/dhclient
   /snap/core/4650/usr/lib/snapd/snap-confine
   /snap/core/4650/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/bin/lxc-start
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/sbin/tcpdump
   docker-default
   lxc-container-default
   lxc-container-default-cgns
   lxc-container-default-with-mounting
   lxc-container-default-with-nesting
   man_filter
   man_groff
   snap-update-ns.core
   snap-update-ns.jq
   snap.core.hook.configure
   snap.jq.jq
0 profiles are in complain mode.
72 processes have profiles defined.
72 processes are in enforce mode.
   /sbin/dhclient (865)
   docker-default (5114)
   docker-default (5117)
   docker-default (5119)
   docker-default (5121)
   docker-default (5128)
   docker-default (9193)
   docker-default (9261)
   docker-default (9314)
   docker-default (9315)
   docker-default (10826)
   docker-default (11335)
   docker-default (11354)
   docker-default (11358)
   docker-default (11360)
   docker-default (11361)
   docker-default (11362)
   docker-default (11363)
   docker-default (12620)
   docker-default (12788)
   docker-default (12789)
   docker-default (12790)
   docker-default (12791)
   docker-default (12792)
   docker-default (13806)
   docker-default (13892)
   docker-default (14768)
   docker-default (14769)
   docker-default (15963)
   docker-default (16076)
   docker-default (16222)
   docker-default (16238)
   docker-default (19667)
   docker-default (19729)
   docker-default (20031)
   docker-default (20154)
   docker-default (20230)
   docker-default (20255)
   docker-default (20326)
   docker-default (20328)
   docker-default (20331)
   docker-default (20347)
   docker-default (20435)
   docker-default (20606)
   docker-default (20721)
   docker-default (20812)
   docker-default (20813)
   docker-default (21123)
   docker-default (21174)
   docker-default (21175)
   docker-default (21176)
   docker-default (21177)
   docker-default (22731)
   docker-default (22776)
   docker-default (22870)
   docker-default (22871)
   docker-default (22872)
   docker-default (22955)
   docker-default (22964)
   docker-default (22965)
   docker-default (22966)
   docker-default (22967)
   docker-default (22970)
   docker-default (23130)
   docker-default (24111)
   docker-default (24375)
   docker-default (24513)
   docker-default (24752)
   docker-default (25182)
   docker-default (27607)
   docker-default (28151)
   docker-default (31713)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

I tried executing
service apparmor stop
service apparmor teardown

but no luck

[ehough]

OK, thanks. I'm still trying to reproduce the issue on my end. Should have it figured out soon. Stand by!

[ehough]

I'm able to reproduce the error. Now to find the fix ...

[ehough]

Any luck? I'd love to add these instructions to the README, and am curious if they worked for you as well.

[michaelrcarroll]

For what it's worth: I was experiencing the same issues described as @theriverman. The apparmor config did the trick and solved my issue.

As an extra-added bonus, here's my incredibly vanilla docker-compose config if you're looking for an example:

version: "2"

services:
  nfs:
    image: erichough/nfs-server
    volumes:
      - ./files:/nfs/files
      - ./exports:/etc/exports
    ports:
      - 2049:2049
    cap_add:
      - SYS_ADMIN
    security_opt:
      - apparmor=erichough-nfs

[Chrislevi]

Hey, @ehough.
I'm having the same issue on Centos 7 though. The selinux is disabled but still having the same error. How can I achieve the same thing without selinux enabled..?

[ehough]

@Chrislevi could you open a new issue with further details of your setup along with what errors you're seeing? I'd be very interested in getting the image to work under selinux as well as apparmor. Thanks!

[cybericius]

Any idea (/path/to/file/from/previous/step=.apparmor)?
AppArmor parser error for .apparmor in profile .apparmor at line 3: Could not open 'abstractions/lxc/container-base'

[cybericius]

Any idea (/path/to/file/from/previous/step=.apparmor)? AppArmor parser error for .apparmor in profile .apparmor at line 3: Could not open 'abstractions/lxc/container-base'

Solution:
sudo apt install lxc