alpinelinux / docker-abuild Goto Github PK

Add an environment variable to let user specify location of .../packages -- don't assume it's ok to be under /home/...

This causes the filesystem to fill up if you don't manually clean up the images every once and while (e.g. docker system prune). I have this issue constantly as I tend to build large groups of packages at once (like KDE updates) using a script. Halfway such a process my system is filled up by leftover docker images from docker-abuild, and there isn't enough disk space to continue.

Currently DABUILD_ARCH is not documented anywhere (readme, potential -h flag for dabuild).

Not directly related to this issue, but setting DABUILD_ARCH=aarch64 with dabuild -r results in a sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges? message, doesn't look like that's intended behavior.

my current distro

bash-4.4# cat /etc/os-release

NAME="Alpine Linux"

ID=alpine

VERSION_ID=3.9.4

PRETTY_NAME="Alpine Linux v3.9"

HOME_URL="https://alpinelinux.org/"

BUG_REPORT_URL="https://bugs.alpinelinux.org/"

bash-4.4#

terraform init -from-module="git::ssh://xxxx

Init works with the repo my local using -from-module but does not work in docker Alpine that was provided by Hashicorp!!!!

What other libraries do I need?

How can I see the error on my container? ( the terraform does not give me the reason I had to find it after trying multiple points )

Here is my docker file I am using at the moment

FROM hashicorp/terraform:0.11.14 AS main

RUN apk -v --update add ncurses
RUN apk -v --update add bash
RUN apk -v --update add git
RUN apk -v --update add openssh
RUN apk -v --update add openssl-dev
RUN apk -v --update add expect
RUN apk -v --update add python
RUN apk -v --update add py-pip

RUN pip install --upgrade awscli==$aws_cli_version
RUN pip install azure-devops==5.0.0b9

RUN apk -v --purge del py-pip curl &&
rm /var/cache/apk/*

The reason i am looking into this is a git module of Terraform is not working I think there a missing dependency

hashicorp/terraform#22791

uname -m on raspbian may give armv6l on rpi0 for instance, which clashes wrt expected armhf tests
https://github.com/alpinelinux/docker-abuild/blob/master/Makefile#L19
https://github.com/alpinelinux/docker-abuild/blob/master/dabuild.in#L27

At the moment, to get the dabuild script, you have to type make which also goes ahead and builds all the images rather than using those published on Docker Hub which is a bit wasteful.

Instead of using a bind mount from hosts $HOME we should switch $HOME/.abuild directory to a named volume. This will solve permissions conflicts if uid of user and container differ. This will also make it possible to have a user host key and config different from what is used in the docker environment.

@mor1 i have thought about mounting /etc/abuild.conf into the container and I would urge to remove this feature. The reason is that if the hosts release (if using an alpine system) is different from the container the settings in your hosts /etc/abuild.conf could be incorrect. The same goes for platform/arch changes which could be inside the /etc/abuild.conf. A user should really only modify its own abuild.conf inside $HOME/.abuild/abuild.conf.

It'd make sense to run some operations (e.g. abuild checksum, unpack) via the normal abuild script, if it's detected on the system so that we don't have to create a new container just for these simple operations.

I dont think it's wise to share this with the host, for sure when users are allowed to use different arches. Instead I would suggest to use docker volumes based on arch.

ref #36

Looking at the readme we support

• x86
• x86_64
• aarch64
• armv6
• armv7
• armv7l
• armv8

but alpine only has support for:

• aarch64
• armhf
• armv7
• ppc64le
• s390x
• x86
• x86_64

ppc64le and s390x will be added ones we switch to GitLab CI.

If running on Alpine directly, useful to bind mount /var/cache/{distfiles,apk}/ and /etc/abuild.conf.

Invoking sudo apk -U upgrade -a in entrypoint.sh significantly slows down invocation of dabuild because the updates aren't cached so have to be applied every time.

Better would be to autobuild and update the alpinelinux/docker-abuild images on Hub when packages are updated I think -- is that feasible?

/cc @clandmeter @tcely

E.g. libgdata's tests fail in docker-abuild but not with abuild -r:

ERROR: oauth2-authorizer - Bail out! libuhttpmock:ERROR:libuhttpmock/uhm-server.c:1398:uhm_server_run: assertion failed (error == NULL): Could not listen on address ::1, port 35143: Error binding to address: Address not available (g-io-error-quark, 0)
ERROR: youtube - Bail out! libuhttpmock:ERROR:libuhttpmock/uhm-server.c:1398:uhm_server_run: assertion failed (error == NULL): Could not listen on address ::1, port 41085: Error binding to address: Address not available (g-io-error-quark, 0)

(continued from #46)

Running env DABUILD_ARCH=aarch64 dabuild -r on an x86_64 machine results in sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges? being printed (tested with aarch64 & armv7)

alpinelinux/docker-abuild   edge-aarch64        1470e8631406        9 days ago          195MB

$ docker run --entrypoint /bin/sh --rm -it alpinelinux/docker-abuild:edge-aarch64
~ $ sudo ls
sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?

On the host:

$ mount | grep docker
/dev/mapper/docker-8:1-20185132-fc7eb3aa34775e1dc50251a64cf4b6320077e1d03f1bce5c269907e2b5b2a7bc on /mnt/hdd/docker/devicemapper/mnt/fc7eb3aa34775e1dc50251a64cf4b6320077e1d03f1bce5c269907e2b5b2a7bc type xfs (rw,relatime,nouuid,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=128,noquota)
nsfs on /run/docker/netns/d72959cccaf9 type nsfs (rw)

In the container:

~ $ mount
/dev/mapper/docker-8:1-20185132-b4824e4ae8418b6c7d99c2d628b3d8def114a44359cd13bd9bc8b83253483ca2 on / type xfs (rw,relatime,nouuid,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=128,noquota)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/rdma type cgroup (ro,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
/dev/sda1 on /etc/resolv.conf type ext4 (rw,relatime,stripe=32738)
/dev/sda1 on /etc/hostname type ext4 (rw,relatime,stripe=32738)
/dev/sda1 on /etc/hosts type ext4 (rw,relatime,stripe=32738)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
proc on /proc/bus type proc (ro,relatime)
proc on /proc/fs type proc (ro,relatime)
proc on /proc/irq type proc (ro,relatime)
proc on /proc/sys type proc (ro,relatime)
proc on /proc/sysrq-trigger type proc (ro,relatime)
tmpfs on /proc/asound type tmpfs (ro,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/latency_stats type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/scsi type tmpfs (ro,relatime)
tmpfs on /sys/firmware type tmpfs (ro,relatime)

So as far as I can tell, the rootfs inside the container is not nosuid and I also don't have anything mounted specially (except ~/.cache and /root/.cache but those shouldn't matter)

Currently the default namespace is mor1/dabuild. I would like to put the images under the alpinelinux namespace. Should we also change the image name to docker-abuild to make it less confusing?

We also discussed to not explicitly define the ORG in the makefile to prevent mistakes, but I think it makes sense dabuild has the correct location to the image so we do not have to provide that manually when creating the dabuild script. We should probably define a separate push ORG so we can keep that empty?

Because Fedora Linux 31 removed cgroups-v2 and docker cannot work now. Podman is CLI-compatible with Docker, so it boils down to switch to the variable instead of calling docker directly.

The repository index for aports/testing/ is disabled by default so any intent to build a package with dependencies on package(s) from aports/testing/ will fail.

A test example on macOS:

cd aports/testing/zimwriterfs/
DABUILD_PACKAGES=~/packages/ ~/docker-abuild/dabuild -r
>>> zimwriterfs: Building testing/zimwriterfs 1.3-r0 (using abuild 3.4.0_rc3-r0) started Fri, 03 May 2019 02:07:57 +0000
>>> zimwriterfs: Checking sanity of /home/builder/aports/testing/zimwriterfs/APKBUILD...
>>> zimwriterfs: Analyzing dependencies...
ERROR: unsatisfiable constraints:
  gumbo-parser-dev (missing):
    required by: .makedepends-zimwriterfs-0[gumbo-parser-dev]
  libzim-dev (missing):
    required by: .makedepends-zimwriterfs-0[libzim-dev]
>>> ERROR: zimwriterfs: builddeps failed
>>> zimwriterfs: Uninstalling dependencies...
WARNING: Ignoring /home/builder/packages/main/x86_64/APKINDEX.tar.gz: No such file or directory
WARNING: Ignoring /home/builder/packages/community/x86_64/APKINDEX.tar.gz: No such file or directory
WARNING: Ignoring /home/builder/packages/testing/x86_64/APKINDEX.tar.gz: No such file or directory

I had the folder called ´aports_local` and running dabuild failed weirdly with this error:

>>> ERROR: : Could not find ./APKBUILD (PWD=/home/builder/aports_local/community/brotli)

After renaming the folder from aports_local to aports, it works fine.

I feel that this code will run every time as the check will return true all time as this directory will be not preserved between runs. This could lead to that with each run the new key pair will be generated and already signed index files will be rendered as un-trusted as they will be signed with one key pair and later they will be intended to be signed with fresh generated key pair.

Maybe we can move key generation to Dockerfile?

Would it be an idea to rename the abuild wrapper script to dabuild or similar so it does not clash with abuild when running this on Alpine?

I've currently setup drone to build images every week, is that ok?

When running dabuild on Arch Linux, this happens:

Unable to find image 'alpinelinux/docker-abuild:edge-x86_64' locally
edge-x86_64: Pulling from alpinelinux/docker-abuild
7692c7e2a2d0: Pull complete 
ad5123831438: Pull complete 
09728ba0c2ff: Pull complete 
709d5ae2a39d: Pull complete 
0eef046ab93a: Pull complete 
8d3e521a1e03: Pull complete 
Digest: sha256:b638013287bd1829051b7a77515f991fcb45710bb1acd2c19b1c3632fe0c749e
Status: Downloaded newer image for alpinelinux/docker-abuild:edge-x86_64
genrsa: Can't open "/home/builder/.abuild/-5d73b350.rsa" for writing, Permission denied
Can't open /home/builder/.abuild/-5d73b350.rsa for reading, No such file or directory
139887471508808:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:72:fopen('/home/builder/.abuild/-5d73b350.rsa','r')
139887471508808:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:79:
unable to load Private Key
>>> 
>>> You'll need to install /home/builder/.abuild/-5d73b350.rsa.pub into 
>>> /etc/apk/keys to be able to install packages and repositories signed with
>>> /home/builder/.abuild/-5d73b350.rsa
/usr/bin/abuild-keygen: line 75: can't create /home/builder/.abuild/abuild.conf: Permission denied
>>> 
>>> Please remember to make a safe backup of your private key:
>>> /home/builder/.abuild/-5d73b350.rsa
>>> 
/home/builder/entrypoint.sh: .: line 16: can't open '/home/builder/.abuild/abuild.conf': No such file or directory

When you try to build a package which uses special filesystem features on MacOS abuild with error.
To reproduce one can build bash and it will error.

make images build the Docker image and tries to upload it.

I try some ideas with Docker image and run make images quite often. It builds the image and tries to upload it producing the error. I do not like error messages. So...

What you think if we will have 2 build targets: one to only build the image and the second one to build and upload the image?

I am not sure we should use %%ALPINE_TAG%% in path for local repos. I think abuild uses .../packages/[main|community|testing]/....

Right now volume mounting do not take %%ALPINE_TAG%% into account and this leads to following errors for edge:

cd aports/testing/zimwriterfs
...
>>> zimwriterfs: Uninstalling dependencies...
WARNING: Ignoring /home/builder/packages/edge/main/x86_64/APKINDEX.tar.gz: No such file or directory
WARNING: Ignoring /home/builder/packages/edge/community/x86_64/APKINDEX.tar.gz: No such file or directory
WARNING: Ignoring /home/builder/packages/edge/testing/x86_64/APKINDEX.tar.gz: No such file or directory

I see no way to configure default value for DABUILD_PACKAGES so I need to specify it on command line all the time.

Hi,

running dabuild with podman (rootless) results in permission denied issues as the mounted aports tree is mapped to the root user:

[john:~/workspace/aports/testing/signal-cli] dabuild checksum
[...]
sed: can't create temp file '/home/builder/aports/testing/signal-cli/APKBUILDXXXXXX': Permission denied

Setting the correct uidmap / gidmap arguments makes it work again:

[john:~/workspace/aports/testing/signal-cli] cat /etc/dabuild.conf 
#!/bin/sh
DABUILD_DOCKER=podman
DABUILD_ARGS="--uidmap 1000:0:1 --uidmap 0:1:1000 --gidmap 1000:0:1 --gidmap 0:1:1000"

[john:~/workspace/aports/testing/signal-cli] id
uid=1000(john) gid=1000(john) groups=1000(john),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),100(users),116(docker),119(lpadmin),131(sambashare),138(libvirt)

Can you please add a note to the README?
Thanks!

Right now users have to be careful keeping DABUILD_PACKAGES and REPODEST in sync, otherwise packages won't end up in the bindmount that dabuild creates and as such won't be kept after the build. It'd be nice if dabuild were to automatically derive DABUILD_PACKAGES from REPODEST so users don't have to worry about that and it "just works".

It looks to me that only docker on MacOS can support multi-arch as can be read here:
https://docs.docker.com/docker-for-mac/multi-arch/

This makes it easy to run dabuild for a different arch with qemu emulation (its slow but it would work).
I have tried this on Docker on Linux but it doesnt seem to work in the same way. I can also not find any reference for multi-arch for other platforms except the link above.

I guess this might fall under user error, but it'd be nice if dabuild checked that REPODEST (from abuild.conf) is writable.

Would be nice if we could move development to gitlab and let it use gitlab ci.
The periodic build of images with drone cloud seems not supported, need to check if we can fix this within gitlab CI. @Ikke any idea?