New containers not showing on dockerhub (yet!) about gentoo-docker-images

[ultrabug]

@mjeveritt @KSmanis I created the stage3 repository in the gentoo org

[KSmanis]

Imho, repos are misused instead of Docker tags in this context. Just two repos suffice: portage and stage3. Docker tags can take care of the rest.

portage is already there, so I would suggest to create just the stage3 repo and start pushing all relevant configurations there, while maintaining backwards compatibility downstream by maintaining or deprecating the existing stage3-* repos.

[KSmanis]

With #92 upping the number of built images to 22, I think it's high time to resolve this issue as well. A member with admin rights would be required to create one or more new repos, according to one of the following options.

tl;dr: I recommend option 2.

1. Flat hierarchy

One repo per built image, i.e., what we currently have. For example, the x86-hardened stage3 from today would be tagged as gentoo/stage3-x86-hardened:20200826. To my knowledge, this is a bit unconventional, since official Docker images commonly make use of tags for mixing configurations (e.g., x86-hardened) with dates/versions. We currently only use dates as tags.

List of repos/tags along with the corresponding Docker arch of the image:

gentoo/portage (ARCH=amd64)
gentoo/stage3-amd64 (ARCH=amd64)
gentoo/stage3-amd64-hardened (ARCH=amd64)
gentoo/stage3-amd64-hardened-nomultilib (ARCH=amd64)
gentoo/stage3-amd64-musl-hardened (ARCH=amd64)
gentoo/stage3-amd64-musl-vanilla (ARCH=amd64)
gentoo/stage3-amd64-nomultilib (ARCH=amd64)
gentoo/stage3-amd64-systemd (ARCH=amd64)
gentoo/stage3-amd64-uclibc-hardened (ARCH=amd64)
gentoo/stage3-amd64-uclibc-vanilla (ARCH=amd64)
gentoo/stage3-arm64 (ARCH=arm64)
gentoo/stage3-arm64-systemd (ARCH=arm64)
gentoo/stage3-armv6j_hardfp (ARCH=arm/v6)
gentoo/stage3-armv7a_hardfp (ARCH=arm/v7)
gentoo/stage3-ppc64le (ARCH=ppc64le)
gentoo/stage3-s390x (ARCH=s390x)
gentoo/stage3-x86 (ARCH=386)
gentoo/stage3-x86-hardened (ARCH=386)
gentoo/stage3-x86-musl-vanilla (ARCH=386)
gentoo/stage3-x86-systemd (ARCH=386)
gentoo/stage3-x86-uclibc-hardened (ARCH=386)
gentoo/stage3-x86-uclibc-vanilla (ARCH=386)

2. Tags with architectures

Two repos: portage and stage3. All configuration combinations are expressed as tags, even architectures. For example, the x86-hardened stage3 from today would be tagged as gentoo/stage3:x86-hardened-20200826.

List of repos/tags along with the corresponding Docker arch of the image:

gentoo/portage (ARCH=amd64)
gentoo/stage3:amd64 (ARCH=amd64)
gentoo/stage3:amd64-hardened (ARCH=amd64)
gentoo/stage3:amd64-hardened-nomultilib (ARCH=amd64)
gentoo/stage3:amd64-musl-hardened (ARCH=amd64)
gentoo/stage3:amd64-musl-vanilla (ARCH=amd64)
gentoo/stage3:amd64-nomultilib (ARCH=amd64)
gentoo/stage3:amd64-systemd (ARCH=amd64)
gentoo/stage3:amd64-uclibc-hardened (ARCH=amd64)
gentoo/stage3:amd64-uclibc-vanilla (ARCH=amd64)
gentoo/stage3:arm64 (ARCH=arm64)
gentoo/stage3:arm64-systemd (ARCH=arm64)
gentoo/stage3:armv6j_hardfp (ARCH=arm/v6)
gentoo/stage3:armv7a_hardfp (ARCH=arm/v7)
gentoo/stage3:ppc64le (ARCH=ppc64le)
gentoo/stage3:s390x (ARCH=s390x)
gentoo/stage3:x86 (ARCH=386)
gentoo/stage3:x86-hardened (ARCH=386)
gentoo/stage3:x86-musl-vanilla (ARCH=386)
gentoo/stage3:x86-systemd (ARCH=386)
gentoo/stage3:x86-uclibc-hardened (ARCH=386)
gentoo/stage3:x86-uclibc-vanilla (ARCH=386)

3. Tags without architectures

Again, two repos: portage and stage3, but the architecture is only part of the image manifest instead. For example, the x86-hardened stage3 would be simply tagged as gentoo/stage3:hardened-20200826 and the Docker runtime would be responsible for picking the right architecture according to the host.

List of repos/tags along with the corresponding Docker arch of the image:

gentoo/portage (ARCH=amd64)
gentoo/stage3 (ARCH=amd64;arm64;arm/v6;arm/v7;ppc64le;s390x;386)
gentoo/stage3:hardened (ARCH=amd64;386)
gentoo/stage3:hardened-nomultilib (ARCH=amd64)
gentoo/stage3:musl-hardened (ARCH=amd64)
gentoo/stage3:musl-vanilla (ARCH=amd64;386)
gentoo/stage3:nomultilib (ARCH=amd64)
gentoo/stage3:systemd (ARCH=amd64;arm64;386)
gentoo/stage3:uclibc-hardened (ARCH=amd64;386)
gentoo/stage3:uclibc-vanilla (ARCH=amd64;386)

Option 3 is what's being used by official Docker images, but in our case I would argue that the architecture is part of the configuration and should therefore appear in the tag as well. It's a little more verbose than option 3, but much clearer imho.

Assuming we move forward with option 2 or 3, we also need a maintenance/deprecation plan for all the existing stage3-* repos. I suggest we just stop pushing to them and delete them several months after putting up big ugly notices that the new place to be is gentoo/stage3.

[tianon]

From the perspective of being a maintainer on https://github.com/docker-library/official-images, I'd like to clarify that we do kind of a combination of 2 & 3 -- 3 is the public facing repos under https://hub.docker.com/u/library, but in order to create those, we implement essentially 2 by pushing the per-arch images to a separate repository per architecture (https://hub.docker.com/u/amd64, https://hub.docker.com/u/arm64v8, etc). 👍

[KSmanis]

Indeed, official Docker images have dedicated orgs for every arch, but given that we don't have official status, we are not allowed to push there. That's why with option 2 we prefix the arch in the tag instead.

In other words, if we had official status, the latest hardened stage3 image would probably appear as:

• amd64/gentoo:stage3-hardened (ARCH=amd64)
• i386/gentoo:stage3-hardened (ARCH=386)
• gentoo:stage3-hardened (ARCH=amd64;386)

Since we don't have official status, option 2 would appear as:

• gentoo/stage3:amd64-hardened (ARCH=amd64)
• gentoo/stage3:x86-hardened (ARCH=386)

And option 3 would appear as:

• gentoo/stage3:hardened (ARCH=amd64;386)

As you can see option 2 and 3 are not mutually exclusive, so we could have both:

• gentoo/stage3:amd64-hardened (ARCH=amd64)
• gentoo/stage3:x86-hardened (ARCH=386)
• gentoo/stage3:hardened (ARCH=amd64;386)

Which is basically what official images do. Notice that with both options, everything is pushed to the stage3 repo of the gentoo org.

[KSmanis]

I believe it's more natural to do 2 then 3, because as @tianon mentioned this is also what the official images do, i.e., they combine the parts (per-arch images) into a single entity (multi-arch manifest).

@tianon feel free to chip in with your take on it!

[mjeveritt]

@ultrabug - hey can you or @mgorny create the new 'stage3' repository (on dockerhub), so we can work on the local configs to push the new containers up to the docker hub to start with? Then we can work on fixing 'official' builds per #65; and also #61 gets closed and #75 can be progressed! Thanks!

[junghans]

https://docs.github.com/en/packages/managing-container-images-with-github-container-registry/pushing-and-pulling-docker-images#pushing-container-images looks like the salient instruction(s) for adapting the existing build script @KSmanis ;)

#93

[mjeveritt]

Imho, repos are misused instead of Docker tags in this context. Just two repos suffice: portage and stage3. Docker tags can take care of the rest.

portage is already there, so I would suggest to create just the stage3 repo and start pushing all relevant configurations there, while maintaining backwards compatibility downstream by maintaining or deprecating the existing stage3-* repos.

Works for me - I can tweak the scripts appropriately, and the new stuff should 'appear' I believe...

[mjeveritt]

It looks a bit like the Gentoo docker 'page'/site/user etc. needs a bit of an overhaul to make it work with this configuration. I thought it looked a little odd at first glance, when compared against other projects. Possibly it needs converting to a project/organisation rather than a plain user?
@ultrabug if you're around on IRC happy to give this some time/discussion if that's feasible for you?

[mjeveritt]

Between 2 & 3 here

[KSmanis]

We should probably follow suit and implement both options 2 and 3, since they are not mutually exclusive and they both have merits. Option 3 is actually quite convenient for creating multiarch downstream images, which seems to be picking up steam for the past few years.

[ultrabug]

So a mix of 2 & 3 would require to create as much new organizations as there are architectures right? I'm not sure about the benefit of this vs option 1 then.

Sorry if I'm missing something obvious.

[KSmanis]

Option 2 or 3 or their mix would only require a single stage3 repo, which we are currently missing.

[ultrabug]

@KSmanis I don't get it. If I check how it's being proposed by @tianon I see that each arch has its own org

What's the mix of 2 & 3 for you then? Could you please provide an example just like you did before?

[ultrabug]

Thanks for clarifying @KSmanis ; I think I got mislead by the arch name of the orgs that Tianon was pointing to

    gentoo/stage3:hardened (ARCH=amd64;386)

I personally like option 3 for simplicity and correctness across arches

But since we're looking into getting official status, if we started with option 3, we could then get option 2 afterwards right?

[tianon]

I agree -- option 3 is going to be a lot easier to implement if you do option 2 first (because unless you use something like buildx to build them all simultaneously, you have to build/push them somewhere before you can assemble the manifest list necessary for option 3).

(None of these choices will really have any bearing on official images or that process, because the process of submission is different. 😅)

[KSmanis]

As a matter of fact, I have already implemented option 3 on top of option 2, as discussed. Once #92 is mainlined, I can submit a new PR (pending the creation of the stage3 Docker repo).

The latest CI run can be found here, pushing images to ksmanis/portage and ksmanis/stage3.

[mjeveritt]

@mjeveritt @KSmanis I created the stage3 repository in the gentoo org

Awesome! Thanks! Let's get this rolling ...

[junghans]

Due to the new limitations on docker hub (https://www.docker.com/pricing/resource-consumption-updates) it would be good to push the containers to the Github Container registry (ghcr.io, https://github.blog/2020-09-01-introducing-github-container-registry/) as well.

[mjeveritt]

Due to the new limitations on docker hub (https://www.docker.com/pricing/resource-consumption-updates) it would be good to push the containers to the Github Container registry (ghcr.io, https://github.blog/2020-09-01-introducing-github-container-registry/) as well.

From the link supplied (re: docker):

Will Official Images be subject to the inactive image retention policy?
No. The inactive image retention policy is not applicable to Official Images. None of the images in ?library? namespace will be deleted. Images that are published by Verified Publishers are also not subject to inactive image retention policy.

However, in the interests of not holding "all eggs in one basket" I see no reason not to explore the github avenue, especially since we're already using Travis-CI to build the container images ...

[junghans]

However, in the interests of not holding "all eggs in one basket" I see no reason not to explore the github avenue, especially since we're already using Travis-CI to build the container images ...

We also have another registry here: https://gitlab.com/gentoo

[mjeveritt]

Is anyone seriously paying any attention to the Gitlab Gentoo organisation though?

[mjeveritt]

https://docs.github.com/en/packages/managing-container-images-with-github-container-registry/pushing-and-pulling-docker-images#pushing-container-images looks like the salient instruction(s) for adapting the existing build script @KSmanis ;)