adevinta / noe

streamline multi arch k8s runtimes

Noe: Kubernetes Mutating Webhook for Node Architecture Selection

Blog post announcing and explaining the effort behind Noe

Noe is a Kubernetes mutating webhook that dynamically assigns node architectures to match the requirements of container images within a Pod. It simplifies mixed-architecture deployments (e.g. ARM and x86) by ensuring that Pods are scheduled on nodes capable of executing all their images.

Features

• Automatically adjusts node affinities based on container images' supported architectures
• Improves deployment efficiency by removing the need for manual node selector configuration
• Facilitates seamless mixed-architecture deployments by ensuring compatibility between ARM and x86 nodes

Running Tests

Run all tests using the following command:

go test ./...

Installing Noe

Noe provides a Helm chart, available exclusively from the code repository. The simplest way to install it is to use ArgoCD and define an application such as:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    repoURL: https://github.com/adevinta/noe.git
    path: charts/noe
    targetRevision: HEAD

Hinting preferred and supported target architectures

By default, Noe will automatically select the appropriate architecture when only one is supported by all the containers in the Pod. If more than one is available, Noe will select the system-defined preferred one if available. This preference can be chosen in the command line for Noe (defaults to amd64 if unspecified):

./noe -preferred-arch amd64

This preference can also be overridden at the Pod level by adding the label:

labels:
  arch.noe.adevinta.com/preferred: amd64

Noe will always prioritize a running Pod, so if the preference is not supported by all the containers in the Pod, the common architecture will be selected.

You can restrict the acceptable common architectures in the command line for Noe:

./noe -cluster-schedulable-archs amd64,arm64

If you specify both a preferred architecture and a list of supported architectures in the command line, the default architecture must be part of the list. Otherwise Noe will fail to start.

If a preferred architecture is specified at the Pod level and is not compatible with the supported architectures listed in the command line, it will be ignored.

noe's Issues

"/vendor": not found on docker build

Hi,

im looking to your project I'm interested in.
I'checkout the source code and simply run
docker build . (because your repository ghcr.io/adevinta/noe:latest in not public )

throws this error

 => ERROR [builder 5/8] COPY vendor/ vendor/                                                                                                                                                   0.0s
------
 > [builder 5/8] COPY vendor/ vendor/:
------
Dockerfile:9
--------------------
   7 |     COPY go.mod go.mod
   8 |     COPY go.sum go.sum
   9 | >>> COPY vendor/ vendor/
  10 |     
  11 |     # Copy the go source
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref 23afac0f-5bd1-4d31-b003-60d06acc2970::mlxrgq9q2tz3cblysydpiyis2: "/vendor": not found

Strict mode for certain namespaces

Background

Currently, Noe follows the "Robustness Principle" or Postel's Law:

"Be conservative in what you send, be liberal in what you accept"

Pull request #10 is an instance of that approach. We've traded off a "perfect allocation" of images to the correct node Operating System — and that may even fail at run time — for the convenience of running images whose manifests do not fully inform these fields.

Problem

In some cases, where teams have complete control of the image build process, they want to ensure all their images are multi-architecture compliant. Hence, they prefer to trade off convenience for correctness, and to have all the respective architectures running a specific image.

For cluster operators and teams that provide cluster-wide features, this issue is even more critical.

Proposal

I propose to add a flag to Noe, for a strict mode that requires Operating System and Architecture fields to be present in the manifest. The interface would be a list of namespaces where we want to enforce this strict mode. Example:

--strict role-binding-operator,platform-services

Support for fetching container manifests from private repositories

Problem Statement: While the project currently handles public container image repositories effectively, it encounters issues when trying to work with private repositories. This limitation is highlighted by an inability to fetch container manifests from private repositories, as evidenced by a 401 Unauthorized status code error when attempting such operations. This issue significantly hampers the project's utility in environments where private repositories are a necessity for security and organizational reasons.

Error Encountered:
The specific error encountered when trying to fetch a manifest from a private repository is as follows:
{"Pod":{"name":"podinfo-blue-app-app-bf8bcc8cf-n9ggr","namespace":"podinfo-private"},"controller":"pod","controllerGroup":"","controllerKind":"Pod","error":"failed to get manifest list. Unexpected status code 401. Expecting 200","level":"error","msg":"Reconciler error","name":"podinfo-blue-app-app-bf8bcc8cf-n9ggr","namespace":"podinfo-private","reconcileID":"9caebb6e-2b9c-4987-9f90-4c8b40b8223f","time":"2024-03-18T02:27:46Z"}

Proposed Feature:
Introduce support for fetching container manifests from private container repositories, including but not limited to AWS Elastic Container Registry (ECR). This feature would ideally handle authentication seamlessly, allowing users to specify credentials in a secure manner that Noe can use to authenticate against private repositories.

`latest` tag not kept updated

The helm chart values uses latest as the image version, but it looks like that's not kept updated with the CI builds.