aurae-runtime / architecture Goto Github PK

So here is a strong opinion that is weakly held. I would like for us to discuss this in detail.

I believe that auraed should only start by exposing gRPC server a unix domain socket, as well as a server bound to the special loopback interface lo man. Additionally I think auraed should reserve TCP port 57 for its default port.

• /var/aurae/aurae.sock
• 127.0.0.1
• ::1

Additional Network Devices

I believe we should be able to start auraed and pass configuration to the daemon at runtime to provision gRPC servers attached to various devices.

Some syntax such as this.

sudo -E auraed --listen="lo:127.0.0.1:57" --listen="lo:::1:57" --listen="eth0:10.0.0.100:57" 

However we expose the network devices should be uniform and there should be no different between configuring a network device via the command line, or via the gRPC server APIs.

To be clear, I'm not advocating for k8s, I just think we will be asked this question and we should have robust answer. This is supposed to be more "food for thought" kind of issue. I hope we can gather good list of gaps that actually prevents us from achieving things we want on Kube, and help us ensure that we won't fall into same traps with Aurae.

According to #4, Auraes main difference to Kubernetes is an app manifest that's actually programming language (as opposed to yaml). That provides flexibility to dynamically define app runtime. It allows logic like "get this image and push it to core docker registry". All that is wonderful utility for users. Aurae value, as of today, is on client and user experience side, which sucks for k8s.

I also think this could be achieved with very opinionated deployment of k8s (setup registry with all the certificates for image pushing, setup storage both block and object etc.) and well crafted client library that would understand these components and be able to communicate with them.
Imagine client library (in any programming language) that would query k8s api, pull addresses and certificates for docker registry and setup port forwarding or network tunnel. With that and few other components like that, add some magic executor inside pod for dynamic changes to pod apps etc. Kubernetes becomes under-the-hood state machine that handles all the infra (runtime, networking, storage) and Aurae becomes a platform that takes it, other components on top of it and gives user coherent and convenient experience.

Pros of this approach would be:

• Kube exists and is well maintained. Setting up Kube is matter of few clicks on any major public cloud and there are plenty of tools that already integrate with it well. helm install aurae + brew install aurae-client and we're off to races. We could focus on providing opinionated set of tools + fill gaps where they don't exist, and focus on designing wonderful client library.
• Migrating from regular Kube to Aurae would be much easier. We could even provide tools to read Kube deployments etc and generate Aurae equivalent.

Now, let me start with things that I, personally, see being potential gaps in Kube that can't be solved by simple wrappers/operators/admission controllers/clients:

• Resource architecture of Kube kind of sucks imo - It doesn't allow "composable" types. We have DaemonSet, Deployment, ReplicaSet, Job, CronJob etc etc. All of these just run pods in slightly different manner. How would you run script on every node in cluster every day? Only way to do it is to create CronJob that creates DeamonSet that has restartPolicy: Never, which is very similar to Job but not quite. It would be easier to create tools that allow to define each of these, somewhat unrelated, parts of manifest (where to run, when to run, how to restart) and compose type that fits our problem. Something like traits in Rust would be great

#[derive(Pod)]
struct CronDaemonset {}

impl Cron for CronDaemonset {
fn crontab(&self) -> CronTab {
    CronTab{"0 8 * * *"}
}
}

impl Scheduling for CronDaemonset {
fn select_nodes(nodes: Vec<Node>) -> Vec<Node> {
    nodes
}
}

• Above example touches second issue I have with k8s - scheduling is very inflexible. Taints, tolerations allow you to model some behaviors, but there are whole slew of things you can't do (example I've hit was to utilize finished job count during scheduling). It's also very complex (requiredDuringSchedulingIgnoredDuringExecution is very clear about what it does, doesn't it? /s). I think we can do better with logic.

I'm sure there are other examples of these, let's think on them a bit please.

From GLOSSARY.md:

Aurae Node is the set of services that provide an Aurae service that exist as a single fault domain

I take the ambiguity of "single fault domain" as a conviction for auraed to work well in multiple production scenarios:

1. Bare metal / VM pid 1
2. Container
3. Bare-metal / VM integrates w/ systemd to replace systemd-nspawn

Is this a correct read on the definition of an Aurae Node?

Nomad seems to be built on similar promise as Aurae. I don't know much about it but I think we should compare notes and make sure we know how it relates to Aurae.

We can replace the architecture repository with GitHub discussions for the active decision making.

After a decision has been made we can capture it in the main repository or in an issue.