kube-rs / version-rs

Example kubernetes reflector and web server

kube-rs

A Rust client for Kubernetes in the style of a more generic client-go, a runtime abstraction inspired by controller-runtime, and a derive macro for CRDs inspired by kubebuilder. Hosted by CNCF as a Sandbox Project

These crates build upon Kubernetes apimachinery + api concepts to enable generic abstractions. These abstractions allow Rust reinterpretations of reflectors, controllers, and custom resource interfaces, so that you can write applications easily.

Installation

Select a version of kube along with the generated k8s-openapi structs at your chosen Kubernetes version:

[dependencies]
kube = { version = "0.91.0", features = ["runtime", "derive"] }
k8s-openapi = { version = "0.22.0", features = ["latest"] }

See features for a quick overview of default-enabled / opt-in functionality.

Upgrading

See kube.rs/upgrading. Noteworthy changes are highlighted in releases, and archived in the changelog.

Usage

See the examples directory for how to use any of these crates.

• kube API Docs
• kube.rs

Official examples:

• version-rs: lightweight deployment reflector using axum
• controller-rs: Controller of a crd inside actix

For real world projects see ADOPTERS.

Api

The Api is what interacts with Kubernetes resources, and is generic over Resource:

use k8s_openapi::api::core::v1::Pod;
let pods: Api<Pod> = Api::default_namespaced(client);

let pod = pods.get("blog").await?;
println!("Got pod: {pod:?}");

let patch = json!({"spec": {
    "activeDeadlineSeconds": 5
}});
let pp = PatchParams::apply("kube");
let patched = pods.patch("blog", &pp, &Patch::Apply(patch)).await?;
assert_eq!(patched.spec.active_deadline_seconds, Some(5));

pods.delete("blog", &DeleteParams::default()).await?;

See the examples ending in _api examples for more detail.

Custom Resource Definitions

Working with custom resources uses automatic code-generation via proc_macros in kube-derive.

You need to add #[derive(CustomResource, JsonSchema)] and some #[kube(attrs..)] on a spec struct:

#[derive(CustomResource, Debug, Serialize, Deserialize, Default, Clone, JsonSchema)]
#[kube(group = "kube.rs", version = "v1", kind = "Document", namespaced)]
pub struct DocumentSpec {
    title: String,
    content: String,
}

Then you can use the generated wrapper struct Document as a kube::Resource:

let docs: Api<Document> = Api::default_namespaced(client);
let d = Document::new("guide", DocumentSpec::default());
println!("doc: {:?}", d);
println!("crd: {:?}", serde_yaml::to_string(&Document::crd()));

There are a ton of kubebuilder-like instructions that you can annotate with here. See the documentation or the crd_ prefixed examples for more.

NB: #[derive(CustomResource)] requires the derive feature enabled on kube.

Runtime

The runtime module exports the kube_runtime crate and contains higher level abstractions on top of the Api and Resource types so that you don't have to do all the watch/resourceVersion/storage book-keeping yourself.

Watchers

A low level streaming interface (similar to informers) that presents Applied, Deleted or Restarted events.

let api = Api::<Pod>::default_namespaced(client);
let stream = watcher(api, Config::default()).applied_objects();

This now gives a continual stream of events and you do not need to care about the watch having to restart, or connections dropping.

while let Some(event) = stream.try_next().await? {
    println!("Applied: {}", event.name_any());
}

NB: the plain items in a watcher stream are different from WatchEvent. If you are following along to "see what changed", you should flatten it with one of the utilities from WatchStreamExt, such as applied_objects.

Reflectors

A reflector is a watcher with Store on K. It acts on all the Event<K> exposed by watcher to ensure that the state in the Store is as accurate as possible.

let nodes: Api<Node> = Api::all(client);
let lp = Config::default().labels("kubernetes.io/arch=amd64");
let (reader, writer) = reflector::store();
let rf = reflector(writer, watcher(nodes, lp));

At this point you can listen to the reflector as if it was a watcher, but you can also query the reader at any point.

Controllers

A Controller is a reflector along with an arbitrary number of watchers that schedule events internally to send events through a reconciler:

Controller::new(root_kind_api, Config::default())
    .owns(child_kind_api, Config::default())
    .run(reconcile, error_policy, context)
    .for_each(|res| async move {
        match res {
            Ok(o) => info!("reconciled {:?}", o),
            Err(e) => warn!("reconcile failed: {}", Report::from(e)),
        }
    })
    .await;

Here reconcile and error_policy refer to functions you define. The first will be called when the root or child elements change, and the second when the reconciler returns an Err.

See the controller guide for how to write these.

TLS

By default rustls is used for TLS, but openssl is supported. To switch, turn off default-features, and enable the openssl-tls feature:

[dependencies]
kube = { version = "0.91.0", default-features = false, features = ["client", "openssl-tls"] }
k8s-openapi = { version = "0.22.0", features = ["latest"] }

This will pull in openssl and hyper-openssl. If default-features is left enabled, you will pull in two TLS stacks, and the default will remain as rustls.

musl-libc

Kube will work with distroless, scratch, and alpine (it's also possible to use alpine as a builder with some caveats).

License

Apache 2.0 licensed. See LICENSE for details.

version-rs's Issues

No license

Hi. Could you possibly add a license to this example?

Get default metrics working with axum

We previously used actix-web-prom to give us good baseline metrics:

$ curl 0.0.0.0:8000/metrics
api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="0.005"} 11
...
...
api_http_requests_duration_seconds_bucket{endpoint="/",method="GET",status="200",le="+Inf"} 11
api_http_requests_duration_seconds_sum{endpoint="/",method="GET",status="200"} 0.001559851
api_http_requests_duration_seconds_count{endpoint="/",method="GET",status="200"} 11
# HELP api_http_requests_total Total number of HTTP requests
# TYPE api_http_requests_total counter
api_http_requests_total{endpoint="/",method="GET",status="200"} 11

but this had a lot of upgrade issues through the whole "required actix beta".

So, after the axum port in #6 we should see if there's a way to get good baseline default metrics added.
Probably requires a bit of a wait for metrics ecosystem to evolve though.

last tokio::select does not print warnings

the last select should function as a join but with a print of what exited, but for some reason, even with DEBUG logs, it does not print the last line without exiting. not sure what causes this..

Try out using axum instead of actix

We already have one POC for actix (controller-rs) and nice to have one that builds quickly. From the discord:

shane: not sure if there's any appetite in this direction beyond myself, but I'm experimenting (relatively successfully) with converting the two example projects in kube-rs github org from actix-web to axum. cursory/anecdotal results are all nice drops in dependency tree count, binary size and build time. I figure since a while back kube started to depend on tower itself there's some nice synergy to this combo.

shane: anecdata for version-rs: number of crates built: ~350->193, release binary size: ~9.5MB -> 7.6MB, release build, time on my 2018 6-core MBP: 2min56sec -> 1min42sec

cc @shanesveller

error trying to connect: invalid dnsname - in Kubernetes

I cross-compiled this on MacOS to aarch64 and deployed to a local kind-cluster.
I built it with rustls for kube-rs with this line in Cargo.toml

[dependencies]
kube = { version = "0.57.0", default-features = false, features = ["client", "rustls-tls"] }

But I get many log-lines with this error:

ERROR HTTP{http.method=GET http.url=https://10.96.0.1:443/apis/apps/v1/namespaces/default/deployments? otel.name="list" otel.kind="client" otel.status_code="ERROR"}: kube::client: failed with error error trying to connect: invalid dnsname

This is when I run the version-app as a Deployment in Kubernetes.

Is this something expected? Is it a bug?