profefe / kube-profefe

continuous profiling made easy in Kubernetes with profefe

Profefe is a project developed by @narqo. I was looking for a solution to do continuous profiling and I realized his code was well abstracted and comfortable to extend. The API server was already done and I decided to write an integration with Kubernetes.

The documentation is located to ./docs/content/docs or to kubernetes.profefe.dev.

Install

./docs/content/docs/installation.md

kube-profefe's Issues

Get list of services via kubectl-profefe

Profefe exposes a new API to get the list of services:

https://github.com/profefe/profefe#get-services-for-which-profiling-data-is-stored

It will be nice to be able to do:

$ kubectl profefe get services

documentation about how to deploy kprofefe in kubernetes

get profiles by id

The command:

kubectl get profiles

should support a list of arguments, those arguments are profefe IDs.

Update krew index

If I'm understanding krew correctly, we need to update the index to make the newer versions of the client available.

Off the back of the latest release which includes auth updates I think it would be a good idea to update. I did notice the version on krew is not parallel with the version of kprofefe, so might be worth updating that too?

Convert from and to in UTC

Currently, we do not convert the time pass as from and to in UTC, this is a bug we need to convert it to UTC

continuous delivery to krew

krew is a plugin manager for the kubectl. We support it and kubectl-profefe
can be installed via:

krew install profefe

They have an action to keep the index up to date with new releases.

We should integrate with that

Returns labels from get profiles

We use labels a lot and it will be useful to get them when returning the list of profiles.
It is useful to get more context and change the label filtering in order to select only what we really need

Improve output for info and error

At the moment the output is very hacky, we use fmt println panic. We can do better than that.

Add support to query profiles

In short we need a comand get that accept -l (label), --profile-type, --service, --from, --to to get a list of profiles links from profefe:

kubectl profefe get profiles -l app=gateway --profile-type=cpu --from 10m

--to by default is now. I would like for --to and --from tu support range and time but I don't know how hard it is.

--service, --from are required

Make the parallelization factor dynamic

At the moment the number of goroutines kprofefe spin up is fixed to 10. We can do better.

First, we should look at GOMAXPROCS has to a limit. No more goroutines that what it specifies.

The GOMAXPROCS variable limits the number of operating system threads that can execute user-level Go code simultaneously. There is no limit to the number of threads that can be blocked in system calls on behalf of Go code; those do not count against the GOMAXPROCS limit. This package's GOMAXPROCS function queries and changes the limit.

In second based on the number of pods to gather, we can calculate the right number of goroutines.

kubectl get profiles timerange is buggy

You can get the list of profiles for a particular service in a timerange with
the command:

kubectl profefe get profiles --service <service-name>

It supports --from and --to it can be a duration -30m, -2h or a fixed
time formatted in RFC3339.

This feature does not work as expected.

wrapper around `go tool pprof` from the kubectl plugin

I would like to have a command that does go tool pprof with the right value for me.

So I am thinking about two things:

kubectl profefe pprof <profile_id>
Fetching profile over HTTP from http://localhost:10100/api/0/profiles/<profile_id>
Saved profile in /home/gianarb/pprof/pprof.queryd.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz
File: queryd
Build ID: 44dcff18a83acb2297e8fb8c209662cd7c3b27a5
type=heap
url=http://10.154.237.244:8093/debug/pprof
Type: inuse_space
Time: Jan 22, 2020 at 2:06am (CET)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) 

To get into the interactive pprof console from a profile_id

But I would also like to have a utility to get merged profiles:

kubectl profefe pprof --service web --from 20h --to 12h --profile_type heap

It brings you inside an interactive pprof console but with the merged profile for the specified time range

Better parallelization for kprofefe, split gather profiles from saving them to profefe

The parallelization made here #10 helped to speed up the gathering of profiles, but the end solution should be to have two different pipelines one to gather profiles and one to save them.

Those two operations can be very different in terms of timing it is good to also have two separate parametrizable concurrency threshold.

No Auth Provider found for name "gcp"

Adding the following import should allow for GCP clients:

_ "k8s.io/client-go/plugin/pkg/client/auth/gcp"

I imagine other managed kubernetes services would appreciate having the auth packages included for them too :)

Use profefe.com/service to set service name

At the moment profefe uses the pod name as service but it is not always the right behavior. Because the pod name may change so frequently that it will make very trivial to filter by service.

We can use another annotation profefe.com/service to specify the service name for the profile. If not present kube-profefe will keep using the pod name

add support binary arm arch for mac

it will be great if the binary will support ARM arch process with kubectl krew install

It does not exists! Write it!

kprofefe is a daemon that will run inside a kubernetes cluster and it is able to accept query selectors and namespace just as the kubectl does, but it is not designed to be run by humans but from cronjobs.

It does not need to do port forwarding because it is already inside the cluster, it will use the same annotations as the kubectl plugin:

• profefe.com/enable=true to identify if the pod wants to be profiled
• profefe.com/port and profefe.com/path to configure where the pprof handler is

Other than using localhost as we do in the kubectl plugin the address will be the pod id.

We do not need to write on disk, we just need a profefe url.

Other things to do:

• improve goreleaser to handle the new binary
• improve goreleaser to build a docker image
• write documentation about how to deploy it in kubernetes as a cronjob

[Security] Workflow test.yaml is using vulnerable action actions/checkout

The workflow test.yaml is referencing action actions/checkout using references v1. However this reference is missing the commit a6747255bd19d7a757dbdda8c654a9f84db19839 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.

Add kubectl plugin in krew

krew is a popular plugin manager that helps users to install kubectl plugins

https://github.com/kubernetes-sigs/krew-index/

We should add kubectl-profefe in there.

Monitor kprofefe with opentelemetry

We should figure out if it is useful to get aggregated metrics from kprofefe (probably yes, I would like to be notified if we stop collecting profiles for some reason).

OpenTelemetry is probably the way to go those days