cloud-bulldozer / go-commons Goto Github PK

We should setup a CI for each tool using go-commons to ensure changes to go-commons doesn't break functionality in our main tools.

At the moment go-commons is an all-in-one module, this is coming with several caveats.

• Cutting a new release affects all the libraries, this is not desired since a project could want to stick on an specific release of a library.

• Creates too many dependencies, a project only using one of the libraries will inherit all go-commons dependencies.

Splitting the project in different modules comes also with some downsides, the versioning handling of the different modules will be the most problematic one.

Let's use this issue to discuss about the possibilities

Kube-burner's node-density uses the number of worker nodes to calculate the number of pods to deploy in the cluster.
With this change recently pushed, kube-burner's failes because the nodes in the CI cluster has the scheduler configured with mastersSchedulable: true, hence the nodes are labeled as master and workers simultaneosly

We should fix this corner case.

The current readme does not have any data. We should add a good description of the repository, how to contribute, etc.

Looking at https://github.com/cloud-bulldozer/go-commons/blob/main/prometheus/prometheus.go#L89 which is used for the kube-burner ocp wrapper reporting mode, it looks like we are extracting all the datapoints over the timeseries and then feeding them into methods in stats package as per the aggregation required. It looks like prometheus supports aggregations natively (https://prometheus.io/docs/prometheus/latest/querying/functions/#aggregation_over_time) and we should switch over to using those for better consistency and confidence in our aggregations.

Integrate Prometheus library in Go-Commons

As a next step to standardize our (golang based) tooling we should enable this project to collect OCP cluster metadata. So different go based tools such as k8s-netperf, kube-burner, ocm-api-load... can use it and index use the same format of cluster metadata.

We can use kube-burner's cluster metadata collection as a starting point.

https://github.com/cloud-bulldozer/kube-burner/blob/4a79917a0dcfb33063b2c965f47631f10f09dd6c/pkg/workloads/helpers.go#L60-L79

type clusterMetadata struct {
	MetricName       string                 `json:"metricName,omitempty"`
	UUID             string                 `json:"uuid"`
	Platform         string                 `json:"platform"`
	OCPVersion       string                 `json:"ocpVersion"`
	K8SVersion       string                 `json:"k8sVersion"`
	MasterNodesType  string                 `json:"masterNodesType"`
	WorkerNodesType  string                 `json:"workerNodesType"`
	InfraNodesType   string                 `json:"infraNodesType"`
	WorkerNodesCount int                    `json:"workerNodesCount"`
	InfraNodesCount  int                    `json:"infraNodesCount"`
	TotalNodes       int                    `json:"totalNodes"`
	SDNType          string                 `json:"sdnType"`
	Benchmark        string                 `json:"benchmark"`
	Timestamp        time.Time              `json:"timestamp"`
	EndDate          time.Time              `json:"endDate"`
	ClusterName      string                 `json:"clusterName"`
	Passed           bool                   `json:"passed"`
	Metadata         map[string]interface{} `json:"metadata,omitempty"`
}