raffis / kjournal

kjournal is a lightweight kubernetes apiserver to serve longterm logging & events as kubernetes API

kjournal

kjournal closes the gap between long-term log storage and your client. kjournal is a simple and lightweight kubernetes api server which makes your longterm log storage available as consumable readonly kubernetes api. With kjournal you can easily access container logs from old (and current pods) as well as other long-term logs including out of the box support for kubernetes audit logs and events.

Documention

Want to get started? Please read the documentation here.

kjournal's Issues

elasticsearch with tls cacert does not work, x509 verify error

Describe the bug

https connection to es backend does not work while a valid ca is configured.
=> failed to close pit" err="x509: certificate signed by unknown authority

Steps to Reproduce

Set es to https. Set cacert to valid root ca.

Expected behavior

Valid connection

Environment

• kjournal-apiserver version: v0.0.15
• kjournal-cli version [e.g. v0.0.1]
• kubernetes version: [e.g. v1.24.0]
• Elasticsearch version: [e.g. v8.0.0]

Additional context

The connection works while setting allowInsecureTLS=true

Field selectors can not be parsed containing special chars

Describe the bug

kjournal audit --field-selector user.username=system:s -v10

I1118 08:26:22.639012 442558 request.go:1154] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"unable to parse requirement: values[0][user.username]: Invalid value: \"system:s\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')","reason":"BadRequest","code":400}

Currently label filters get used behind the scenes. The problem is the values are quite restrictive.
The reason why label and not field selector are used is because of the operators. Label selectors also support >, <.

Expected behavior

The field selector gets parsed and used without an error if it contains special characters.

Environment

• kjournal-apiserver version: v0.0.10
• kjournal-cli version: v0.0.10
• kubernetes version: [e.g. v1.24.0]
• Elasticsearch version: [e.g. v8.0.0]

Additional context

Support d for days

Current situation

--since and --range do not support time in days. 3d does not work but 72h does.

E1206 08:36:21.733788 960439 main.go:116] "execution failed" err="time: unknown unit \"d\" in duration \"3d\""

Proposal

Support d.

requests are closed after 1m

Describe the bug

Currently there is an apiserver global timeout for requests which is 60s.

Steps to Reproduce

time kjournal pods

==> Will stop after 1m.

Expected behavior

If in stream mode the apiserver likely should not cancel requests.

Environment

• kjournal-apiserver version: v0.0.13
• kjournal-cli version: v0.0.13
• kubernetes version: [e.g. v1.24.0]
• Elasticsearch version: [e.g. v8.0.0]

Additional context

kjournal version prints kube-apiserver instead kjournal-apiserver version

Describe the bug

Client Version: version.Info{Major:"0", Minor:"0", GitVersion:"0.0.0-dev", GitCommit:"none", GitTreeState:"clean", BuildDate:"unknown", GoVersion:"go1.18", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-19T15:39:43Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

Steps to Reproduce

kjournal version

Expected behavior

Environment

• kjournal-apiserver version: v0.0.12
• kjournal-cli version: v0.0.12
• kubernetes version: [e.g. v1.24.0]
• Elasticsearch version: [e.g. v8.0.0]

Additional context

kjournal install is incomplete

Describe the bug

• image version wrong, not the latest
• domain part lost
• certmanager patches wrong container name
• default es uri

Steps to Reproduce

kjournal install --export

Expected behavior

Environment

• kjournal-apiserver version: v0.0.13
• kjournal-cli version: v0.0.13
• kubernetes version: [e.g. v1.24.0]
• Elasticsearch version: [e.g. v8.0.0]

Additional context

Support gcloud log as storage backend

Is your feature request related to a problem? Please describe

No

Describe the solution you'd like

Support google gcloud log backend for gke clusters.

Support loki as storage backend

Is your feature request related to a problem? Please describe

No

Describe the solution you'd like

Integrate grafanas loki as kjournal backend storage.

Additional context

https://github.com/grafana/loki/