odigos-io / odigos Goto Github PK

Is your feature request related to a problem? Please describe.
We have an ELK-like stack (FLOOD stack?) with OpenSearch for monitoring and observability. This project would be a game changer if OpenSearch was supported as a destination.

Describe the solution you'd like
OpenSearch as log/metric/trace destination for odigos

Use generated name for InstrumentedApplication instead of fixed name

Destination name: HyperDX
Support OpenTelemetry format: yes, see: https://www.hyperdx.io/docs/install/opentelemetry
Supported signals: traces, metrics, logs

Describe the bug
Java applications in bank-of-athnos always crash when instrumentation is enabled

To Reproduce
Steps to reproduce the behavior:

# install bank-of-athnos
kubectl apply -f https://raw.githubusercontent.com/keyval-dev/bank-of-athnos/main/release/kubernetes-manifests.yaml

# install odigos 
helm repo add odigos https://keyval-dev.github.io/odigos-charts/
helm install my-odigos odigos/odigos --namespace odigos-system --create-namespace

Enable instrumentation with the UI, and find pod are crashed

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Ubuntu 16.04.7 LTS
• kind: 0.17.0

Additional context
Add any other context about the problem here.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Describe the solution you'd like
Currently, instrumentation is achieved through device plugins, which are deployed through daemonsets. This approach does not work well in serverless services.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Hello,
when we set an url starting with https:// the config on the gateway configmap has an http:// prefix.

Thank for your work, this sounds very promising

Describe the bug
Current UI requires a "http" or "https" prefix for OTLP endpoint. While in fact, it is using otlp/grpc instead of otlp/http. According to the OpenTelemetry Collector naming convention, we should use ip:port to indicate the use of the otlp/grpc protocol and http://ip:port or https://ip:port to indicate the use of the otlp/http protocol.

To Reproduce
Steps to reproduce the behavior:

• Use UI to add destination of OpenTelemetry.

Expected behavior
The UI should not requires a "http" or "https" prefix for OTLP endpoint.

Screenshots

https://github.com/keyval-dev/odigos/blob/2f7d45c9a2deb1751bab3e120fe86a1ea1fdc7f5/frontend/endpoints/namespaces.go#L96

Is happening a lot and does not indicate an error. We should either log it in a higher debug level or just remove this line.

Add to GitHub actions workflow: build and push container images of all the instrumentation containers

In logz.io destination, the region field should be a dropdown that contains the following values: https://docs.logz.io/user-guide/accounts/account-region.html

Similar to the site field in Datadog destination.

The main steps involved when debugging Odigos locally:

• Use Kind kubernetes cluster
• Build custom images of Odigos and load them to Kind via:

 TAG=<CURRENT-ODIGOS-VERSION> make build-images load-to-kind

• Make sure TAG is similar to the Odgiso version output: odigos version
• Restart all pods in odigos-system namespace:

kubectl delete pods --all -n odigos-system

Just really wanna know what is the url to be added for destination when using Loki as backend, as I can't find any example for the configuration, nor any detail docs on this, and the configuration failed. pls. advise, thanks.

my Loki deploy is as below:
root@xxxxx:# kubectl -n logging get pods
NAME READY STATUS RESTARTS AGE
loki-0 1/1 Running 0 4m32s
loki-grafana-7595cff4f5-4szxs 2/2 Running 0 4m31s
loki-promtail-2n28d 1/1 Running 0 4m32s
loki-promtail-46q2l 1/1 Running 0 4m32s
loki-promtail-6fjfm 1/1 Running 0 4m32s
loki-promtail-8fzzp 1/1 Running 0 4m32s
loki-promtail-8ndgj 1/1 Running 0 4m32s
loki-promtail-h7nfx 1/1 Running 0 4m32s
loki-promtail-nn25b 1/1 Running 0 4m32s
loki-promtail-r7nmv 1/1 Running 0 4m32s
loki-promtail-xbxpg 1/1 Running 0 4m32s
root@xxxxx:# kubectl -n logging get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
loki ClusterIP 10.68.124.157 3100/TCP 4m9s
loki-grafana ClusterIP 10.68.25.148 80/TCP 4m9s
loki-headless ClusterIP None 3100/TCP 4m9s
loki-memberlist ClusterIP None 7946/TCP 4m9s

Odigos UI binary should include the git tag version.
This can be done using ldflags in goreleaser, similar to the way that we do it for the CLI:
https://github.com/keyval-dev/odigos/blob/02639256a750cfa8e0e7555954af046fc8140c23/.goreleaser.yaml#L19-L22

Change Odigos logo background to be transparent instead of black.

Describe the bug
The pod of the demo app is crashed when instrumentation is enabled.

To Reproduce
Steps to reproduce the behavior:

1. Run command according to the doc:
   kubectl apply -f https://raw.githubusercontent.com/keyval-dev/microservices-demo/master/release/kubernetes-manifests.yaml

2. Install Odigos
   helm repo add odigos https://keyval-dev.github.io/odigos-charts/
   helm install my-odigos odigos/odigos --namespace odigos-system --create-namespace

3. Enable instrumentation with the UI

4. Find pod are crashed
   kubectl get pod -n default

Expected behavior
The pod should not be crashed

Screenshots
The following screenshot is after I enable instrumentation for frontend pod:

Environment:

• OS: Ubuntu 18.04.6 LTS
• kubectl version
  WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
  Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:58:16Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v4.5.7
  Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:51:25Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
• Browser Firefox 109.0

Additional context
It seems the problem is caused by some compatibility issue of Golang packages:

kubectl logs -f frontend-54d7ddf7f4-54tcn -p -c server-instrumentation
... ...
{"level":"info","ts":1675158604.7114024,"caller":"grpc/probe.go:213","msg":"closing gRPC instrumentor"}
{"level":"info","ts":1675158604.7114253,"caller":"server/probe.go:223","msg":"closing gRPC server instrumentor"}
{"level":"info","ts":1675158604.7114346,"caller":"server/probe.go:179","msg":"closing net/http instrumentor"}
{"level":"info","ts":1675158604.7114527,"caller":"mux/probe.go:179","msg":"closing gorilla/mux instrumentor"}
{"level":"error","ts":1675158604.7114713,"caller":"cli/main.go:79","msg":"error while running instrumentors","error":"field UprobeClientConnInvoke: program uprobe_ClientConn_Invoke: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)","stacktrace":"main.main\n\t/app/cli/main.go:79\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

Describe the bug
Installing odigos stack without version specified via cli results in odigos pods failure.

To Reproduce
Steps to reproduce the behavior:

1. Run odigos cli install command without the --version tag specified.
2. Observe InvalidImageName status for odigos pods.

Expected behavior
If version isn't specifically set up latest should be apply even if this docker images tag doesn't exist yet.

Screenshots

“Failed to apply default image tag "keyval/odigos-odiglet:": couldn't p
arse image reference "keyval/odigos-odiglet:": invalid reference format”

When running $odigos --help, the following output is generated:

A longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:

Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.

Usage:
  odigos [command]

Available Commands:
  add         A brief description of your command
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  install     Install Odigos
  ui          Start the Odigos UI
  uninstall   A brief description of your command
  version     A brief description of your command

Flags:
  -h, --help                help for odigos
      --kubeconfig string   (optional) absolute path to the kubeconfig file (default "/Users/amirblum/.kube/config")

Use "odigos [command] --help" for more information about a command.

Odigos should replace all the default texts such as "A brief description of your command" with meaningful information

Greetings from ZincObserve team - https://github.com/zinclabs/zincobserve . Would love to see support for ZincObserve in odigo.

Is your feature request related to a problem? Please describe.
Support for sending logs, metrics and traces to ZincObserve

Describe the solution you'd like
A direct selection for ZincObserve in odigos that allows people to send logs, metrics and traces to ZincObserve. ZincObserve supports:

Logs - JSON array or elasticsearch _bulk format
Metrics - Prometheus remote write
Traces - OTLP

Here are details around it - https://zinc.dev/docs/guide/ingestion/

There should not be really much to do for odigo team to achieve this as all the elements are already there.

Describe alternatives you've considered
use elasticsearch, opentelemtry, and prometheus enpoints separately.

Additional context
No

Describe the bug
some Golang and Java language services cannot be recognized.
eg.

Expected behavior
i expect that all Java , Golang and Python language service can be recognized.

Desktop (please complete the following information):
AWS EKS 1.21

Additional context
what can i do next?

GitHub just announced new ARM-based runners:
https://github.blog/2023-10-02-introducing-the-new-apple-silicon-powered-m1-macos-larger-runner-for-github-actions/

Currently, the majority of the build time is spent on building ARM images on x64 runners.
Moving to building the ARM images on ARM runners should make our build time much faster.

Kubernetes clusters with PodSecurityPolicy enabled may refuse to deploy Odiglet due to the privileged permissions required (hostPID, hostNetwork, hostPath, etc).
We should detect if PodSecurityPolicy is enabled in this cluster (maybe by finding if the resource exists?)
and if so we should create a new policy and bind it to the odiglet service account.

In the create destination page (for example at /overview/destinations/create/form?dest=splunk)
Pressing enter should be the same as clicking the create destination button.

Destination name: Elastic Observability
Support OpenTelemetry format: yes, see: https://www.elastic.co/guide/en/apm/guide/8.10/open-telemetry-direct.html#connect-open-telemetry-collector
Supported signals: traces, metrics, logs

Describe the bug
While installing in a cluster with existing deployments, installing and using the opt-out option for odigos leads to a situation where cluster services end up in crashloopbackoff with no easy way to fix other than reapplying all deployment manifests.

I understand that the documentation states uninstalling is a work in progress, but the mutation of deployments and statefulsets leaves them in a bad state after uninstall, and users without gitops or similar deployment methods will face challenges in fixing their deployments. A workaround and a prominent warning should be displayed on this repo until there is a clean uninstall path.

To Reproduce
Steps to reproduce the behavior:

1. Install Odigos
2. In UI setup, go through the opt-out option flow for setup
3. Setup a backend (optional)
4. wait for services to be instrumented
5. disable services that are enabled for instrumentation in odigos UI
6. helm uninstall odigos
7. Pods that start up that were instrumented still have odigos sidecar, configurations, volumemount, and command changes. These pods may also be in crashloop/containerconfigcreateerror states.

Expected behavior
Changes by odigos to deployments should be reverted when odigos is disabled and uninstalled.

Describe the bug
Minor duplication in README - "Getting Started Guide", "Documentation", and "quickstart guide" all end up at https://docs.odigos.io/intro as it is the default docs page

To Reproduce
Steps to reproduce the behavior:

1. Go to https://github.com/keyval-dev/odigos#readme
2. Click on any of the 3 links mentioned ("Getting Started Guide", "Documentation", and "quickstart guide")

Expected behavior
Links lead to different information/user flows, or links are deduplicated and have consistent labeling.

Build and increment agents' container images as part of the CI/CD workflow

helm install my-odigos odigos/odigos --namespace odigos-system --create-namespace ends in the following pods installed but odiglet is not coming up.

odiglet-gg9tw                          0/1     CrashLoopBackOff   69 (4m35s ago)   6h9m
odigos-autoscaler-6f98b9f9-86m9s       2/2     Running            13 (5h15m ago)   6h9m
odigos-instrumentor-66bd6d6f5b-ckl7q   2/2     Running            13 (5h15m ago)   6h9m
odigos-scheduler-6d694ff89f-vhlxv      2/2     Running            14 (5h15m ago)   6h9m
odigos-ui-fb4485899-mrs9p              1/1     Running            2 (5h15m ago)    6h9m

Docker Desktop 4.17.0 (99724). on a M1 Macbook MacOS Ventura 13.3.1 (22E261)

Error via k logs odiglet-gg9tw -n odigos-system is:

failed to try resolving symlinks in path "/var/log/pods/odigos-system_odiglet-gg9tw_3ca139e5-35f1-48d3-a748-afe2fbcf40e8/odiglet/69.log": lstat /var/log/pods/odigos-system_odiglet-gg9tw_3ca139e5-35f1-48d3-a748-afe2fbcf40e8/odiglet/69.log: no such file or directory%

k version -o yaml shows:

clientVersion:
  buildDate: "2022-12-08T19:58:30Z"
  compiler: gc
  gitCommit: b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d
  gitTreeState: clean
  gitVersion: v1.26.0
  goVersion: go1.19.4
  major: "1"
  minor: "26"
  platform: darwin/arm64
kustomizeVersion: v4.5.7
serverVersion:
  buildDate: "2022-11-09T13:29:58Z"
  compiler: gc
  gitCommit: 872a965c6c6526caa949f0c6ac028ef7aff3fb78
  gitTreeState: clean
  gitVersion: v1.25.4
  goVersion: go1.19.3
  major: "1"
  minor: "25"
  platform: linux/arm64

thanks

The opentelemetry service traces does not show service name

like this

The current guide for adding new destination is not updated.
We changed the way a new destinations are added.
Here is an example of a PR that added destination with the new way: #535

The relevant doc file is: docs/adding-new-dest.mdx

It seems to odigos add cli command is not doing anything.

I think it can be safely removed

We would like to have the documentation in the same repository as Odigos' code.
Currently, all documentation files are stored at: https://github.com/keyval-dev/odigos-docs/
We should create a new subdirectory called docs and put everything there

The current language detection mechanism of creating a pod, detecting the language, and then terminating the pod is error prune. I saw some OutOfPods errors when running on EKS clusters.

Moving the same lang detection logic to odiglet will make the language detection process faster and more robust.

Destination name: Axiom
Support OpenTelemetry format: yes, see: https://axiom.co/docs/send-data/opentelemetry
Supported signals: traces, metrics, logs

Currently, logs are extracted using the filelog receiver. Moving to OTLP for exporting has the following advantages:

• Correlation: attach traceId and spanId without messing with the output written to stdout.
• Flexible deployment: currently the collectors that receive logs have to be on the same node as the application (deployed as DaemonSet)
• Alignment with traces and metrics

Getting logs by scraping pod files will still be in use for unrecognized pods or for SDKs without a logs exporter.
For example, there is no SDK for MySQL but we still want to let the users ship MySQL logs to the selected destinations.

Implementing this issue includes two tasks:
A. configure all the SDKs to export logs via OTLP exporter
B. configure the collectors' include attribute of the filelog receiver to only include unrecognized applications or known languages without logs exporter.

At the time of writing this issue, the following SDKs implements OTLP logs exporter:

As VictoriaMetrics gains popularity, we would like to add it as a new destination in Odigos.

Hello,

Could you allow advanced users to set the full url of the endpoints,
and maybe on the front the port and protocol used by the gateway directly on the web page.

This would allow us to have,

• basic authentication support rfc7617
• custom port support
• path
• headers

This will allow a good integration with a more complex architecture like this one:

• Kube cluster A idigos-gateway> Kube cluster B loki ingress :

  • https://user:[email protected]:443/api/v1/push

• Kube cluster A idigos-gateway --> Kube cluster B tempo ingress :

  • https://user:[email protected]:443/

What do you think about it?

Describe the bug
While uninstalling odigos via cli, label odigos-instrumentation=enabled in instrumented namespace isn't removed. This situation results with the issue with discovering services in affected namespaces while setting up odigos once again from scratch.

To Reproduce
Steps to reproduce the behavior:

1. Install odigos using cli .
2. Instrument services in one of the namespaces.
3. Uninstall odigos.
4. Try to configure instrumentation for same services in same namespace once again.

Expected behavior
No issues with discovering previously instrumented services during odigos reinstallation

Screenshots
During reinstrumentation:

The Odigos UI binary should have a flag that will print the current version of the UI.

When installing odigos via the CLI by running odigos install the cli reports installation successful even if Odiglet (the deployed DaemonSet) is not running. Kubernetes may refuse to start the pods of Odiglet (for example: if there is a PodSecurityPolicy that blocks hostPID).

arePodsReady function should also check that for any DaemonSet in the ns namespace the number of running pods equal to the number of desired and is bigger than 1.
https://github.com/keyval-dev/odigos/blob/2f7d45c9a2deb1751bab3e120fe86a1ea1fdc7f5/cli/cmd/install.go#L86

Issue

Problem Statement
Currently, our application displays scrollbars in certain areas of the app, which can be visually distracting and affect the overall user experience. We need to implement a solution to remove these scrollbars entirely by using CSS.

Expected Behavior

When users interact with our app, there should be no visible scrollbars on the page, even when the content exceeds the viewport height or width. Instead, the content should seamlessly flow without any scroll indicators.

Steps to Reproduce

1. Launch the application.
2. Navigate to pages where scrollbars are currently visible (e.g., long content, modals, etc.).

Current Implementation

Currently, scrollbars are visible and are handled individually.

Desired Implementation

We need to apply CSS styles that hide the scrollbars throughout the entire app, ensuring a clean and seamless user experience. This should be achieved without breaking any functionality or causing layout issues.

Possible Solution

One approach to remove scrollbars is by using CSS properties such as overflow: hidden on specific elements or globally. We should carefully consider where and how this CSS is applied to ensure it doesn't interfere with the app's functionality.

Acceptance Criteria

• Scrollbars are not visible throughout the entire app.
• The app's functionality remains intact.
• Content can still be scrolled using other methods like mouse wheels or touch gestures.

Note

Before implementing any changes, it's essential to thoroughly test the proposed solution to ensure it doesn't introduce any unforeseen issues or negatively impact the user experience.

Please feel free to assign this issue to a developer who can work on implementing the solution and provide updates as progress is made. If you have any questions or need further clarification, don't hesitate to ask.

As more and more destinations are added, it will be harder for users to find the exact destination they are looking for.
A common way to solve this problem is by implementing a search.
Search can probably happen on the client side.

Destination name: Highlight.io
Support OpenTelemetry format: yes, see: https://www.highlight.io/docs/getting-started/backend-logging/file
Supported signals: logs

In Odigos CLI, when running odigos uninstall the user is prompted to confirm the uninstall.
We should treat pressing Enter as writing y.

Current state:

➜ odigos uninstall
About to uninstall Odigos from namespace odigos-system
Are you sure? [Y/n]:
Aborting uninstall

Desired:

➜ odigos uninstall
About to uninstall Odigos from namespace odigos-system
Are you sure? [Y/n]: 
Uninstalling Odigos Deployments                   ✔
Uninstalling Odigos DaemonSets                    ✔
Uninstalling Odigos ConfigMaps                    ✔
Uninstalling Odigos CRDs                          ✔
Uninstalling Odigos RBAC                          ✔
Uninstalling Odigos Secrets                       ✔
Uninstalling Namespace odigos-system              ✔
Waiting for namespace to be deleted               ✔
Rolling back odigos changes to pods               ✔

The list of applications that appear on the source selection page defaults to the applications of the default namespace, the problem occurs when there is a namespace that starts with a lowercase letter "d" and is displayed first in the dropdown list.

To Reproduce
Steps to reproduce the behavior:

1. Make sure you have a namespace that starts with a lowercase d, If it doesn't exist, you can create one with the command: kubectl create ns <name>
2. Go to Setup page
3. See error

Expected behavior
The apps listed should correspond to the information displayed in the dropdown menu.

Screenshots

Currently, every time a new Kubernetes version comes out or when a new OpenTelemetry instrumentation is updated, dependabot opens many pull requests for every dependency. We can use the group version updates feature of dependabot to open a single pull request that contains all the relevant dependencies changes.

Currently, our README.md contains a hard-coded table of all the destinations supported by Odigos.
The same data is available in destination/data/*.yaml files.
The destination table could be automatically generated from the YAML files, making it always in sync with the destinations actually supported.

Currently, the only validation the frontend performs is that the destination URL is a valid URL.
We can help users avoid typing the wrong URLs by performing more validations in the frontend before persisting the destination.
Examples of possible additional validations:

• Elasticsearch: URL should contain port 9200
• Generic OTLP: URL should contain port 4317
• Datadog: URL should be a valid site
• and many more, we can probably think of a better validation for every destination.

The first step in implementing this issue may be just validating a single destination in addition to thinking about a good design for validations in frontend.

The current categories are only managed / open source.
We plan to add many more observability backends in the future, which will require more granular categories such as Cloud Providers, Security, Finance, etc.