cap-js / telemetry Goto Github PK

see @opentelemetry/instrumentation-runtime-node

should we do the same as with @opentelemetry/host-metrics?

Currently the plugin seems to crash with an uncaught exception in case of missing information. That should maybe be catched as missing telemetry shouldn't be a show stopper but a warning. Also it seems like the actual api token name required deviates from the readme file (for version 0.0.2).

... uncaughtException: Neither metrics_apitoken nor rest_apitoken.token found in Dynatrace credentials\nError: Neither metrics_apitoken nor rest_apitoken.token found in Dynatrace credentials\n    at _getExporter (/home/vcap/deps/0/node_modules/@cap-js/telemetry/lib/metrics/index.js:27:23)\n

Yet the documentation describes:

Excerpt from example mta.yaml:

requires:
  - name: my-dynatrace-instance
    parameters:
      config:
        tokens:
          - name: ingest_apitoken #> default lookup name, configurable via cds.requires.telemetry.token_name
            scopes:
              - metrics.ingest

Which is present in the credentials.

I implemented two custom spans (using this https://opentelemetry.io/docs/languages/js/instrumentation/#acquiring-a-tracer as the example) just to see how it looks like in the logs:

const { trace } = require("@opentelemetry/api");
const tracer = trace.getTracer('@cap-js/telemetry');

module.exports = class AdminService extends cds.ApplicationService {
  init() {
    const { Books } = this.entities

   ...

    this.before(['CREATE', 'UPDATE'], Books, (req) => {
      const isValid = tracer.startActiveSpan('my-book-validation', (span) => {
        let result = true
        if (req.data.stock && req.data.stock < 0) {
          result = false
        }
        span.end();
        return result;
      });
      if (!isValid) {
        req.reject(400, 'Stock must be positive');
      }
    });

    this.after("READ", Books, (books) => {
      tracer.startActiveSpan('my-after-handler-on-books', async (span) => {
        for (let item of books) {
          let result = await SELECT.one.from(Books).where({ ID: item.ID });
          item.title = result.title + "___my____";
        }
        span.end();
      });
    });
    return super.init();
  }
};

The output of cds watch does not include my custom spans, but I can see them in Jaeger, for example. Is this a bug or a feature?

Hello, currently when we're setting log level to anything other than "info" it corrupts handle method for every service existing:

What happens here, you're wrapping handle method for Service class in https://github.com/cap-js/telemetry/blob/main/lib/tracing/cds.js#L39, but then later cds itself wrap handle method for ApplicationService https://github.tools.sap/cap/cds/blob/main/libx/_runtime/fiori/lean-draft.js#L208 to handle lean draft, so by the moment you're trying to "unwrap" there is no __original property in handle anymore. So any service call would fail with:

[cds] - TypeError: this.handle is not a function
    at CatalogService.dispatch (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/lib/srv/srv-dispatch.js:45:15)
    at _readCollection (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/handlers/read.js:250:28)
    at _readAndTransform (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/handlers/read.js:456:12)
    at /Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/handlers/read.js:544:22
    at /Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/okra/odata-server/core/Dispatcher.js:135:7
    at new Promise (<anonymous>)
    at Dispatcher._handle (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/okra/odata-server/core/Dispatcher.js:134:12)
    at Dispatcher.dispatch (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/okra/odata-server/core/Dispatcher.js:91:21)
    at DispatcherCommand.execute (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/okra/odata-server/invocation/DispatcherCommand.js:63:10)
    at CommandExecutor._execute (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/okra/odata-server/invocation/CommandExecutor.js:71:17) {
  id: '1931710',
  level: 'ERROR',
  timestamp: 1710410306617
}

To test it just add:

    "log": {
      "levels": {
        "cds": "error"
      }
    },

to your https://github.com/cap-js/telemetry/blob/main/test/bookshop/package.json, run the server and try to access any service.

Not sure, that tracing should depend on "cds" log level at all, and also unwrapping should be done with keeping in mind, there could be other wrappers around.

Hello,
we have following configuration:

1. Our service has dynatrace and cloud-logging services attached
2. We have oneagent installed
3. Our Cloud Logging has no ingest_otlp configuration
4. Our cds configuration is set cds.env.requires.telemetry.kind="to-dynatrace"

If we start our backend it crashes:

   2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR Error: No OpenTelemetry credentials found in SAP Cloud Logging binding. Create instance with config: "{ ingest_otlp: { enabled: true } }".
   2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at getCloudLoggingCredentials (/app/node_modules/@cap-js/telemetry/lib/utils.js:103:13)
   2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at _getExporter (/app/node_modules/@cap-js/telemetry/lib/metrics/index.js:33:15)
   2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at module.exports (/app/node_modules/@cap-js/telemetry/lib/metrics/index.js:61:20)
   2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at module.exports (/app/node_modules/@cap-js/telemetry/lib/index.js:27:3)
   2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at Object.<anonymous> (/app/node_modules/@cap-js/telemetry/cds-plugin.js:1:88)
   2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at Module._compile (node:internal/modules/cjs/loader:1241:14)
   2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at Module._extensions..js (node:internal/modules/cjs/loader:1295:10)
   2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at Module.load (node:internal/modules/cjs/loader:1091:32)
   2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at Module._load (node:internal/modules/cjs/loader:938:12)
   2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at H.loadModule (/opt/dynatrace/oneagent/agent/bin/1.275.146.20231002-095820/any/nodejs/nodejsagent.js:16603:70)
   2024-03-18T09:34:46.79+0100 [APP/PROC/WEB/0] OUT Exit status 1

I believe getCloudLoggingCredentials could behave better and do not throw if we have no intention to send any telemetry to Cloud Logging.
What do you think?

Some description

Hi all,

I have several calls in systems. However, as soon as an action is included, the telemetry is no longer output.

https://opentelemetry.io/docs/specs/otel/metrics/api/#get-a-meter

currently we do metrics.getMeter(${cds._telemetry.name}-meter)

either cds._telemetry.name has to be made public or we need something else...

the name should be an Instrumentation Scope -> why not simply @cap-js/telemetry?

and remove from name

-> we need a new name that produces readable output on console

Check if we can update @opentelemetry/* dependencies
(+ improve dependabot settings?)
(+ improve test setup and coverage?)

The traces generated by the plugin has Last date as invalid, Please refer to the below screenshot:

It's difficult to figure traces based on timestamp, We need to manually choose each of them to find the relevant trace. It would be good to get the Correct Last Updated value

Background

Our application leverages Express and CAP frameworks. CAP is initialized during the startup of Express. Both services, the application service, and the CAP service require tracing capabilities. The application service utilizes OpenTelemetry with HTTP instrumentation, while the CAP service utilizes @cap-js/telemetry, also with HTTP instrumentation. Both services share the same OTEL_EXPORTER_OTLP_TRACES_ENDPOINT for exporting trace data.

Issue Description

During testing, we've encountered difficulties with simultaneously sending trace data from both services to OTEL_EXPORTER_OTLP_TRACES_ENDPOINT. However, when sent separately, each service's trace data is successfully exported.

Expectation

We aim for both the application service and CAP service to be effectively traced and exported to OTEL_EXPORTER_OTLP_TRACES_ENDPOINT.

Screenshot

• Include CAP initialization within the express server.ts file.
  

• @cap-js/telemetry configuration
  

• Add application service exporter
  

example: @opentelemetry/exporter-prometheus

with mocked auth... probably the identical pool in set of pools. tenant information only needed in multitenant mode.

Hi,

Thanks for this extension. Is there any plan for on-the-fly config of tracing ratio (e.g. load config from a file at an interval)? Currently we can use environment variable to change the number. But this means in system like k8s, the change of tracing ratio will cause rolling update of pod, which will increase the risks of failure.

Thanks,
Samuel

Hi,
Currently if we want to deploy application to the cloud foundry runtime + cloud logging service, we need to manually add the service instance configuration and binding to srv. This does leads to error if someone miss configures the ingest configuration.

     - name: incidents-cloud-logging  
       type: org.cloudfoundry.managed-service  
       parameters: 
         service: cloud-logging  
         service-plan: standard  
         config:
          ingest_otlp: 
            enabled: true 

steps defined here: https://github.com/SAP-samples/btp-developer-guide-cap/blob/main/documentation/observability/4-deploy-to-cf.md#set-up-the-sap-cloud-logging-service.
If cds add mta or adding the telemetry plugin can add the configuration to mta automatically,
It will improve user experience and avoid any chances of error due to miss configured mta.

https://www.npmjs.com/package/prom-client

shall not prohibit other paths! e.g., wrapper packages and env vars...

currently one parent and flat list of children. should be multi-level, i.e., a tree.

then replace timestamp-based sorting with a tree-based sorting.

example where current approach renders incorrect:

Flag HOST_METRICS_LOG_SYSTEM seems to only influence the output to console.
System.* metrics are still received in Cloud Logging.

We only need the process.* metrics and receive a lot of data from cloud foundry.
Host metrics are currently reporting 655 entries per minute that are not even available in the BTP cockpit (more cpu/ram than is assigned to the current instance).

You implemented process.hrtime() for fulfilling startTime and endTime of spans.

The first value of the array represents secends beginning from 1970-01-01 theoretically, but it's not true.

I found two pages describing this issue:

https://stackoverflow.com/questions/74658634/node-process-hrtime-is-five-years-off
https://stackoverflow.com/questions/56597772/what-time-are-the-process-hrtime-and-process-hrtime-bigint-functions-referri

And terminal observability system (I'm using Cloud Logging Service) received tracing data like this:

startTime and endTime are totally wrong.

I suggest you use the time value with the Date type.

Sincerely,

Phil

Hello CAP JS Telemetry Team,

can you please provide some insights how this project is related to the SAP Cloud ALM - Expert Portal - Next Generation – Data Collection Infrastructure? SAP Cloud ALM Provides the not public NPM packages:

• @sap/xotel-agent-ext-js
• @sap/fesr-to-otel-js

which provide additional data beside the Standard OpenTelemetry data. Can this project and the approach described at OpenTelemetry@SAP be combined?

Best Regards
Gregor

Hello,

We are using HANA as the database for our application. I do not see any traces with db sql statements in the traces. I have added the following as dependency:
"@cap-js/hana": "^0.1.0",
"@cap-js/telemetry": "^0.1.0"

Question : Does the database statements from @cap-js/hana gets added to the traces by default, or are there any other steps required?

Thanks and Regards,
Prajul

also: support user provided service

• Instrumentation scope:
  • is: @capire/telemetry 1.0.0
  • java: com.sap.cds (no version!)
  • should: ??? ???
• ...

Hi Team,
I have installed the plugin version "0.0.3". When i am running the command cds watch, i am getting the following error

[cds] - ❗️Uncaught TypeError: Cannot read properties of undefined (reading 'code')
    at onDone (/home/user/projects/incidents-app/node_modules/@cap-js/telemetry/lib/tracing/trace.js:229:23)
    at <anonymous>
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
TypeError: Cannot read properties of undefined (reading 'code')
    at onDone (/home/user/projects/incidents-app/node_modules/@cap-js/telemetry/lib/tracing/trace.js:229:23)
    at /home/user/projects/incidents-app/node_modules/@cap-js/telemetry/lib/tracing/trace.js:239:7
    at NoopContextManager.with (/home/user/projects/incidents-app/node_modules/@cap-js/telemetry/node_modules/@opentelemetry/api/build/src/context/NoopContextManager.js:25:19)
    at ContextAPI.with (/home/user/projects/incidents-app/node_modules/@cap-js/telemetry/node_modules/@opentelemetry/api/build/src/api/context.js:60:46)
    at trace (/home/user/projects/incidents-app/node_modules/@cap-js/telemetry/lib/tracing/trace.js:218:27)
    at OKRAService.process (/home/user/projects/incidents-app/node_modules/@cap-js/telemetry/lib/tracing/okra.js:33:12)
    at OData.process (/home/user/projects/incidents-app/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/OData.js:264:24)
    at Layer.handle [as handle_request] (/home/user/projects/incidents-app/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/user/projects/incidents-app/node_modules/express/lib/router/index.js:328:13)
    at /home/user/projects/incidents-app/node_modules/express/lib/router/index.js:286:9
    

Not sure if it's the app or framework.

As for Dynatrace and Jaeger, there should be a predefined kind for SAP Cloud Logging.

includes (but not limited to):

• config in package.json
• section in readme

telemetry-to-cls is just the working title.

Is it currently foreseen to export to e.g. Dynatrace as well as the Cloud Logging Service at the same time?

Creating new as unable to reopen #91

Hi @sjvans ,

This problem is still reproducible for our multiple apps. We have also upgraded our otel libraries which did not help.
Please find more more details which may be helpful for you to fix the same .
This problem seems to occur for missing handling for below type of traces which are marked as non-recording via sampler or suppressed via ignore path.

Instrumentation suppressed, returning Noop Span
Recording is off, propagating context in a non-recording span
Hence it did not occur in localhost for us. However now we are able to reproduce in localhost as well.
In case you want to reproduce in localhost you can try below 2 options:

In package.json add ignore path for say $batch calls (just to simulate the suppression of trace)
"instrumentations": {
"http": {
"module": "@opentelemetry/instrumentation-http",
"class": "HttpInstrumentation",
"config": {
"ignoreIncomingPaths": [
"/model/$batch"
]
}
}
}
Now this will result in reported exception.

Change sampling decision in debugger for one of the spans for $batch call in localhost
Code -> node_modules/@opentelemetry/sdk-trace-base/build/src/Tracer.js -> line no 88. In debug set the samplingResult.decision to 0 i.e. NOT_RECORD .
Post this when the call will reach
node_modules/@cap-js/telemetry/lib/tracing/trace.js line no 229 , it will result in exception as such Non Recording spans do not have status in them.

Hope these details help you fix the problem

Regards,
Pankaj

Hello Colleagues,

We have upgraded the Intelligent Forecasting app to version 7.8.0 and used the default /health endpoint for liveliness and readiness probe. The probes work correctly but they are not ignored by the telemetry traces resulting in huge number of traces.

The understanding from #102 was that the default /health endpoint traces would be ignored by default and we would not need to add them to ignore paths.

Best Regards,
Prajul

Hey there,

Given the usage of the ConsoleSpanExporter it seems that other validations not relevant to this kind are performed.

So assuming that "kind": "to-dynatrace" is configured for production (or not configured at all) if there's a default env file containing dynatrace credentials at all, the plugin fails the start as:

[cds] - loading plugin @cap-js/telemetry: { impl: 'node_modules/@cap-js/telemetry/cds-plugin.js' }
[telemetry] - @opentelemetry/api: Registered a global for diag v1.6.0.
[telemetry] - @opentelemetry/api: Registered a global for trace v1.6.0.
[telemetry] - @opentelemetry/api: Registered a global for context v1.6.0.
[telemetry] - @opentelemetry/api: Registered a global for propagation v1.6.0.
[telemetry] - Using tracing exporter: ConsoleSpanExporter { _temporaryStorage: Map(0) {} }
[telemetry] - @opentelemetry/instrumentation-http Applying patch for [email protected]
[telemetry] - @opentelemetry/instrumentation-http Applying patch for [email protected]

leading to:

Error: Neither metrics_apitoken nor rest_apitoken.token found in Dynatrace credentials
    at _getExporter (/.../Documents/ServiceRequests/node_modules/@cap-js/telemetry/lib/metrics/index.js:27:23)
    at module.exports (/.../Documents/ServiceRequests/node_modules/@cap-js/telemetry/lib/metrics/index.js:44:20)
    at module.exports (/.../Documents/ServiceRequests/node_modules/@cap-js/telemetry/lib/index.js:27:3)
    at Object.<anonymous> (/.../Documents/ServiceRequests/node_modules/@cap-js/telemetry/cds-plugin.js:1:48)
    at Module._compile (node:internal/modules/cjs/loader:1376:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)
    at Object.require.extensions.<computed> [as .js] (/usr/local/lib/node_modules/ts-node/src/index.ts:1608:43)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1023:12)
    at Module.require (node:internal/modules/cjs/loader:1235:19)

I'd understand this behaviour if it's instructed to export to dynatrace, yet not if the console export is supposed to be used.

Hey,

As it's currently implemented I cant seem to find a way to adjust the temporalityPreference when exporting to Cloud Logging.
For dynatrace this seems to be defaulted to DELTA while it defaults to CUMMULATIVE for Cloud Logging

Would it be possible to have this be set via env like cds.env.requires.telemetry.metrics.config.temporalityPreference?

Marc

Hello, in https://github.com/cap-js/telemetry/blob/main/lib/tracing/trace.js#L94 you're checking if it is an HTTP request by only checking content of cds.context?.http?.req.
What @cap-js-community/event-queue is doing, is creating a context which looks like http context, but actually is not.

Even if this context of the request is inconsistent, I think tracing library should be more resilient and should not throw if input object is incorrect.

Is it possible to add some safeguards there?

UPD: I've also opened an issue in @cap-js-community/event-queue as I'm not sure who should fix this. Maybe you could have an internal discussion on the issue?