cap-js / telemetry Goto Github PK
View Code? Open in Web Editor NEWCDS plugin providing observability features, incl. automatic OpenTelemetry instrumentation.
Home Page: https://cap.cloud.sap/docs
License: Apache License 2.0
CDS plugin providing observability features, incl. automatic OpenTelemetry instrumentation.
Home Page: https://cap.cloud.sap/docs
License: Apache License 2.0
see @opentelemetry/instrumentation-runtime-node
should we do the same as with @opentelemetry/host-metrics?
Currently the plugin seems to crash with an uncaught exception in case of missing information. That should maybe be catched as missing telemetry shouldn't be a show stopper but a warning. Also it seems like the actual api token name required deviates from the readme file (for version 0.0.2).
... uncaughtException: Neither metrics_apitoken nor rest_apitoken.token found in Dynatrace credentials\nError: Neither metrics_apitoken nor rest_apitoken.token found in Dynatrace credentials\n at _getExporter (/home/vcap/deps/0/node_modules/@cap-js/telemetry/lib/metrics/index.js:27:23)\n
Yet the documentation describes:
Excerpt from example mta.yaml:
requires:
- name: my-dynatrace-instance
parameters:
config:
tokens:
- name: ingest_apitoken #> default lookup name, configurable via cds.requires.telemetry.token_name
scopes:
- metrics.ingest
Which is present in the credentials.
I implemented two custom spans (using this https://opentelemetry.io/docs/languages/js/instrumentation/#acquiring-a-tracer as the example) just to see how it looks like in the logs:
const { trace } = require("@opentelemetry/api");
const tracer = trace.getTracer('@cap-js/telemetry');
module.exports = class AdminService extends cds.ApplicationService {
init() {
const { Books } = this.entities
...
this.before(['CREATE', 'UPDATE'], Books, (req) => {
const isValid = tracer.startActiveSpan('my-book-validation', (span) => {
let result = true
if (req.data.stock && req.data.stock < 0) {
result = false
}
span.end();
return result;
});
if (!isValid) {
req.reject(400, 'Stock must be positive');
}
});
this.after("READ", Books, (books) => {
tracer.startActiveSpan('my-after-handler-on-books', async (span) => {
for (let item of books) {
let result = await SELECT.one.from(Books).where({ ID: item.ID });
item.title = result.title + "___my____";
}
span.end();
});
});
return super.init();
}
};
The output of cds watch
does not include my custom spans, but I can see them in Jaeger, for example. Is this a bug or a feature?
Hello, currently when we're setting log level to anything other than "info" it corrupts handle method for every service existing:
Line 76 in 337e92a
What happens here, you're wrapping handle method for Service class in https://github.com/cap-js/telemetry/blob/main/lib/tracing/cds.js#L39, but then later cds itself wrap handle method for ApplicationService https://github.tools.sap/cap/cds/blob/main/libx/_runtime/fiori/lean-draft.js#L208 to handle lean draft, so by the moment you're trying to "unwrap" there is no __original
property in handle
anymore. So any service call would fail with:
[cds] - TypeError: this.handle is not a function
at CatalogService.dispatch (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/lib/srv/srv-dispatch.js:45:15)
at _readCollection (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/handlers/read.js:250:28)
at _readAndTransform (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/handlers/read.js:456:12)
at /Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/handlers/read.js:544:22
at /Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/okra/odata-server/core/Dispatcher.js:135:7
at new Promise (<anonymous>)
at Dispatcher._handle (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/okra/odata-server/core/Dispatcher.js:134:12)
at Dispatcher.dispatch (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/okra/odata-server/core/Dispatcher.js:91:21)
at DispatcherCommand.execute (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/okra/odata-server/invocation/DispatcherCommand.js:63:10)
at CommandExecutor._execute (/Users/dmitry/sap/telemetry/test/bookshop/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/okra/odata-server/invocation/CommandExecutor.js:71:17) {
id: '1931710',
level: 'ERROR',
timestamp: 1710410306617
}
To test it just add:
"log": {
"levels": {
"cds": "error"
}
},
to your https://github.com/cap-js/telemetry/blob/main/test/bookshop/package.json
, run the server and try to access any service.
Not sure, that tracing should depend on "cds" log level at all, and also unwrapping should be done with keeping in mind, there could be other wrappers around.
Hello,
we have following configuration:
ingest_otlp
configurationcds.env.requires.telemetry.kind="to-dynatrace"
If we start our backend it crashes:
2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR Error: No OpenTelemetry credentials found in SAP Cloud Logging binding. Create instance with config: "{ ingest_otlp: { enabled: true } }".
2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at getCloudLoggingCredentials (/app/node_modules/@cap-js/telemetry/lib/utils.js:103:13)
2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at _getExporter (/app/node_modules/@cap-js/telemetry/lib/metrics/index.js:33:15)
2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at module.exports (/app/node_modules/@cap-js/telemetry/lib/metrics/index.js:61:20)
2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at module.exports (/app/node_modules/@cap-js/telemetry/lib/index.js:27:3)
2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at Object.<anonymous> (/app/node_modules/@cap-js/telemetry/cds-plugin.js:1:88)
2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at Module._compile (node:internal/modules/cjs/loader:1241:14)
2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at Module._extensions..js (node:internal/modules/cjs/loader:1295:10)
2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at Module.load (node:internal/modules/cjs/loader:1091:32)
2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at Module._load (node:internal/modules/cjs/loader:938:12)
2024-03-18T09:34:46.59+0100 [APP/PROC/WEB/0] ERR at H.loadModule (/opt/dynatrace/oneagent/agent/bin/1.275.146.20231002-095820/any/nodejs/nodejsagent.js:16603:70)
2024-03-18T09:34:46.79+0100 [APP/PROC/WEB/0] OUT Exit status 1
I believe getCloudLoggingCredentials could behave better and do not throw if we have no intention to send any telemetry to Cloud Logging.
What do you think?
Some description
https://opentelemetry.io/docs/specs/otel/metrics/api/#get-a-meter
currently we do metrics.getMeter(${cds._telemetry.name}-meter
)
either cds._telemetry.name
has to be made public or we need something else...
the name should be an Instrumentation Scope -> why not simply @cap-js/telemetry
?
and remove from name
-> we need a new name that produces readable output on console
Check if we can update @opentelemetry/* dependencies
(+ improve dependabot settings?)
(+ improve test setup and coverage?)
Our application leverages Express and CAP frameworks. CAP is initialized during the startup of Express. Both services, the application service, and the CAP service require tracing capabilities. The application service utilizes OpenTelemetry with HTTP instrumentation, while the CAP service utilizes @cap-js/telemetry, also with HTTP instrumentation. Both services share the same OTEL_EXPORTER_OTLP_TRACES_ENDPOINT for exporting trace data.
During testing, we've encountered difficulties with simultaneously sending trace data from both services to OTEL_EXPORTER_OTLP_TRACES_ENDPOINT. However, when sent separately, each service's trace data is successfully exported.
We aim for both the application service and CAP service to be effectively traced and exported to OTEL_EXPORTER_OTLP_TRACES_ENDPOINT.
example: @opentelemetry/exporter-prometheus
with mocked auth... probably the identical pool in set of pools. tenant information only needed in multitenant mode.
Hi,
Thanks for this extension. Is there any plan for on-the-fly config of tracing ratio (e.g. load config from a file at an interval)? Currently we can use environment variable to change the number. But this means in system like k8s, the change of tracing ratio will cause rolling update of pod, which will increase the risks of failure.
Thanks,
Samuel
Hi,
Currently if we want to deploy application to the cloud foundry runtime + cloud logging service, we need to manually add the service instance configuration and binding to srv. This does leads to error if someone miss configures the ingest configuration.
- name: incidents-cloud-logging
type: org.cloudfoundry.managed-service
parameters:
service: cloud-logging
service-plan: standard
config:
ingest_otlp:
enabled: true
steps defined here: https://github.com/SAP-samples/btp-developer-guide-cap/blob/main/documentation/observability/4-deploy-to-cf.md#set-up-the-sap-cloud-logging-service.
If cds add mta or adding the telemetry plugin can add the configuration to mta automatically,
It will improve user experience and avoid any chances of error due to miss configured mta.
shall not prohibit other paths! e.g., wrapper packages and env vars...
currently one parent and flat list of children. should be multi-level, i.e., a tree.
then replace timestamp-based sorting with a tree-based sorting.
Flag HOST_METRICS_LOG_SYSTEM seems to only influence the output to console.
System.* metrics are still received in Cloud Logging.
We only need the process.* metrics and receive a lot of data from cloud foundry.
Host metrics are currently reporting 655 entries per minute that are not even available in the BTP cockpit (more cpu/ram than is assigned to the current instance).
You implemented process.hrtime()
for fulfilling startTime and endTime of spans.
The first value of the array represents secends
beginning from 1970-01-01 theoretically, but it's not true.
I found two pages describing this issue:
https://stackoverflow.com/questions/74658634/node-process-hrtime-is-five-years-off
https://stackoverflow.com/questions/56597772/what-time-are-the-process-hrtime-and-process-hrtime-bigint-functions-referri
And terminal observability system (I'm using Cloud Logging Service) received tracing data like this:
startTime
and endTime
are totally wrong.
I suggest you use the time value with the Date
type.
Sincerely,
Phil
Hello CAP JS Telemetry Team,
can you please provide some insights how this project is related to the SAP Cloud ALM - Expert Portal - Next Generation – Data Collection Infrastructure? SAP Cloud ALM Provides the not public NPM packages:
which provide additional data beside the Standard OpenTelemetry data. Can this project and the approach described at OpenTelemetry@SAP be combined?
Best Regards
Gregor
Line 76 in 337e92a
Hello,
We are using HANA as the database for our application. I do not see any traces with db sql statements in the traces. I have added the following as dependency:
"@cap-js/hana": "^0.1.0",
"@cap-js/telemetry": "^0.1.0"
Question : Does the database statements from @cap-js/hana gets added to the traces by default, or are there any other steps required?
Thanks and Regards,
Prajul
telemetry/lib/metrics/index.js
Line 27 in 44dae09
also: support user provided service
Hi Team,
I have installed the plugin version "0.0.3". When i am running the command cds watch, i am getting the following error
[cds] - ❗️Uncaught TypeError: Cannot read properties of undefined (reading 'code')
at onDone (/home/user/projects/incidents-app/node_modules/@cap-js/telemetry/lib/tracing/trace.js:229:23)
at <anonymous>
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
TypeError: Cannot read properties of undefined (reading 'code')
at onDone (/home/user/projects/incidents-app/node_modules/@cap-js/telemetry/lib/tracing/trace.js:229:23)
at /home/user/projects/incidents-app/node_modules/@cap-js/telemetry/lib/tracing/trace.js:239:7
at NoopContextManager.with (/home/user/projects/incidents-app/node_modules/@cap-js/telemetry/node_modules/@opentelemetry/api/build/src/context/NoopContextManager.js:25:19)
at ContextAPI.with (/home/user/projects/incidents-app/node_modules/@cap-js/telemetry/node_modules/@opentelemetry/api/build/src/api/context.js:60:46)
at trace (/home/user/projects/incidents-app/node_modules/@cap-js/telemetry/lib/tracing/trace.js:218:27)
at OKRAService.process (/home/user/projects/incidents-app/node_modules/@cap-js/telemetry/lib/tracing/okra.js:33:12)
at OData.process (/home/user/projects/incidents-app/node_modules/@sap/cds/libx/_runtime/cds-services/adapter/odata-v4/OData.js:264:24)
at Layer.handle [as handle_request] (/home/user/projects/incidents-app/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/home/user/projects/incidents-app/node_modules/express/lib/router/index.js:328:13)
at /home/user/projects/incidents-app/node_modules/express/lib/router/index.js:286:9
Not sure if it's the app or framework.
incidents | https://github.com/cap-js/incidents-app |
---|---|
Node.js | v18.14.2 |
@sap/cds | 7.6.1 |
@sap/cds-compiler | 4.5.0 |
@sap/cds-dk | 7.5.0 |
@sap/cds-dk (global) | 7.5.1 |
@sap/eslint-plugin-cds | 2.6.4 |
@sap/cds-mtxs | 1.14.0 |
As for Dynatrace and Jaeger, there should be a predefined kind for SAP Cloud Logging.
includes (but not limited to):
package.json
telemetry-to-cls
is just the working title.
Is it currently foreseen to export to e.g. Dynatrace as well as the Cloud Logging Service at the same time?
Creating new as unable to reopen #91
Hi @sjvans ,
This problem is still reproducible for our multiple apps. We have also upgraded our otel libraries which did not help.
Please find more more details which may be helpful for you to fix the same .
This problem seems to occur for missing handling for below type of traces which are marked as non-recording via sampler or suppressed via ignore path.
Instrumentation suppressed, returning Noop Span
Recording is off, propagating context in a non-recording span
Hence it did not occur in localhost for us. However now we are able to reproduce in localhost as well.
In case you want to reproduce in localhost you can try below 2 options:
In package.json add ignore path for say $batch calls (just to simulate the suppression of trace)
"instrumentations": {
"http": {
"module": "@opentelemetry/instrumentation-http",
"class": "HttpInstrumentation",
"config": {
"ignoreIncomingPaths": [
"/model/$batch"
]
}
}
}
Now this will result in reported exception.
Change sampling decision in debugger for one of the spans for $batch call in localhost
Code -> node_modules/@opentelemetry/sdk-trace-base/build/src/Tracer.js -> line no 88. In debug set the samplingResult.decision to 0 i.e. NOT_RECORD .
Post this when the call will reach
node_modules/@cap-js/telemetry/lib/tracing/trace.js line no 229 , it will result in exception as such Non Recording spans do not have status in them.
Hope these details help you fix the problem
Regards,
Pankaj
Hello Colleagues,
We have upgraded the Intelligent Forecasting app to version 7.8.0 and used the default /health endpoint for liveliness and readiness probe. The probes work correctly but they are not ignored by the telemetry traces resulting in huge number of traces.
The understanding from #102 was that the default /health endpoint traces would be ignored by default and we would not need to add them to ignore paths.
Best Regards,
Prajul
Hey there,
Given the usage of the ConsoleSpanExporter it seems that other validations not relevant to this kind are performed.
So assuming that "kind": "to-dynatrace"
is configured for production (or not configured at all) if there's a default env file containing dynatrace credentials at all, the plugin fails the start as:
[cds] - loading plugin @cap-js/telemetry: { impl: 'node_modules/@cap-js/telemetry/cds-plugin.js' }
[telemetry] - @opentelemetry/api: Registered a global for diag v1.6.0.
[telemetry] - @opentelemetry/api: Registered a global for trace v1.6.0.
[telemetry] - @opentelemetry/api: Registered a global for context v1.6.0.
[telemetry] - @opentelemetry/api: Registered a global for propagation v1.6.0.
[telemetry] - Using tracing exporter: ConsoleSpanExporter { _temporaryStorage: Map(0) {} }
[telemetry] - @opentelemetry/instrumentation-http Applying patch for [email protected]
[telemetry] - @opentelemetry/instrumentation-http Applying patch for [email protected]
leading to:
Error: Neither metrics_apitoken nor rest_apitoken.token found in Dynatrace credentials
at _getExporter (/.../Documents/ServiceRequests/node_modules/@cap-js/telemetry/lib/metrics/index.js:27:23)
at module.exports (/.../Documents/ServiceRequests/node_modules/@cap-js/telemetry/lib/metrics/index.js:44:20)
at module.exports (/.../Documents/ServiceRequests/node_modules/@cap-js/telemetry/lib/index.js:27:3)
at Object.<anonymous> (/.../Documents/ServiceRequests/node_modules/@cap-js/telemetry/cds-plugin.js:1:48)
at Module._compile (node:internal/modules/cjs/loader:1376:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)
at Object.require.extensions.<computed> [as .js] (/usr/local/lib/node_modules/ts-node/src/index.ts:1608:43)
at Module.load (node:internal/modules/cjs/loader:1207:32)
at Function.Module._load (node:internal/modules/cjs/loader:1023:12)
at Module.require (node:internal/modules/cjs/loader:1235:19)
I'd understand this behaviour if it's instructed to export to dynatrace, yet not if the console export is supposed to be used.
Hey,
As it's currently implemented I cant seem to find a way to adjust the temporalityPreference when exporting to Cloud Logging.
For dynatrace this seems to be defaulted to DELTA while it defaults to CUMMULATIVE for Cloud Logging
telemetry/lib/metrics/index.js
Lines 29 to 37 in fcac18d
Would it be possible to have this be set via env like cds.env.requires.telemetry.metrics.config.temporalityPreference
?
Marc
Hello, in https://github.com/cap-js/telemetry/blob/main/lib/tracing/trace.js#L94 you're checking if it is an HTTP request by only checking content of cds.context?.http?.req
.
What @cap-js-community/event-queue is doing, is creating a context which looks like http context, but actually is not.
Even if this context of the request is inconsistent, I think tracing library should be more resilient and should not throw if input object is incorrect.
Is it possible to add some safeguards there?
UPD: I've also opened an issue in @cap-js-community/event-queue as I'm not sure who should fix this. Maybe you could have an internal discussion on the issue?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.