Proposal: Add support for Elastic Common Schema (ECS) in OpenTelemetry about oteps

[cyrille-leclerc]

Hello @arminru , thanks for your support.

The primary outcomes we are discussing are :

• To enrich OpenTelemetry Semantic Conventions with the modeling defined by the Elastic Common Schema (ECS). Note that we acknowledge the existence of some overlaps/conflicts between the two schemas and we want to resolve those.
• Then we want to have the enriched Otel Semantic Conventions becoming the new schema of the Elastic ecosystem, becoming the kind of next generation schema

Not in scope for the moment:

• Producing converters between ECS as of today and OpenTelemetry Semantic Conventions as of today (e.g. Otel Collector receiver or exporter for ECS). We don't exclude that it could be a goal for a subsequent milestone.

Or is the intention entirely different and you propose to adapt (rewrite) the OTel semantic conventions to entirely follow ECS instead?
As explained above, rewriting the Otel Semantic conventions introducing many backward incompatible changes is definitivly not the goal.

How are entities present in ECS but missing in OTel treated? Will the missing ones be added to OTel semconv so everything can be mapped or is the intention that those are left untouched?

As described above the goal is to enrich Otel Semantic Conventions with entities present in ECS and not yet covered by Otel.

... as having separate data models could likely lead to complexity and confusion and that consistency would be desirable instead.

We at Elastic share your point of view, we would like to contribute to the enrichment of Otel Semantic Conventions and then adopt these Enriched Otel Semantic Conventions as the new schema of the Elastic ecosystem.

Did I clarify the goals?

Once you deem the proposal complete it would be great if you could open a PR for your OTEP so it can be reviewed and discussed by the community (but Google Docs is fine while you're still drafting it as it makes editing by multiple collaborators easier for you I assume). Thanks!

Moving to a collaboration using GitHub Pull Requests is the plan. We have to define a process, we discussed the idea of an incremental approach to integrate ECS namespaces one after the other. The methodology has to be clarified.

[cyrille-leclerc]

@arminru we have created the PR, please feel free to comment and contribute:

• #199

[arminru]

Thank you for the proposal, Alolita and others!
I read through the document and understand the motivation and advantages but still fail to comprehend what the proposed approach or solution is.
Will the outcome of this be a mapping from data (data types, built-in fields, attribute keys) following ECS to the OTel data model and semantic conventions? Will this mapping be implemented in the OTel collector so it can ingest ECS data and then process and export it just as any other data that would follow the OTel model and semconv in first place?
Will this mapping be bidirectional so data collected from OTel sources can be exported (by the collector) as ECS data?
How are entities present in ECS but missing in OTel treated? Will the missing ones be added to OTel semconv so everything can be mapped or is the intention that those are left untouched?
Or is the intention entirely different and you propose to adapt (rewrite) the OTel semantic conventions to entirely follow ECS instead? If so, would this be constrained to Logs only or should this be extended to all signals? I would assume the latter as having separate data models could likely lead to complexity and confusion and that consistency would be desirable instead. We also need to think about the OTel resources here that logs share with other signal types.
What would the next action items for implementing this proposal be?

[arminru]

Once you deem the proposal complete it would be great if you could open a PR for your OTEP so it can be reviewed and discussed by the community (but Google Docs is fine while you're still drafting it as it makes editing by multiple collaborators easier for you I assume). Thanks!

[AlexanderWert]

We created a new PR to address this proposal: #222