Comments (11)
- By upgrading the stack version from 8.12 to 8.14 we may need to remove the ECS field definitions (ecs.yml file). We may also need to validate the
ecs@mapping
behaviour. - Starting from 8.13 the
ecs@mapping
component template is referenced by any integration as mentioned here.
cc: @zmoog
from integrations.
As Ishleen mentions, I think this is a similar case to #9453 (comment). Summarizing, elastic-package ignores validation on some fields for legacy reasons when these fields are not defined, this includes event.*
. This package didn't have definitions for some of these fields. Now that it supports only versions >= 8.13, it includes all ECS definitions, and these validations apply.
The possible solutions would be similar too:
- Adapt the values of
event.type
according to what ECS expects. But this could be a breaking change if there is something working with these values. - Define
event.type
and/orevent.category
according to the expectations of this package, so the definition in ECS is not used for validation.
from integrations.
Pinging @elastic/integrations (Team:Integrations)
from integrations.
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
from integrations.
With the above-mentioned changes applied, you can observe successful pipeline execution run.
ES version : 8.14
--- Test results for package: aws - START ---
╭─────────┬─────────────────┬───────────┬────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │
├─────────┼─────────────────┼───────────┼────────────────────────────┼────────┼──────────────┤
│ aws │ cloudfront_logs │ pipeline │ test-cloudfront.log │ PASS │ 10.098706ms │
│ aws │ cloudfront_logs │ pipeline │ (ingest pipeline warnings) │ PASS │ 571.062007ms │
╰─────────┴─────────────────┴───────────┴────────────────────────────┴────────┴──────────────╯
from integrations.
For AWS-WAF, I have noticied below mentioned error additionally
--- Test results for package: aws - START ---
FAILURE DETAILS:
aws/waf test-waf.log:
[0] parsing field value failed: field "event.type" value "allowed" is not one of the expected values (access, error, info) for any of the values of "event.category" (web)
[1] parsing field value failed: field "event.type" value "denied" is not one of the expected values (access, error, info) for any of the values of "event.category" (web)
So, it may be determined what append
processor values may be appropriate
But, I do notice the allowed value list: https://www.elastic.co/guide/en/ecs/current/ecs-allowed-values-event-type.html
which mentions values such as allowed
and denied
are supported.
from integrations.
Closing the issue as the issue is not reproducible.
from integrations.
Reopening the issue, as i could recreate the issue.
The mandatory condition to recreate the issue is
kibana:
version: "^8.14.0"
from integrations.
@jsoriano , do you know why the issue mentioned here occur?
from integrations.
This issue will occur from 8.13 onwards, where the default ECS mappings are now applied to fields. Previously, event.category
was not explicitly defined in ecs.yml, so it was treated as a keyword. The Elastic package did not previously flag such errors for undefined fields.
For your information, this upgrade of ECS mappings will be addressed as part of this issue. However, if this change is a prerequisite for your task, please feel free to make the necessary adjustments.
from integrations.
Related issue : #10135
from integrations.
Related Issues (20)
- Palo Alto NGFW Integration throwing errors HOT 3
- [CSPM] Update documentation and manifest to include supported platforms
- [M365 Defender] Change `event.type` of `AlertInfo` HOT 1
- [AWS] Create AWS Health Integration package HOT 3
- [stormshield] Follow-up tasks for new integration HOT 1
- [Azure OpenAI] Add PTU Metrics Dashboard
- [Azure OpenAI] Enable Advance Logging
- [RFC] Crowdstrike unified Device ID HOT 3
- [EA Input] Adding support for Jamf as a provider
- [sql_input] Support for SAP Hana
- [Auth0] Update Auth0 Integration to use Management API for Log Ingestion HOT 8
- M365 Defender Integration: Rather than removing HKEY_CURRENT_USER and and HKEY_LOCAL_MACHINE, replace them with HKCU and HKLM to avoiding breaking detection rules HOT 2
- Microsoft Exchange Online Message Trace documentation HOT 1
- AWS Firehose endpoint returning 200 even when ingestion is failing
- [AWS] AWS ELB metrics integration OOTB Dashboard does not reflect ALBs data for Inbound and Outbound Traffic Visualisations HOT 11
- [Prometheus] Add HTTP config options to all data streams
- [Meta] AWS ELB Integration Improvements
- [Citrix ADC] Syslog messages are not according to documentation HOT 6
- Add full text search on specified fields in the Okta integration. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from integrations.