endgameinc / eql Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
Request an additional filter type of limit
to limit the number of records returned from a query.
$ eql query -f my-sysmon-data.json 'ParentImage != null | limit 1' | jq
This would be comparable to the LIMIT
keyword in SQL, returning only the top N records returned by the query. This is useful for getting a handle on the structure and format of the data that the analyst is working with.
When building an OR of two wildcard checks of the same field, there are two separate function calls. But it would be more optimal for many backends if these were grouped up. The AST for wildcard(x, "a", "b", "c") or wildcard(x, "d", "e", "f")
could be rewritten to wildcard(x, "a", "b", "c", "d", "e", "f")
Here's the current behavior:
$ python -m eql optimize 'a == "foo*" or a == "*bar*" or a == "*b*a*z*"'
a == "foo*" or a == "*bar*" or a == "*b*a*z*"
Expected:
$ python -m eql optimize 'a == "foo*" or a == "*bar*" or a == "*b*a*z*"'
wildcard(a, "foo*", "*bar*", "*b*a*z*")
New functionality was added in elasticsearch for sample
, which needs to be supported here.
There's an uncaught exception when macros return fields.
Steps to reproduce the behavior:
macro SELF(a)
)foo where SELF(bar) == "baz"
Should successfully return a type hint instead.
Traceback (most recent call last):
...
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 1262, in parse_analytics
return [parse_analytic(r, preprocessor=preprocessor, **kwargs) for r in analytics]
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 1247, in parse_analytic
query = parse_query(text, preprocessor=preprocessor, **kwargs)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 1198, in parse_query
subqueries=subqueries, pipes=pipes)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 1134, in _parse
eql_node = walker.visit(tree)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 275, in visit
return Interpreter.visit(self, tree)
File "<python root directory>/venv/lib/python2.7/site-packages/lark/visitors.py", line 279, in visit
return f(tree)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 861, in piped_query
first = self.visit(node["base_query"])
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 275, in visit
return Interpreter.visit(self, tree)
File "<python root directory>/venv/lib/python2.7/site-packages/lark/visitors.py", line 279, in visit
return f(tree)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 856, in base_query
return self.visit(node.children[0])
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 275, in visit
return Interpreter.visit(self, tree)
File "<python root directory>/venv/lib/python2.7/site-packages/lark/visitors.py", line 279, in visit
return f(tree)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 827, in event_query
node_info = self.visit(node.children[-1]) # type: NodeInfo
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 275, in visit
return Interpreter.visit(self, tree)
File "<python root directory>/venv/lib/python2.7/site-packages/lark/visitors.py", line 279, in visit
return f(tree)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 662, in and_expr
return self.bool_expr(node, ast.And)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 650, in bool_expr
terms = self.visit_children(node) # type: list[NodeInfo]
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 265, in visit_children
return Interpreter.visit_children(self, tree)
File "<python root directory>/venv/lib/python2.7/site-packages/lark/visitors.py", line 283, in visit_children
for child in tree.children]
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 275, in visit
return Interpreter.visit(self, tree)
File "<python root directory>/venv/lib/python2.7/site-packages/lark/visitors.py", line 279, in visit
return f(tree)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 666, in or_expr
return self.bool_expr(node, ast.Or)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 650, in bool_expr
terms = self.visit_children(node) # type: list[NodeInfo]
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 265, in visit_children
return Interpreter.visit_children(self, tree)
File "<python root directory>/venv/lib/python2.7/site-packages/lark/visitors.py", line 283, in visit_children
for child in tree.children]
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 275, in visit
return Interpreter.visit(self, tree)
File "<python root directory>/venv/lib/python2.7/site-packages/lark/visitors.py", line 279, in visit
return f(tree)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 566, in comparison
left, comp_op, right = self.visit_children(node) # type: (NodeInfo, str, NodeInfo)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 265, in visit_children
return Interpreter.visit_children(self, tree)
File "<python root directory>/venv/lib/python2.7/site-packages/lark/visitors.py", line 283, in visit_children
for child in tree.children]
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 275, in visit
return Interpreter.visit(self, tree)
File "<python root directory>/venv/lib/python2.7/site-packages/lark/visitors.py", line 279, in visit
return f(tree)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 752, in function_call
type_hint = self._get_type_hint(node, expanded)
File "<python root directory>/venv/lib/python2.7/site-packages/eql/parser.py", line 722, in _get_type_hint
ast_node, type_hint = self._update_field_info(NodeInfo(ast_node, source=node))
TypeError: 'NodeInfo' object is not iterable
With the Lark upgrade, would like to bump a new release so downstream projects can leverage the updated dependency.
Looks like the current version isn't tagged on GitHub, it would be nice to tag a release and push to pypi.
Relates to elastic/elasticsearch#56771
String comparisons are always case-insensitive.
The only exceptions: <
<=
>=
>
Consistency should be preserved, and all string comparisons should be case-insensitive.
None
Recently, alias sequences merged into endpoint-dev EQL, which will require updates to support this.
Ex.
sequence by user.name
[process where process.name == "cmd.exe"] as parentEvent
[network where parentEvent.process.id == process.id]
Example of a detection use case of process A running as user1 spawns a child process running as a different user:
sequence with maxspan=1m
[process where event.action == "start" and user.id : "S-1-5-21*"] as event0 by process.entity_id
[process where event.action == "start" and event0.user.id != user.id ] by process.parent.entity_id
EQL is being developed directly in Elasticsearch and can be tracked here: elastic/elasticsearch#51556
We've identified places where we need to tighten the semantics for EQL, make changes to existing behavior, or limit what can be expressed in the language. This meta issue is to track all the changes we need to back port to resolve incongruities between Endpoint and Elasticsearch EQL. This will help prepare users of EQL and Elastic endpoint security to make migration easier.
Changes will be tracked in the feature/backport branch.
null
to be compared to anything. currently, null == 5
is not allowed #18arraySearch
and arrayContains
redundant). this isn't actually supported yet within ES, which only uses scalar values for painless==
and other comparisons to check array membershiptest_queries.toml
file should be updatedSome regular expressions use matching groups, but these are unnecessary and wasteful.
Instead, we should prefer non-matching groups with ?:
.
A few affected places
Related to elastic/elasticsearch#68639
When in eql.parser.elasticsearch_syntax
mode
like
and regex
syntax should be recognized~
) should be recognizedin~
should be recognized=~
, !~
shouldn't be recognized (yet)Otherwise, the above should recognized as invalid syntax
Recently, repeated sequences merged into elasticsearch EQL, which will require updates to support this.
Ex:
sequence
[process where opcode == 1] by unique_pid
[file where opcode == 0] by unique_pid with runs=2
[network where opcode == 0] by unique_pid
Support for missing events was recently added upstream elastic/elasticsearch#92348. We should add support with python eql for rule validation for elasticsearch_syntax (unless we also need to add to endpoint_syntax).
!
maxspan
is included for queries with the negation.New Feature Description
Add short hand for queries, so lengthy query templates can be reused.
This could be accomplished with a custom definition. The name filter
is up for debate, and does currently collide with a filter
pipe.
filter services_child = process where subtype.create and parent_process_name == "services.exe"
you've created a new event type, and can use it:
services_child where process_name == "cmd.exe"
We need to backport elastic/elasticsearch#62650 into Python and Endgame.
Unlike elastic/elasticsearch#62756, this should return a special error message because this is a breaking change within Python EQL. This will also help nudge users towards the new syntax of only ==
.
Semantic errors sometimes miss source text.
$ eql query 'sequence
[process where true]
[file where badFunction()]
[network where true]
[process where true]
[network where true]
'
Error at line:3,column:14
Unknown function badFunction
^^^^^^^^^^^
$ python -m eql query 'sequence
[process where true]
[file where badFunction()]
[network where true]
[process where true]
[network where true]
'
Error at line:3,column:14
Unknown function badFunction
sequence
[process where true]
[file where badFunction()]
^^^^^^^^^^^
[network where true]
[process where true]
[network where true]
It seems that packages are being deployed to pypi but there is no changelog, release notes, or tags in git for releases. Is it possible to have details on what has changed in each release when it happens?
On the Elasticsearch side, the :
syntax has been updated. We should update the syntax for :
here. Currently, this syntax is only recognized when opting-in to the Elasticsearch grammar. See #40
On the right side of :
:
Part of #15
We need to implement three-value logic for null
, true
and false
consistent with Elasticsearch EQL and Elasticsearch SQL.
Some necessary changes
== null
or != null
(plus commutative versions)
null
return null (< null
, <= null
, etc)process_name == parent_process_name
will be null even if both values are null. This is a change from the current implementation... and false
=> false, and ... or true
=> trueAnd of course, update existing tests to be consistent
We would like to add support for IPv6 to the CidrMatch function to better enable usage of the eql library to evaluate queries that use CidrMatch with IPv6 addresses to better match the features of Elasticsearch.
For more implementation specific details please see the PR.
Support was added to the elasticsearch implementation: elastic/elasticsearch#79677
Part of #15
Update the grammar to accept `backtick` fields. Dots between the backticks will be interpreted as part of the field name, not as an additional attribute.
Add a feature flag to accept the syntax for the Elasticsearch changes to EQL.
Leave the runtime behavior as-is for now, but we should be able to validate with the Elasticsearch 7.10 changes. This'll be necessary for elastic/detection-rules unit testing.
Changes to syntax with this flag:
=
not allowed with ==
?"
and ?'
not allowed:
is allowed"""
allowed.We could also use the toggle to be extra strict and disable additional functions, ancestry, etc.
FYI @brokensound77
Part of #15
Per elastic/elasticsearch#55032, some sequence behavior is no longer supported and some changes need to be backported. This should be done in a way that still recognizes previous syntax but has custom error messages, to ease the migration path.
h
but no longer hours
)Yes, we can remove the case sensitive parameter from this function.
The function uses case sensitivity when finding the matching substrings
left
andright
withinsource
. But it always returns the the substring between those in its original case. If we remove that parameter, we'll have to figure out how to perform case-(in)sensitive matching while returning the original string. It's not as simple asbetween(lower(source), lower(left), lower(right))
.
When
greedy=false
, that makes the function roughly equivalent to
substring(source, // notice the original casing
indexOf(lower(source), lower(left)),
indexOf(lower(source), lower(right), indexOf(lower(source), lower(right), lower(left)))
)
Originally posted by @rw-access in elastic/elasticsearch#54411 (comment)
There are cases where we want to get subfields from within a macro, to make things more flexible and composable.
One example:
macro IS_TRUSTED(authenticode)
nested_authenticodeauth_info.signature_signer == "Microsoft Corportation" and authenticode.signature_status == "trusted"
This will make it easier to do things like
IS_TRUSTED(events[0])
IS_TRUSTED(data.triggering_fact_array.events[5].data_buffer)
IS_TRUSTED(SOURCE_PROCESS())
None of the core features (see changelog) have unit tests to test the evaluation works as expected with the PythonEngine. We should add more unit tests to ensure features have small sample data loaded and evaluate as expected.
The new as
keyword introduced in 0.9.14
overlaps with the ecs as fields and causes an error if events (e.g. client.as.organization.name.text
) use as fields.
Run eql.parse_query
with any query that has as
fields.
This edge case for keywords should not throw an Invalid use of keyword
error.
Relates to #15
Currently in EQL, all divisions are forced to be floating-point divisions, with no way to opt-out. Instead, if both the dividend and divisor are integers, we should use integer division. We should follow the behavior 3 / 2 ==> 1
instead of 3 / 2 ==> 1.5
.
To continue using, floating point division, make either the dividend or divisor a float/double.
We'll also need to update the tests for changing this behavior.
For better practice, the optimize step should be separated from converting the Lark tree to the EQL tree. Instead of optimizing along the way, it should be completed as a separate phase.
With this, we can also disable optimizations entirely. That will enable better testing, manipulation and conversion on the unmodified tree.
Based on some feedback, the tests should be simplified.
Every test must specify case_sensitive=true
, case_insensitive=true
or both.
Currently, tests that support both do not specify, but this implicit behavior is a little backwards. Instead, we should always be explicit. Tests that work for both will look like:
case_sensitive = true
case_insensitive = true
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.