mfesiem / msiempy Goto Github PK
View Code? Open in Web Editor NEWMcAfee SIEM API Python wrapper
Home Page: https://mfesiem.github.io/docs/msiempy/msiempy.html
License: MIT License
McAfee SIEM API Python wrapper
Home Page: https://mfesiem.github.io/docs/msiempy/msiempy.html
License: MIT License
Typo causes "Interpolation failed probably because of the private API calls formatting... Unexpected behaviours can happend." Error.
Solution
"ds_last_times": ("QRY%5FGETDEVICELASTALERTTIME","""{}"""),
Should be:
"ds_last_times": ("QRY_GETDEVICELASTALERTTIME","""{}"""),
In the interest of creating a more stable live test environment, what are the requirements for a test ESM? It should have alarms, watchlists, events, etc - going how far back? Ideally testing would be limited to shorter timeframes but this requires some amount of intent to ensure those conditions always exist on the ESM for the test. Can we define what needs to exist for tests not to fail?
Describe
Like in the code snippet, the add()
method should call dsAddDataSourcesStatus
, make sure there is no error or raise an Exception if the Datasource were not added.
Only if special argument check_added=True
to keep compatibility.
** msiempy versions:**
Additional context
if ds.get('client', None):
print("Adding Client Datasource: {}".format(ds))
resp = devtree.add_client(ds)
else:
print("Adding Datasource: {}".format(ds))
resp = devtree.add(ds)
if not resp:
print('Something went wrong, Datasource {} not added.')
continue
else:
# Wait for the add DS query to execuite ...
time.sleep(1)
ds_status = NitroSession().api_request('dsAddDataSourcesStatus', {"jobId": resp}, retry=0)
if not isinstance(ds_status, dict):
print('Something went wrong, Datasource {} not added.\n{}'.format(ds['name'], ds_status))
continue
while not ds_status['jobStatus'] == 'COMPLETE':
time.sleep(1)
ds_status = NitroSession().api_request('dsAddDataSourcesStatus', {"jobId": resp}, retry=0)
if len(ds_status['unsuccessfulDatasources'])>0:
print('Something went wrong, Datasource {} not added. {}'.format(ds['name'], ds_status['unsuccessfulDatasources'][0]))
continue
else:
ds_to_verify.append(ds['name'])
devtree.refresh()
Describe
When trying to create a datasource object from id, the KeyError
pops up.
Workaround: use DevTree:
devtree = DevTree()
ds = list(devtree.search_ds_group(field='ds_id', term='144116290808709120'))
if len(ds): ds=ds[0]
else: print("Datasource not found")
SIEM and msiempy versions:
Additional context
>>> from msiempy import DataSource
>>> d=DataSource(id=str(144116290808709120))
INFO - Logged into ESM url with username NGCP. Last login 09/25/2020 03:12:30
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/types.py", line 101, in __init__
"""Call the __prepare__ method of the appropriate metaclass.
File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/device.py", line 1134, in data_from_id
File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/device.py", line 1198, in _map_parameters
KeyError: 'desc_id'
Line 469 in 735863c
Shouldn't be IPS_GETFLOWSSNOW
? @andywalden @rhart211
First of all, thank you for your work on this project. There's not a lot of resources out there for this kind of stuff.
I'm running the example event query script on the README.md
page, but I run into this msiempy.NitroError: Sorry the filters must be either a tuple(fiels, [values]) or a QueryFilter sub class.
error.
import msiempy.event
import msiempy.query
filter = msiempy.query.FieldFilter(name='DstIP',values=['10.1.13.0/24'],operator='IN')
customfilter = filter.config_dict
events = msiempy.event.EventManager(
time_range='LAST_MINUTE',
filters=[
msiempy.query.FieldFilter('DstIP', ['8.8.0.0/8',]),
msiempy.query.FieldFilter('HostID', ['mydomain.local'], operator='CONTAINS') ],
limit=1,
max_query_depth=1)
events.load_data(delta='2h', slots='4', workers=5)
print(events.get_text(fields=['Alert.SrcPort', 'Rule.NormID']))
INFO - New ESM session instance is created with : [REDACTED]
Traceback (most recent call last):
File "event.py", line 13, in <module>
max_query_depth=1)
File "/usr/local/lib/python3.6/dist-packages/msiempy/event.py", line 141, in __init__
super(self.__class__, self.__class__).filters.__set__(self, filters)
File "/usr/local/lib/python3.6/dist-packages/msiempy/__init__.py", line 1460, in filters
self.add_filter(f)
File "/usr/local/lib/python3.6/dist-packages/msiempy/event.py", line 195, in add_filter
raise NitroError("Sorry the filters must be either a tuple(fiels, [values]) or a QueryFilter sub class.")
msiempy.NitroError: Sorry the filters must be either a tuple(fiels, [values]) or a QueryFilter sub class.
Removing the filters
keys seems to work, however.
NitroSession
should handle correctly SIEM errors :
Currently, calls to NitroSession.esm_request
that raises requests.HTTPError
errors can raise NitroError
if the error is unhandled.
See docs : https://mfesiem.github.io/docs/msiempy/index.html#msiempy.NitroSession.esm_request
Unhandled 400 errors :
com.mcafee.siem.api.data.query.EsmTimeRange
java.lang.String
out of START_OBJECT tokencom.mcafee.siem.api.data.alert.EsmAlertId
Unhandled 500 errors :
Connexion errors -> Raises !
Describe
The library should handle a easy way to Write Data Sources and Roll Policy.
May be create a DevTree.write_policy()
method?
May be also create a DevTree.add(write_policy=True)
argument that will call the method? (See #82 for another DevTree.add()
improvement)
SIEM and msiempy versions:
Additional context
The APIs that need to be called are dsWriteThirdpartyConfig
, plcyRollPolicy
, and miscJobStatus
Describe
Datasource fails to delete.
SIEM and msiempy versions:
Additional context
./dstools.py --remove 144116290775154688
DEBUG - Calling nitro request : get_devtree kwargs={}
DEBUG - Calling nitro request : login kwargs=***
DEBUG - Requesting HTTP post login ***
DEBUG - Returning raw requests Response object : <Response [201]>
DEBUG - Unpacking SIEM response: {
"privileges" : {
"master" : true,
"admin" : false,
"power" : false,
"audit" : false,
"crypto" : false,
"systemSettings" : {
"read" : true,
"write" : true
},
DEBUG - Calling nitro request : build_stamp kwargs={}
DEBUG - Requesting HTTP post essmgtGetBuildStamp with data None
DEBUG - Unpacking SIEM response: { "return":
{
"buildStamp" : "11.3.0 20191109004423"
} }
DEBUG - <Response [200]> -> Result (<class 'dict'>): {'buildStamp': '11.3.0 20191109004423'}
INFO - Logged into ESM URL with username NGCP. Last login 09/25/2020 17:11:31
DEBUG - Requesting HTTP post GRP_GETVIRTUALGROUPIPSLISTDATA with data {'ITEMS': '#{DC1 + DC2}', 'DID': '1', 'HD': 'F', 'NS': '0'}
DEBUG - Private API call : GRP_GETVIRTUALGROUPIPSLISTDATA Formatted params : Request=API%13GRP_GETVIRTUALGROUPIPSLISTDATA%13%14ITEMS%13#{DC1 + DC2}%13%14DID%131%13%14HD%13F%13%14NS%130%13%14
DEBUG - Unpacking SIEM response: Response=EC%130%13%14AC%130%13%14AF%13F%13%14FC%130%13%14FF%13F%13%14LC%130%13%14LF%13F%13%14DF%130%13%14DNAME%13Physical%20Display%13%14NS%131%13%14ITEMS%1314%11Local%20ESM%11144115188075855872%110%1
DEBUG - <Response [200]> -> Result (<class 'dict'>): {'EC': '0', 'AC': '0', 'AF': 'F', 'FC': '0', 'FF': 'F', 'LC': '0', 'LF': 'F', 'DF': '0', 'DNAME': 'Physical Display', 'NS': '1', 'ITEMS': '14%11Local%20ESM%11144115188075855872%110%11T%11T%11T%11T%11T
DEBUG - Calling nitro request : get_zones_devtree kwargs={}
DEBUG - Requesting HTTP post GRP_GETVIRTUALGROUPIPSLISTDATA with data {'ITEMS': '#{DC1 + DC2}', 'DID': '3', 'HD': 'F', 'NS': '0'}
DEBUG - Private API call : GRP_GETVIRTUALGROUPIPSLISTDATA Formatted params : Request=API%13GRP_GETVIRTUALGROUPIPSLISTDATA%13%14ITEMS%13#{DC1 + DC2}%13%14DID%133%13%14HD%13F%13%14NS%130%13%14
DEBUG - Unpacking SIEM response: Response=EC%130%13%14AC%130%13%14AF%13F%13%14FC%130%13%14FF%13F%13%14LC%130%13%14LF%13F%13%14DF%130%13%14DNAME%13Zone%20Display%13%14NS%131%13%14ITEMS%131%11Undefined%114294967295%11T%1117%111%11%123%
DEBUG - <Response [200]> -> Result (<class 'dict'>): {'EC': '0', 'AC': '0', 'AF': 'F', 'FC': '0', 'FF': 'F', 'LC': '0', 'LF': 'F', 'DF': '0', 'DNAME': 'Zone Display', 'NS': '1', 'ITEMS': '1%11Undefined%114294967295%11T%1117%111%11%123%11CS%20CEF%1114411
DEBUG - Calling nitro request : zonetree kwargs={}
DEBUG - Requesting HTTP post zoneGetZoneTree with data None
DEBUG - Unpacking SIEM response: { "return":
[ ] }
DEBUG - <Response [200]> -> Result (<class 'list'>): []
DEBUG - Calling nitro request : ds_last_times kwargs={}
DEBUG - Requesting HTTP post QRY_GETDEVICELASTALERTTIME with data {}
DEBUG - Private API call : QRY_GETDEVICELASTALERTTIME Formatted params : Request=API%13QRY_GETDEVICELASTALERTTIME%13%14
DEBUG - Unpacking SIEM response: Response=EC%130%13%14NS%1316%13%14ITEMS%13CS%20CEF%11144116288594116608%11Common%20Event%20Format%1103%2F11%2F2020%2011%3A54%3A02%11%12DC01_DNS%11144116290775154688%11Linux%11%11%12Local%20Receiver-EL
DEBUG - <Response [200]> -> Result (<class 'dict'>): {'EC': '0', 'NS': '16', 'ITEMS': 'CS%20CEF%11144116288594116608%11Common%20Event%20Format%1103%2F11%2F2020%2011%3A54%3A02%11%12DC01_DNS%11144116290775154688%11Linux%11%11%12Local%20Receiver-ELM%111441
Delete the datasource and all the data?
12, 3, DC01_DNS, 144116290775154688, True, 10.10.1.34, , 65, , Linux, , , , , 0, , 0, False, Local Receiver-ELM, 144116287587483648, never
[y/n]y
DEBUG - Calling nitro request : del_ds2 kwargs={'parent_id': '144116287587483648', 'ds_id': '144116290775154688'}
DEBUG - Requesting HTTP post dsDeleteDataSources with data {'receiverId': {'value': '144116287587483648'}, 'datasourceIds': [{'value': '144116290775154688'}]}
WARNING - An HTTP error occured (400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources ErrMsg=NotOk Argument "linux" isn't numeric in numeric ne (!=) at /usr/local/bin/SetThirdPartyConfig line 796.
), retrying api_request()
DEBUG - Requesting HTTP post dsDeleteDataSources with data {'receiverId': {'value': '144116287587483648'}, 'datasourceIds': [{'value': '144116290775154688'}]}
ERROR - Error with method (dsDeleteDataSources) and data : {'receiverId': {'value': '144116287587483648'}, 'datasourceIds': [{'value': '144116290775154688'}]}. From requests.HTTPError 400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources ErrMsg=NotOk Argument "linux" isn't numeric in numeric ne (!=) at /usr/local/bin/SetThirdPartyConfig line 796.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 744, in api_request
File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 744, in api_request
File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "./dstools.py", line 451, in <module>
main()
File "./dstools.py", line 429, in main
ds.delete()
File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/device.py", line 1160, in delete
File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 957, in request
File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 781, in api_request
File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 792, in api_request
msiempy.core.session.NitroError: Error with method (dsDeleteDataSources) and data : {'receiverId': {'value': '144116287587483648'}, 'datasourceIds': [{'value': '144116290775154688'}]}. From requests.HTTPError 400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources ErrMsg=NotOk Argument "linux" isn't numeric in numeric ne (!=) at /usr/local/bin/SetThirdPartyConfig line 796.
tristan@Tristans-MBP dstools % ./dstools.py --list
| name | ds_ip | ds_id | parent_id | client | type_id | last_time |
| Local ESM | 55.69.24.6 | 144115188075855872 | 0 | False | 306 | None |
| ePolicy Orchestrator | 55.69.22.235 | 144117387099111424 | 144115188075855872 | False | 0 | never |
| ePolicy Orchestrator_Endpoint Security Adaptive Threat Protection (ePO) | 55.69.22.235 | 144117387216551936 | 144117387099111424 | False | 583 | never |
| ePolicy Orchestrator_Endpoint Security Firewall (ePO) | 55.69.22.235 | 144117387149443072 | 144117387099111424 | False | 575 | never |
| ePolicy Orchestrator_Endpoint Security Platform (ePO) | 55.69.22.235 | 144117387115888640 | 144117387099111424 | False | 574 | never |
| ePolicy Orchestrator_Endpoint Security Threat Prevention (ePO) | 55.69.22.235 | 144117387182997504 | 144117387099111424 | False | 576 | never |
| ePolicy Orchestrator_Endpoint Security Web Control (ePO) | 55.69.22.235 | 144117387233329152 | 144117387099111424 | False | 577 | never |
| ePolicy Orchestrator_ePO Audit Log (ePO) | 55.69.22.235 | 144117387166220288 | 144117387099111424 | False | 466 | 01/14/2020 01:49:09 |
| ePolicy Orchestrator_ePolicy Orchestrator Agent (ePO) | 55.69.22.235 | 144117387132665856 | 144117387099111424 | False | 360 | never |
| ePolicy Orchestrator_TIE | 55.69.22.235 | 144117387199774720 | 144117387099111424 | False | 524 | never |
| Local Receiver-ELM | 127.0.0.1 | 144116287587483648 | 144115188075855872 | False | 0 | 03/11/2020 11:29:23 |
| CS CEF | 10.0.99.2 | 144116288594116608 | 144116287587483648 | False | 143 | 03/11/2020 11:54:02 |
| Test Datasource | 55.69.24.4 | 144116287604260864 | 144116287587483648 | False | 65 | 09/25/2020 17:11:03 |
| Test DataSource 11 | 10.10.1.41 | 144116290859040768 | 144116287587483648 | False | 65 | never |
| Test DataSource 12 | 10.10.1.42 | 144116290875817984 | 144116287587483648 | False | 65 | never |
| Test DataSource 13 | 10.10.1.43 | 144116290825486336 | 144116287587483648 | False | 65 | never |
Describe
When trying to get Rule_NDSNormSigID.msg, all results break, for example:
Result without asking for Rule_NDSNormSigID.msg field
| LastTime | Rule.msg |
| 10/27/2020 14:10:44 | TCP Scan (horizontal) |
Result asking for Rule_NDSNormSigID.msg
| LastTime | Rule.msg | Rule_NDSNormSigID.msg |
| 10/27/2020 14:10:44 | Unknown_0 | TCP Scan (horizontal) |
Results are mixed, Rule.msg gents wrong info, Rule_NDSNormSigID.msg contains Rule.msg, Rule_NDSNormSigID.msg shoud contain correct normalization name, however it puts that "unknown_0".
SIEM and msiempy versions:
Additional context
Every other query works with every field except Rule_NDSNormSigID.msg
Code Used:
from msiempy import EventManager, FieldFilter
print('Simple event query sorted by AlertID')
events = EventManager(
time_range='PREVIOUS_DAY',
fields=['LastTime','Rule.msg','Rule_NDSNormSigID.msg'],
filters=[
FieldFilter('SrcIP', ['X.X.X.X'], operator='IN'),
],
order=(('ASCENDING', 'AlertID')),
limit=100) # Will only load 100 events (per query)
events.load_data()
print(events)
print(events.get_text(fields=['LastTime','Rule.msg','Rule_NDSNormSigID.msg']))
Describe
After adding anew DataSource withDevTree.add(
), if we call DevTree.search(
) if raises a KeyError
"hostname".
We need to call DevTree.refresh()
before being able to search the tree again.
The expected behaviour is that the add() method would verify if the Data source was indeed added, then refresh only the Datasource infos.
Describe
The Datasource and DevTree codebase have to be rewritten to:
search()
method across list-like objects of the library as described in the 0.3.4 release notesSIEM and msiempy versions:
Additional context
Add any other context about the problem here.
Hi Andy,
Just wondering why you declared all those Watchlist attributes as properties and not simply as key/value of the underlying dict object ? Is there a special reason for this ?
You can alway access underlying dict obejct of any NitroDict class by calling the data
property. example : self.data['name']='the watchlist name'
Thanks
Line 123 in 372ed4e
Describe
The password is not base64 encoded in msiempy\core\session.py
Details:
Password is not base64 encoded in msiempy\core\session.py. Because of this, I was not able to login
to ESM. I don't know if it is supposed to be base64 encoded always, or if there are version that accept it not base-64 encoded.
def login(self, retry=1):
"""
Authentication is done lazily upon the first call to `msiempy.core.session.NitroSession.request` method, but you can still do
it manually by calling this method.
Raises:
`msiempy.core.session.NitroError` if login fails
"""
userb64 = tob64(self.config.user)
passb64 = self.config.passwd
Describe
DataSource objects should allow value modification with ease.
Maybe with a DataSource.edit(field, value)
Additional context
The readme says the library can edit datasources.. but it's untrue.
Describe
When using AlarmManager
, only the first event is compared against events filters.
Line 351 in ad5c754
Expected behaviour would consider the filter matching if any triggering event match all passed filters.
msiempy version:
msiempy : 0.3.3
Describe
The library should cover simple maintenance tasks like
AlarmManager
ESM
object to create a start_full_backup()
method maybe.ESM
objectSIEM and msiempy versions:
Additional context
Ticket open after review of this thread: https://community.mcafee.com/t5/Security-Information-and-Event/Periodic-ESM-Maintenance-Tasks/m-p/672378/highlight/false#
Describe
The documentation regarding Datasources is not 100% clear regarding the field desc_id
:
{"1": "Zone",
"2": "ERC",
"3": "Datasource",
"4": "Database Event Monitor (DBM)",
"5": "DBM Database",
"7": "Policy Auditor",
"10": "Application Data Monitor (ADM)",
"12": "ELM",
"14": "Local ESM",
"15": "Advanced Correlation Engine (ACE)",
"16": "Asset datasource",
"17": "Score-based Correlation",
"19": "McAfee ePolicy Orchestrator (ePO)",
"20": "EPO",
"21": "McAfee Network Security Manager (NSM)",
"22": "McAfee Network Security Platform (NSP)",
"23": "NSP Port",
"24": "McAfee Vulnerability Manager (MVM)",
"25": "Enterprise Log Search (ELS)",
"254": "Client group",
"256": "Client"}
We should rehabilitate the method _insert_desc_names
(which is commented) in DevTree to insert the datasource type as String in a new DataSource field : desc
Hey there, I'm running into this bug where the library will crash if I try to perform an action on an alarm.
Any insight?
import msiempy.alarm
alarms = msiempy.alarm.AlarmManager()
alarms.load_data()
testalarm = alarms[0]
print(testalarm.json)
{
"id": {
"value": 134
},
"severity": 50,
"summary": "Event rate exceeded 1500 by 3983",
"alarmName": "EPS Rate Exceeded",
"conditionType": 14,
"assignee": "[REDACTED]",
"triggeredDate": "10/31/2019 17:55:15",
"acknowledgedDate": "",
"acknowledgedUsername": "",
"filters": null,
"queryId": "0",
"alretRateMin": "5",
"alertRateCount": "7500",
"percentAbove": "0",
"percentBelow": "0",
"offsetMinutes": "0",
"maximumConditionTriggerFrequency": null,
"useWatchlist": null,
"matchField": null,
"matchValue": null,
"assigneeId": "14",
"escalatedDate": null,
"caseId": "135",
"caseName": "EPS rate exceeded (5k)",
"iocName": null,
"iocId": "0",
"description": "Triggers when the system receives more then 10000 events within one minute",
"actions": "5|EPS rate exceeded (5k)|135|[REDACTED]| 1|| ",
"events": [
{}
]
}
testalarm.delete()
WARNING - An HTTP error occured (Cannot deserialize instance of `java.util.ArrayList` out of START_OBJECT token
at [Source: (StringReader); line: 1, column: 1]), retrying request
ERROR -
400 Client Error: 400 for url: https://[REDACTED]/rs/esm/alarmDeleteTriggeredAlarm A JSONObject text must begin with '{' at character 1 of "{\"triggeredIds\": {\"alarmIdList\": [39]}}"
See debug log for more infos.
testalarm.acknowlege()
WARNING - An HTTP error occured (Cannot deserialize instance of `java.util.ArrayList` out of START_OBJECT token
at [Source: (StringReader); line: 1, column: 1]), retrying request
ERROR -
400 Client Error: 400 for url: https://[REDACTED]/rs/esm/alarmAcknowledgeTriggeredAlarm A JSONObject text must begin with '{' at character 1 of "{\"triggeredIds\": {\"alarmIdList\": [134]}}"
2019-10-31 11:06:22,429 - DEBUG - Calling nitro request : delete_alarms kwargs={'ids': [135]}
2019-10-31 11:06:22,430 - DEBUG - Requesting HTTP post alarmDeleteTriggeredAlarm with data {'triggeredIds': {'alarmIdList': [135]}}
2019-10-31 11:06:23,448 - WARNING - An HTTP error occured (Cannot deserialize instance of `java.util.ArrayList` out of START_OBJECT token
at [Source: (StringReader); line: 1, column: 1]), retrying request
2019-10-31 11:06:23,649 - DEBUG - Requesting HTTP post alarmDeleteTriggeredAlarm with data {"triggeredIds": {"alarmIdList": [135]}}
2019-10-31 11:06:24,673 - ERROR -
400 Client Error: 400 for url: https://[REDACTED]/rs/esm/alarmDeleteTriggeredAlarm A JSONObject text must begin with '{' at character 1 of "{\"triggeredIds\": {\"alarmIdList\": [135]}}"
See debug log for more infos.
2019-10-31 11:06:24,673 - DEBUG - 400 Client Error: 400 for url: https://10.60.7.51/rs/esm/alarmDeleteTriggeredAlarm A JSONObject text must begin with '{' at character 1 of "{\"triggeredIds\": {\"alarmIdList\": [135]}}" {'_content': b'A JSONObject text must begin with \'{\' at character 1 of "{\\"triggeredIds\\": {\\"alarmIdList\\": [135]}}"', '_content_consumed': True, '_next': None, 'status_code': 400, 'headers': {'Date': 'Thu, 31 Oct 2019 18:06:24 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=63072000; includeSubdomains; preload', 'Cache-Control': 'no-cache, no-store, must-revalidate', 'Pragma': 'no-cache', 'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'Content-Type': 'text/plain; charset=utf-8', 'Content-Length': '102', 'Content-Security-Policy': "default-src 'none'; script-src 'self' 'unsafe-inline'; object-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self'; form-action 'self'; media-src 'self'; font-src 'self'; connect-src 'self'; plugin-types application/pdf application/x-shockwave-flash; reflected-xss block ; frame-src 'self';frame-ancestors 'self'", 'Connection': 'close'}, 'raw': <urllib3.response.HTTPResponse object at 0x7f240bf801d0>, 'url': 'https://[REDACTED]/rs/esm/alarmDeleteTriggeredAlarm', 'encoding': 'utf-8', 'history': [], 'reason': '400', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(0, 1, 21153), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7f2410e202b0>}
Describe
Random "Invalid username or password for the ESM" errors.
SIEM and msiempy versions:
msiempy verison: 0.2.2
ESM version: 11.2.1
Additional context
Traceback (most recent call last):
File "/home/script-server/isrm-scripts/McAfeeSIEM/Alarms/ack_irrelevants.py", line 59, in <module>
alarms.load_data(pages=args.pages, use_query=True, extra_fields= ['Description', 'Device_URL', 'Alert.DstIP', 'Rule.msg'])
File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 182, in load_data
items, completed = self.qry_load_data(**kwargs)
File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 242, in qry_load_data
page_number=page_number
File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 919, in request
self.login()
File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 650, in login
raise NitroError('Invalid username or password for the ESM')
msiempy.NitroError: Invalid username or password for the ESM
Describe
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 758, in esm_request
result.raise_for_status()
File "/home/script-server/.local/lib/python3.6/site-packages/requests-2.22.0-py3.6.egg/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: 400 for url: https://nitro.mtl.cbc.ca/rs/esm/qryExecuteDetail?type=EVENT&reverse=false
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 758, in esm_request
result.raise_for_status()
File "/home/script-server/.local/lib/python3.6/site-packages/requests-2.22.0-py3.6.egg/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: 400 for url: https://nitro/rs/esm/qryExecuteDetail?type=EVENT&reverse=false
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/script-server/isrm-scripts/McAfeeSIEM/Alarms/ack_irrelevants.py", line 59, in <module>
alarms.load_data(pages=args.pages, use_query=True, extra_fields= ['Description', 'Device_URL', 'Alert.DstIP', 'Rule.msg'])
File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 182, in load_data
items, completed = self.qry_load_data(**kwargs)
File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 267, in qry_load_data
workers=workers)
File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 1394, in perform
func, elements))
File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
yield fs.pop().result()
File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 442, in load_events
the_first_event.data = Event().data_from_id(id=the_id, use_query=use_query, extra_fields=extra_fields)
File "/usr/local/lib/python3.6/site-packages/msiempy/event.py", line 1133, in data_from_id
e.load_data()
File "/usr/local/lib/python3.6/site-packages/msiempy/event.py", line 241, in load_data
items, completed = self.qry_load_data()
File "/usr/local/lib/python3.6/site-packages/msiempy/event.py", line 180, in qry_load_data
includeTotal=False
File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 931, in request
return self.esm_request(method=method, data=data, **params)
File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 777, in esm_request
return self.esm_request(method, data, http, callback, raw, secure, retry=retry-1)
File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 798, in esm_request
raise error
msiempy.NitroError: Error with method (qryExecuteDetail?type=EVENT&reverse=false) and data : {'config': {'timeRange': 'CUSTOM', 'customStart': '2019-07-26T00:00:53.000Z', 'customEnd': '2020-07-26T00:00:53.000Z', 'fields': [{'name': 'LastTime'}, {'name': 'Rule.msg'}, {'name': 'DstIP'}, {'name': 'Description'}, {'name': 'IPSIDAlertID'}, {'name': 'Device_URL'}], 'filters': [{'type': 'EsmFieldFilter', 'field': {'name': 'IPSIDAlertID'}, 'operator': 'IN', 'values': [{'type': 'EsmBasicValue', 'value': '144126183242467328|1333439283'}]}], 'limit': 2, 'offset': 0, 'order': [{'field': {'name': 'LastTime'}, 'direction': 'DESCENDING'}]}}. From requests.HTTPError 400 Client Error: 400 for url: https://url/rs/esm/qryExecuteDetail?type=EVENT&reverse=false ERROR_JobEngine_CannotConnectToJobEngine (3014)
SIEM and msiempy versions:
msiempy verison: 0.2.3
ESM version: 11.2.1 20190725050014
Additional context
Need to add ERROR_JobEngine_CannotConnectToJobEngine
to the qry_load_data()
retry conditions.
Here :
Line 206 in 59b5da7
Describe
NitroSession object should call the "miscKeepAlive" API call every 5mn to minimise the need of (automatically) re-login all the time
msiempy version:
Additional context
This issue was open after the review of the following thread: https://community.mcafee.com/t5/Security-Information-and-Event/ESM-Rest-API-client-connection-session-best-practicies/m-p/671599
Describe
We should be able to handle grouped queries as offed by the API
Additional context
We could access a new object: GroupedEventManager(FilteredQueryList)
GroupedEventManager(field, filters)
Reminder of the API documentation
qryExecuteGrouped
Description
Execute a grouped query giving the count and sum
Parameters
queryType
Type: EsmGroupedQueryType
Description: Query type
Accepted Values:
EVENT
FLOW
config
Type: EsmGroupedQueryConfig
Description: The parameters to apply to the query including filters, time range, field, etc
Return Value ("return" JSON root element IS returned)
Type: EsmRunningQuery
Description: The active query information, created as a result of executing the query successfully.
Example REST Call (with JSON if applicable)
https://URL:4443/rs/esm/qryExecuteGrouped?queryType=EVENT
Example JSON Content:
{"config": {
"filters": [{
"type": "EsmFieldFilter",
"field": {"name": "(name)"},
"operator": "IN",
"values": [{
"type": "EsmWatchlistValue",
"watchlist": {"value": 0}
}]
}],
"field": {"name": "(name)"},
"timeRange": "CUSTOM",
"customStart": "2020-03-11T11:30:12.595Z",
"customEnd": "2020-03-11T11:30:12.596Z"
}}
This issue is from someone sending us emails, please directly create new issue in the future :)
Trying to filter data from a specific normalize ID.
here is the code:
import msiempy.event
events = msiempy.event.EventManager(
time_range='CUSTOM',
start_time='2019-10-16',
end_time='2019-10-31',
fields=['Name','EventCount','Category','UserIDSrc','IPSID'],
filters=[('NormID', ['537919488']) ],
limit=500,
order=('DESCENDING','LastTime'),
max_query_depth=2)
events.load_data(delta='2h', slots='10', workers=1)
print (events.json)
here is the output
Microsoft Windows [Version 10.0.17134.648]
(c) 2018 Microsoft Corporation. All rights reserved.
c:\msiempy>c:\SIEM_Reporter\venv\Scripts\python.exe test.py
INFO - New ESM session instance is created with : 10.28.0.50
Traceback (most recent call last):
File "test.py", line 12, in <module>
events.load_data(delta='2h', slots='10', workers=1)
File "c:\msiempy\msiempy\event.py", line 351, in load_data
sub_query = copy.copy(self)
File "C:\Users\eaguilar.DOMINIO\AppData\Local\Programs\Python\Python37\lib\copy.py", line 88, in copy
return copier(x)
File "C:\Users\eaguilar.DOMINIO\AppData\Local\Programs\Python\Python37\lib\collections\__init__.py", line 1131, in __copy__
inst.__dict__["data"] = self.__dict__["data"][:]
File "C:\Users\eaguilar.DOMINIO\AppData\Local\Programs\Python\Python37\lib\collections\__init__.py", line 1096, in __getitem__
return self.__class__(self.data[i])
File "c:\msiempy\msiempy\event.py", line 111, in __init__
self.fields=list(set(self.fields+fields))
TypeError: unhashable type: 'Event'
Can you help me to fix the issue?
Describe
When trying to call the GroupEventManager API with SIEM nicknames such as with below code an error message throws saying "EVENT Field not supported". Checking the code for event.py on line 566, I see that "self.get_field_nickname(field)" method is being called, shouldn't it be rather different and be trying to get the original internal name for Mcafee instead of the nickname?
Code link:
Line 566 in e5a4f34
Full error message
Error with method (v2/qryExecuteGrouped?queryType=EVENT) and data : {'config': {'filters': [{'type': 'EsmFieldFilter', 'field': {'name': 'IPSID'}, 'operator': 'IN', 'values': [{'type': 'EsmBasicValue', 'value': '144125089401536512'}, {'type': 'EsmBasicValue', 'value': '144125084385148928'}, {'type': 'EsmBasicValue', 'value': '144125089418313728'}, {'type': 'EsmBasicValue', 'value': '144125089435090944'}]}], 'field': {'name': 'New_Value'}, 'timeRange': 'CURRENT_DAY'}}. From requests.HTTPError 400 Client Error: 400 for url: https://SIEM_IP/rs/esm/v2/qryExecuteGrouped?queryType=EVENT Field not supported: New_Value"
Code
query = GroupedEventManager(
time_range='CURRENT_DAY',
field="Alert.4259885",
filters=[
FieldFilter("IPSID", ["144125089401536512","144125084385148928","144125089418313728","144125089435090944"]),#'SrcIP', 'AlertID',
#FieldFilter("Alert.Action", ["11","12"]),
])
query.load_data()
results = list(reversed(sorted(query, key=lambda k: int(k['SUM(Alert.EventCount)']))))
top10=results[:10]
pprint.pprint(top10)
SIEM and msiempy versions:
SIEM and msiempy versions:
msiempy: 0.3.5
ESM version: 11.4.7
Is there an alternative to ipsAddAlertNote in the new v11.2.1 API ?
I'm unable to generate the documentation using Sphinx, and the docs on readthedocs appears to be missing.
Expected result: issuing "make " from ./msiempy/docs/ should generate docs.
Actual result: multiple errors in autodoc:
WARNING: autodoc: failed to import module 'alarm' from module 'msiempy'; the following exception was raised: No module named 'msiempy'
Attempts to resolve the issue include modifying the second parameter in conf.py file's "sys.path.insert" function call to "..", "../msiempy/".
More of a heads up than a current bug but looking at the 10.4 release notes, the notifyGetTriggeredNotification
external API endpoint is being "removed to improve security", among some 77 others.
Looking for another way to get alarm details out of the API surfaces; I had hoped to use qryExecuteDetail
with the TRIGGERED_ALARMS_QUERY
type supplied but the API spits Unsupported query type: TRIGGERED_ALARMS_QUERY
back at me, so go figure.
Line 201 in 85d6ed8
Describe
The whole library uses API V1 and the Private ESM API. Which is not a major issue, in any case the library will depend on private ESM methods.
Calls to ESM API should be edited inside PARAMS property ensuring the parameters stays the same (if they can)
See complete McAfee note about this: https://kc.mcafee.com/corporate/index?page=content&id=KB90289&locale=en_US
Also, not to forget, the session already handles "ESM API V1" changes across different SIEM versions with Session api_v
property (can be 1
or 2
) Not be confused with the ESM API v1 and v2 which are different. (It's confusing, should be changed). See #11 and #27
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.