mfesiem / msiempy Goto Github PK

Typo causes "Interpolation failed probably because of the private API calls formatting... Unexpected behaviours can happend." Error.

Solution
"ds_last_times": ("QRY%5FGETDEVICELASTALERTTIME","""{}"""),
Should be:
"ds_last_times": ("QRY_GETDEVICELASTALERTTIME","""{}"""),

In the interest of creating a more stable live test environment, what are the requirements for a test ESM? It should have alarms, watchlists, events, etc - going how far back? Ideally testing would be limited to shorter timeframes but this requires some amount of intent to ensure those conditions always exist on the ESM for the test. Can we define what needs to exist for tests not to fail?

Describe
Like in the code snippet, the add() method should call dsAddDataSourcesStatus , make sure there is no error or raise an Exception if the Datasource were not added.

Only if special argument check_added=True to keep compatibility.

** msiempy versions:**

• msiempy: 0.3.3

Additional context

                if ds.get('client', None):
                    print("Adding Client Datasource: {}".format(ds))
                    resp = devtree.add_client(ds)
                else: 
                    print("Adding Datasource: {}".format(ds))
                    resp = devtree.add(ds)
                
                if not resp:
                    print('Something went wrong, Datasource {} not added.')
                    continue
                else:
                    # Wait for the add DS query to execuite ...
                    time.sleep(1)
                    ds_status = NitroSession().api_request('dsAddDataSourcesStatus', {"jobId": resp}, retry=0)
                    if not isinstance(ds_status, dict):
                        print('Something went wrong, Datasource {} not added.\n{}'.format(ds['name'], ds_status))
                        continue
                    while not ds_status['jobStatus'] == 'COMPLETE':
                        time.sleep(1)
                        ds_status = NitroSession().api_request('dsAddDataSourcesStatus', {"jobId": resp}, retry=0)
                    if len(ds_status['unsuccessfulDatasources'])>0:
                        print('Something went wrong, Datasource {} not added. {}'.format(ds['name'], ds_status['unsuccessfulDatasources'][0]))
                        continue
                    else:
                        ds_to_verify.append(ds['name'])
                        devtree.refresh()

Describe
When trying to create a datasource object from id, the KeyError pops up.

Workaround: use DevTree:

devtree = DevTree()
ds = list(devtree.search_ds_group(field='ds_id', term='144116290808709120'))
if len(ds): ds=ds[0]
else: print("Datasource not found")

SIEM and msiempy versions:

• msiempy: 0.3.3
• ESM version: 11.3.0

Additional context

>>> from msiempy import DataSource
>>> d=DataSource(id=str(144116290808709120))
INFO - Logged into ESM url with username NGCP. Last login 09/25/2020 03:12:30
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/types.py", line 101, in __init__
    """Call the __prepare__ method of the appropriate metaclass.
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/device.py", line 1134, in data_from_id
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/device.py", line 1198, in _map_parameters
KeyError: 'desc_id'

Shouldn't be IPS_GETFLOWSSNOW ? @andywalden @rhart211

First of all, thank you for your work on this project. There's not a lot of resources out there for this kind of stuff.

I'm running the example event query script on the README.md page, but I run into this msiempy.NitroError: Sorry the filters must be either a tuple(fiels, [values]) or a QueryFilter sub class. error.

Script being used

import msiempy.event
import msiempy.query

filter = msiempy.query.FieldFilter(name='DstIP',values=['10.1.13.0/24'],operator='IN')
customfilter = filter.config_dict

events = msiempy.event.EventManager(
        time_range='LAST_MINUTE',
        filters=[
        msiempy.query.FieldFilter('DstIP', ['8.8.0.0/8',]),
        msiempy.query.FieldFilter('HostID', ['mydomain.local'], operator='CONTAINS') ],
        limit=1,
        max_query_depth=1)
events.load_data(delta='2h', slots='4', workers=5)
print(events.get_text(fields=['Alert.SrcPort', 'Rule.NormID']))

Output

INFO - New ESM session instance is created with : [REDACTED]
Traceback (most recent call last):
  File "event.py", line 13, in <module>
    max_query_depth=1)
  File "/usr/local/lib/python3.6/dist-packages/msiempy/event.py", line 141, in __init__
    super(self.__class__, self.__class__).filters.__set__(self, filters)
  File "/usr/local/lib/python3.6/dist-packages/msiempy/__init__.py", line 1460, in filters
    self.add_filter(f)
  File "/usr/local/lib/python3.6/dist-packages/msiempy/event.py", line 195, in add_filter
    raise NitroError("Sorry the filters must be either a tuple(fiels, [values]) or a QueryFilter sub class.")
msiempy.NitroError: Sorry the filters must be either a tuple(fiels, [values]) or a QueryFilter sub class.

Removing the filters keys seems to work, however.

NitroSession should handle correctly SIEM errors :

Currently, calls to NitroSession.esm_request that raises requests.HTTPError errors can raise NitroError if the error is unhandled.
See docs : https://mfesiem.github.io/docs/msiempy/index.html#msiempy.NitroSession.esm_request

• Unhandled 400 errors :

  • ERROR_InvalidFilter (228)
  • No such command
  • ERROR_Internal (-1)
  • ERROR_JEC_ResponseNotAvailable (3503)
  • Cannot deserialize value of type com.mcafee.siem.api.data.query.EsmTimeRange
  • Cannot deserialize instance of java.lang.String out of START_OBJECT token
  • Both customStart and customEnd properties must be set when running a query with a custom time range.
  • Cannot construct instance of com.mcafee.siem.api.data.alert.EsmAlertId
  • Parameter "xxx" cannot be missing
  • All other errors

• Unhandled 500 errors :

  • All

• Connexion errors -> Raises !

  • Read timed out
  • Max retries exceeded

Describe
The library should handle a easy way to Write Data Sources and Roll Policy.

May be create a DevTree.write_policy() method?
May be also create a DevTree.add(write_policy=True) argument that will call the method? (See #82 for another DevTree.add() improvement)

SIEM and msiempy versions:

• msiempy: 0.3.5

Additional context
The APIs that need to be called are dsWriteThirdpartyConfig, plcyRollPolicy, and miscJobStatus

Describe
Datasource fails to delete.

SIEM and msiempy versions:

• msiempy: 0.3.3
• ESM version: 11.3.0

Additional context

./dstools.py --remove 144116290775154688                   
DEBUG - Calling nitro request : get_devtree kwargs={}
DEBUG - Calling nitro request : login kwargs=***
DEBUG - Requesting HTTP post login ***
DEBUG - Returning raw requests Response object : <Response [201]>
DEBUG - Unpacking SIEM response: {
  "privileges" : {
    "master" : true,
    "admin" : false,
    "power" : false,
    "audit" : false,
    "crypto" : false,
    "systemSettings" : {
      "read" : true,
      "write" : true
    },
DEBUG - Calling nitro request : build_stamp kwargs={}
DEBUG - Requesting HTTP post essmgtGetBuildStamp with data None
DEBUG - Unpacking SIEM response:  { "return":
{
  "buildStamp" : "11.3.0 20191109004423"
} }
DEBUG - <Response [200]> -> Result (<class 'dict'>): {'buildStamp': '11.3.0 20191109004423'}
INFO - Logged into ESM URL with username NGCP. Last login 09/25/2020 17:11:31
DEBUG - Requesting HTTP post GRP_GETVIRTUALGROUPIPSLISTDATA with data {'ITEMS': '#{DC1 + DC2}', 'DID': '1', 'HD': 'F', 'NS': '0'}
DEBUG - Private API call : GRP_GETVIRTUALGROUPIPSLISTDATA Formatted params : Request=API%13GRP_GETVIRTUALGROUPIPSLISTDATA%13%14ITEMS%13#{DC1 + DC2}%13%14DID%131%13%14HD%13F%13%14NS%130%13%14
DEBUG - Unpacking SIEM response: Response=EC%130%13%14AC%130%13%14AF%13F%13%14FC%130%13%14FF%13F%13%14LC%130%13%14LF%13F%13%14DF%130%13%14DNAME%13Physical%20Display%13%14NS%131%13%14ITEMS%1314%11Local%20ESM%11144115188075855872%110%1
DEBUG - <Response [200]> -> Result (<class 'dict'>): {'EC': '0', 'AC': '0', 'AF': 'F', 'FC': '0', 'FF': 'F', 'LC': '0', 'LF': 'F', 'DF': '0', 'DNAME': 'Physical Display', 'NS': '1', 'ITEMS': '14%11Local%20ESM%11144115188075855872%110%11T%11T%11T%11T%11T
DEBUG - Calling nitro request : get_zones_devtree kwargs={}
DEBUG - Requesting HTTP post GRP_GETVIRTUALGROUPIPSLISTDATA with data {'ITEMS': '#{DC1 + DC2}', 'DID': '3', 'HD': 'F', 'NS': '0'}
DEBUG - Private API call : GRP_GETVIRTUALGROUPIPSLISTDATA Formatted params : Request=API%13GRP_GETVIRTUALGROUPIPSLISTDATA%13%14ITEMS%13#{DC1 + DC2}%13%14DID%133%13%14HD%13F%13%14NS%130%13%14
DEBUG - Unpacking SIEM response: Response=EC%130%13%14AC%130%13%14AF%13F%13%14FC%130%13%14FF%13F%13%14LC%130%13%14LF%13F%13%14DF%130%13%14DNAME%13Zone%20Display%13%14NS%131%13%14ITEMS%131%11Undefined%114294967295%11T%1117%111%11%123%
DEBUG - <Response [200]> -> Result (<class 'dict'>): {'EC': '0', 'AC': '0', 'AF': 'F', 'FC': '0', 'FF': 'F', 'LC': '0', 'LF': 'F', 'DF': '0', 'DNAME': 'Zone Display', 'NS': '1', 'ITEMS': '1%11Undefined%114294967295%11T%1117%111%11%123%11CS%20CEF%1114411
DEBUG - Calling nitro request : zonetree kwargs={}
DEBUG - Requesting HTTP post zoneGetZoneTree with data None
DEBUG - Unpacking SIEM response:  { "return":
[ ] }
DEBUG - <Response [200]> -> Result (<class 'list'>): []
DEBUG - Calling nitro request : ds_last_times kwargs={}
DEBUG - Requesting HTTP post QRY_GETDEVICELASTALERTTIME with data {}
DEBUG - Private API call : QRY_GETDEVICELASTALERTTIME Formatted params : Request=API%13QRY_GETDEVICELASTALERTTIME%13%14
DEBUG - Unpacking SIEM response: Response=EC%130%13%14NS%1316%13%14ITEMS%13CS%20CEF%11144116288594116608%11Common%20Event%20Format%1103%2F11%2F2020%2011%3A54%3A02%11%12DC01_DNS%11144116290775154688%11Linux%11%11%12Local%20Receiver-EL
DEBUG - <Response [200]> -> Result (<class 'dict'>): {'EC': '0', 'NS': '16', 'ITEMS': 'CS%20CEF%11144116288594116608%11Common%20Event%20Format%1103%2F11%2F2020%2011%3A54%3A02%11%12DC01_DNS%11144116290775154688%11Linux%11%11%12Local%20Receiver-ELM%111441
Delete the datasource and all the data? 
12, 3, DC01_DNS, 144116290775154688, True, 10.10.1.34, , 65, , Linux, , , , , 0, , 0, False, Local Receiver-ELM, 144116287587483648, never
[y/n]y
DEBUG - Calling nitro request : del_ds2 kwargs={'parent_id': '144116287587483648', 'ds_id': '144116290775154688'}
DEBUG - Requesting HTTP post dsDeleteDataSources with data {'receiverId': {'value': '144116287587483648'}, 'datasourceIds': [{'value': '144116290775154688'}]}
WARNING - An HTTP error occured (400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources ErrMsg=NotOk Argument "linux" isn't numeric in numeric ne (!=) at /usr/local/bin/SetThirdPartyConfig line 796.

), retrying api_request()
DEBUG - Requesting HTTP post dsDeleteDataSources with data {'receiverId': {'value': '144116287587483648'}, 'datasourceIds': [{'value': '144116290775154688'}]}
ERROR - Error with method (dsDeleteDataSources) and data : {'receiverId': {'value': '144116287587483648'}, 'datasourceIds': [{'value': '144116290775154688'}]}. From requests.HTTPError 400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources ErrMsg=NotOk Argument "linux" isn't numeric in numeric ne (!=) at /usr/local/bin/SetThirdPartyConfig line 796.


Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 744, in api_request
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 744, in api_request
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./dstools.py", line 451, in <module>
    main()
  File "./dstools.py", line 429, in main
    ds.delete()
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/device.py", line 1160, in delete
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 957, in request
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 781, in api_request
  File "/usr/local/lib/python3.8/site-packages/msiempy-0.3.3-py3.8.egg/msiempy/core/session.py", line 792, in api_request
msiempy.core.session.NitroError: Error with method (dsDeleteDataSources) and data : {'receiverId': {'value': '144116287587483648'}, 'datasourceIds': [{'value': '144116290775154688'}]}. From requests.HTTPError 400 Client Error: 400 for url: https://URL/rs/esm/dsDeleteDataSources ErrMsg=NotOk Argument "linux" isn't numeric in numeric ne (!=) at /usr/local/bin/SetThirdPartyConfig line 796.


tristan@Tristans-MBP dstools % ./dstools.py --list     
|                                   name                                  |    ds_ip     |       ds_id        |     parent_id      | client | type_id |      last_time      |
|                                Local ESM                                |  55.69.24.6  | 144115188075855872 |         0          | False  |   306   |         None        |
|                           ePolicy Orchestrator                          | 55.69.22.235 | 144117387099111424 | 144115188075855872 | False  |    0    |        never        |
| ePolicy Orchestrator_Endpoint Security Adaptive Threat Protection (ePO) | 55.69.22.235 | 144117387216551936 | 144117387099111424 | False  |   583   |        never        |
|          ePolicy Orchestrator_Endpoint Security Firewall (ePO)          | 55.69.22.235 | 144117387149443072 | 144117387099111424 | False  |   575   |        never        |
|          ePolicy Orchestrator_Endpoint Security Platform (ePO)          | 55.69.22.235 | 144117387115888640 | 144117387099111424 | False  |   574   |        never        |
|      ePolicy Orchestrator_Endpoint Security Threat Prevention (ePO)     | 55.69.22.235 | 144117387182997504 | 144117387099111424 | False  |   576   |        never        |
|         ePolicy Orchestrator_Endpoint Security Web Control (ePO)        | 55.69.22.235 | 144117387233329152 | 144117387099111424 | False  |   577   |        never        |
|                 ePolicy Orchestrator_ePO Audit Log (ePO)                | 55.69.22.235 | 144117387166220288 | 144117387099111424 | False  |   466   | 01/14/2020 01:49:09 |
|          ePolicy Orchestrator_ePolicy Orchestrator Agent (ePO)          | 55.69.22.235 | 144117387132665856 | 144117387099111424 | False  |   360   |        never        |
|                         ePolicy Orchestrator_TIE                        | 55.69.22.235 | 144117387199774720 | 144117387099111424 | False  |   524   |        never        |
|                            Local Receiver-ELM                           |  127.0.0.1   | 144116287587483648 | 144115188075855872 | False  |    0    | 03/11/2020 11:29:23 |
|                                  CS CEF                                 |  10.0.99.2   | 144116288594116608 | 144116287587483648 | False  |   143   | 03/11/2020 11:54:02 |
|                             Test Datasource                             |  55.69.24.4  | 144116287604260864 | 144116287587483648 | False  |    65   | 09/25/2020 17:11:03 |
|                            Test DataSource 11                           |  10.10.1.41  | 144116290859040768 | 144116287587483648 | False  |    65   |        never        |
|                            Test DataSource 12                           |  10.10.1.42  | 144116290875817984 | 144116287587483648 | False  |    65   |        never        |
|                            Test DataSource 13                           |  10.10.1.43  | 144116290825486336 | 144116287587483648 | False  |    65   |        never        |

Describe
When trying to get Rule_NDSNormSigID.msg, all results break, for example:

Result without asking for Rule_NDSNormSigID.msg field

| LastTime | Rule.msg |
| 10/27/2020 14:10:44 | TCP Scan (horizontal) |

Result asking for Rule_NDSNormSigID.msg

| LastTime | Rule.msg | Rule_NDSNormSigID.msg |
| 10/27/2020 14:10:44 | Unknown_0 | TCP Scan (horizontal) |

Results are mixed, Rule.msg gents wrong info, Rule_NDSNormSigID.msg contains Rule.msg, Rule_NDSNormSigID.msg shoud contain correct normalization name, however it puts that "unknown_0".

SIEM and msiempy versions:

• msiempy: 0.3.5
• ESM version: breaks in 11.3.2, works in 10.4

Additional context
Every other query works with every field except Rule_NDSNormSigID.msg

Code Used:

from msiempy import EventManager, FieldFilter
print('Simple event query sorted by AlertID')
events = EventManager(
time_range='PREVIOUS_DAY',
fields=['LastTime','Rule.msg','Rule_NDSNormSigID.msg'],
filters=[
FieldFilter('SrcIP', ['X.X.X.X'], operator='IN'),
],
order=(('ASCENDING', 'AlertID')),
limit=100) # Will only load 100 events (per query)
events.load_data()
print(events)
print(events.get_text(fields=['LastTime','Rule.msg','Rule_NDSNormSigID.msg']))

Describe
After adding anew DataSource withDevTree.add(), if we call DevTree.search() if raises a KeyError "hostname".
We need to call DevTree.refresh() before being able to search the tree again.

The expected behaviour is that the add() method would verify if the Data source was indeed added, then refresh only the Datasource infos.

Describe
The Datasource and DevTree codebase have to be rewritten to:

• Fix #78 and #79
• Unify the search() method across list-like objects of the library as described in the 0.3.4 release notes

SIEM and msiempy versions:

• msiempy: 0.3.4

Additional context
Add any other context about the problem here.

Hi Andy,

Just wondering why you declared all those Watchlist attributes as properties and not simply as key/value of the underlying dict object ? Is there a special reason for this ?

You can alway access underlying dict obejct of any NitroDict class by calling the data property. example : self.data['name']='the watchlist name'

Thanks

Describe
The password is not base64 encoded in msiempy\core\session.py

Details:
Password is not base64 encoded in msiempy\core\session.py. Because of this, I was not able to login
to ESM. I don't know if it is supposed to be base64 encoded always, or if there are version that accept it not base-64 encoded.

    def login(self, retry=1):
    """
    Authentication is done lazily upon the first call to `msiempy.core.session.NitroSession.request` method, but you can still do 
    it manually by calling this method.
    
    Raises:
        `msiempy.core.session.NitroError` if login fails
    """
    userb64 = tob64(self.config.user)
    passb64 = self.config.passwd

Describe
DataSource objects should allow value modification with ease.

Maybe with a DataSource.edit(field, value)

Additional context
The readme says the library can edit datasources.. but it's untrue.

Describe
When using AlarmManager, only the first event is compared against events filters.

Expected behaviour would consider the filter matching if any triggering event match all passed filters.

msiempy version:
msiempy : 0.3.3

Describe
The library should cover simple maintenance tasks like

1. Removing the old triggered alarms:
   OK: This can be done with AlarmManager
2. Removing older reports
   I don't think so.
3. Creating a ESM Settings backup
   I don't think so.
4. Creating a Full Data backup.
   Andy's code should be imported to the ESM object to create a start_full_backup() method maybe.
5. Removing the older Settings back up files:
   I don't think so.
6. Removing the older full back up files locally stored in the ESM
   I don't think so.
7. Performing a dummy write and policy rollout
   Not yet, See #96
8. Running Cron jobs to capture hardware faults, device health status, available disk space in the partitions like checking the index_hd and data_hd available space, swap memory, load average CPU utilization, RAM utilization etc.
   Ok: This can be done with ESM object
9. Performing manual rules update if automatic rules update is not enabled.
   I don't think so.

SIEM and msiempy versions:

• msiempy: 0.3.5

Additional context
Ticket open after review of this thread: https://community.mcafee.com/t5/Security-Information-and-Event/Periodic-ESM-Maintenance-Tasks/m-p/672378/highlight/false#

Describe
The documentation regarding Datasources is not 100% clear regarding the field desc_id :

• The field “desc_id” represent the device type, I understood it can be 13, 3 or '254' for a client but the docs is unclear on the meaning of this. Here is the mapping:

{"1": "Zone",
"2": "ERC",
 "3": "Datasource",
"4": "Database Event Monitor (DBM)",
"5": "DBM Database",
 "7": "Policy Auditor",
"10": "Application Data Monitor (ADM)",
 "12": "ELM",
"14": "Local ESM",
"15": "Advanced Correlation Engine (ACE)",
"16": "Asset datasource",
 "17": "Score-based Correlation",
"19": "McAfee ePolicy Orchestrator (ePO)",
"20": "EPO",
"21": "McAfee Network Security Manager (NSM)",
"22": "McAfee Network Security Platform (NSP)",
 "23": "NSP Port",
 "24": "McAfee Vulnerability Manager (MVM)",
"25": "Enterprise Log Search (ELS)",
"254": "Client group",
 "256": "Client"}

We should rehabilitate the method _insert_desc_names (which is commented) in DevTree to insert the datasource type as String in a new DataSource field : desc

Hey there, I'm running into this bug where the library will crash if I try to perform an action on an alarm.

Any insight?

Load alarm data:

import msiempy.alarm
alarms = msiempy.alarm.AlarmManager()
alarms.load_data()

Test to see if we got valid data back

testalarm = alarms[0]
print(testalarm.json)

{
    "id": {
        "value": 134
    },
    "severity": 50,
    "summary": "Event rate exceeded 1500 by 3983",
    "alarmName": "EPS Rate Exceeded",
    "conditionType": 14,
    "assignee": "[REDACTED]",
    "triggeredDate": "10/31/2019 17:55:15",
    "acknowledgedDate": "",
    "acknowledgedUsername": "",
    "filters": null,
    "queryId": "0",
    "alretRateMin": "5",
    "alertRateCount": "7500",
    "percentAbove": "0",
    "percentBelow": "0",
    "offsetMinutes": "0",
    "maximumConditionTriggerFrequency": null,
    "useWatchlist": null,
    "matchField": null,
    "matchValue": null,
    "assigneeId": "14",
    "escalatedDate": null,
    "caseId": "135",
    "caseName": "EPS rate exceeded (5k)",
    "iocName": null,
    "iocId": "0",
    "description": "Triggers when the system receives more then 10000 events within one minute",
    "actions": "5|EPS rate exceeded (5k)|135|[REDACTED]| 1|| ",
    "events": [
        {}
    ]
}

Then try to perform an action on our selected alarm

testalarm.delete()

WARNING - An HTTP error occured (Cannot deserialize instance of `java.util.ArrayList` out of START_OBJECT token
 at [Source: (StringReader); line: 1, column: 1]), retrying request
ERROR -
400 Client Error: 400 for url: https://[REDACTED]/rs/esm/alarmDeleteTriggeredAlarm A JSONObject text must begin with '{' at character 1 of "{\"triggeredIds\": {\"alarmIdList\": [39]}}"
See debug log for more infos.

It seems like other actions don't work either

testalarm.acknowlege()

WARNING - An HTTP error occured (Cannot deserialize instance of `java.util.ArrayList` out of START_OBJECT token
 at [Source: (StringReader); line: 1, column: 1]), retrying request
ERROR -
400 Client Error: 400 for url: https://[REDACTED]/rs/esm/alarmAcknowledgeTriggeredAlarm A JSONObject text must begin with '{' at character 1 of "{\"triggeredIds\": {\"alarmIdList\": [134]}}"

Here is the debug log

2019-10-31 11:06:22,429 - DEBUG - Calling nitro request : delete_alarms kwargs={'ids': [135]}
2019-10-31 11:06:22,430 - DEBUG - Requesting HTTP post alarmDeleteTriggeredAlarm with data {'triggeredIds': {'alarmIdList': [135]}}
2019-10-31 11:06:23,448 - WARNING - An HTTP error occured (Cannot deserialize instance of `java.util.ArrayList` out of START_OBJECT token
 at [Source: (StringReader); line: 1, column: 1]), retrying request
2019-10-31 11:06:23,649 - DEBUG - Requesting HTTP post alarmDeleteTriggeredAlarm with data {"triggeredIds": {"alarmIdList": [135]}}
2019-10-31 11:06:24,673 - ERROR -
400 Client Error: 400 for url: https://[REDACTED]/rs/esm/alarmDeleteTriggeredAlarm A JSONObject text must begin with '{' at character 1 of "{\"triggeredIds\": {\"alarmIdList\": [135]}}"
See debug log for more infos.
2019-10-31 11:06:24,673 - DEBUG - 400 Client Error: 400 for url: https://10.60.7.51/rs/esm/alarmDeleteTriggeredAlarm A JSONObject text must begin with '{' at character 1 of "{\"triggeredIds\": {\"alarmIdList\": [135]}}" {'_content': b'A JSONObject text must begin with \'{\' at character 1 of "{\\"triggeredIds\\": {\\"alarmIdList\\": [135]}}"', '_content_consumed': True, '_next': None, 'status_code': 400, 'headers': {'Date': 'Thu, 31 Oct 2019 18:06:24 GMT', 'Server': 'Apache', 'X-Frame-Options': 'SAMEORIGIN', 'X-XSS-Protection': '1; mode=block', 'X-Content-Type-Options': 'nosniff', 'Strict-Transport-Security': 'max-age=63072000; includeSubdomains; preload', 'Cache-Control': 'no-cache, no-store, must-revalidate', 'Pragma': 'no-cache', 'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'Content-Type': 'text/plain; charset=utf-8', 'Content-Length': '102', 'Content-Security-Policy': "default-src 'none'; script-src 'self' 'unsafe-inline'; object-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self'; form-action 'self'; media-src 'self'; font-src 'self'; connect-src 'self'; plugin-types application/pdf application/x-shockwave-flash; reflected-xss block ; frame-src 'self';frame-ancestors 'self'", 'Connection': 'close'}, 'raw': <urllib3.response.HTTPResponse object at 0x7f240bf801d0>, 'url': 'https://[REDACTED]/rs/esm/alarmDeleteTriggeredAlarm', 'encoding': 'utf-8', 'history': [], 'reason': '400', 'cookies': <RequestsCookieJar[]>, 'elapsed': datetime.timedelta(0, 1, 21153), 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x7f2410e202b0>}

Describe
Random "Invalid username or password for the ESM" errors.

SIEM and msiempy versions:
msiempy verison: 0.2.2
ESM version: 11.2.1

Additional context

Traceback (most recent call last):
  File "/home/script-server/isrm-scripts/McAfeeSIEM/Alarms/ack_irrelevants.py", line 59, in <module>
    alarms.load_data(pages=args.pages, use_query=True, extra_fields= ['Description', 'Device_URL', 'Alert.DstIP', 'Rule.msg'])
  File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 182, in load_data
    items, completed = self.qry_load_data(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 242, in qry_load_data
    page_number=page_number
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 919, in request
    self.login()
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 650, in login
    raise NitroError('Invalid username or password for the ESM')
msiempy.NitroError: Invalid username or password for the ESM

Describe

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 758, in esm_request
    result.raise_for_status()
  File "/home/script-server/.local/lib/python3.6/site-packages/requests-2.22.0-py3.6.egg/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: 400 for url: https://nitro.mtl.cbc.ca/rs/esm/qryExecuteDetail?type=EVENT&reverse=false

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 758, in esm_request
    result.raise_for_status()
  File "/home/script-server/.local/lib/python3.6/site-packages/requests-2.22.0-py3.6.egg/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: 400 for url: https://nitro/rs/esm/qryExecuteDetail?type=EVENT&reverse=false

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/script-server/isrm-scripts/McAfeeSIEM/Alarms/ack_irrelevants.py", line 59, in <module>
    alarms.load_data(pages=args.pages, use_query=True, extra_fields= ['Description', 'Device_URL', 'Alert.DstIP', 'Rule.msg'])
  File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 182, in load_data
    items, completed = self.qry_load_data(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 267, in qry_load_data
    workers=workers)
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 1394, in perform
    func, elements))
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/msiempy/alarm.py", line 442, in load_events
    the_first_event.data = Event().data_from_id(id=the_id, use_query=use_query, extra_fields=extra_fields)
  File "/usr/local/lib/python3.6/site-packages/msiempy/event.py", line 1133, in data_from_id
    e.load_data()
  File "/usr/local/lib/python3.6/site-packages/msiempy/event.py", line 241, in load_data
    items, completed = self.qry_load_data()
  File "/usr/local/lib/python3.6/site-packages/msiempy/event.py", line 180, in qry_load_data
    includeTotal=False
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 931, in request
    return self.esm_request(method=method, data=data, **params)
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 777, in esm_request
    return self.esm_request(method, data, http, callback, raw, secure, retry=retry-1)
  File "/usr/local/lib/python3.6/site-packages/msiempy/__init__.py", line 798, in esm_request
    raise error
msiempy.NitroError: Error with method (qryExecuteDetail?type=EVENT&reverse=false) and data : {'config': {'timeRange': 'CUSTOM', 'customStart': '2019-07-26T00:00:53.000Z', 'customEnd': '2020-07-26T00:00:53.000Z', 'fields': [{'name': 'LastTime'}, {'name': 'Rule.msg'}, {'name': 'DstIP'}, {'name': 'Description'}, {'name': 'IPSIDAlertID'}, {'name': 'Device_URL'}], 'filters': [{'type': 'EsmFieldFilter', 'field': {'name': 'IPSIDAlertID'}, 'operator': 'IN', 'values': [{'type': 'EsmBasicValue', 'value': '144126183242467328|1333439283'}]}], 'limit': 2, 'offset': 0, 'order': [{'field': {'name': 'LastTime'}, 'direction': 'DESCENDING'}]}}. From requests.HTTPError 400 Client Error: 400 for url: https://url/rs/esm/qryExecuteDetail?type=EVENT&reverse=false ERROR_JobEngine_CannotConnectToJobEngine (3014)

SIEM and msiempy versions:
msiempy verison: 0.2.3
ESM version: 11.2.1 20190725050014

Additional context
Need to add ERROR_JobEngine_CannotConnectToJobEngine to the qry_load_data() retry conditions.
Here :

Currently all NitroList sub-classes have to cast their item in the __init__ method.

For exemple EventManager :

NitroList class should handle dynamically the casting of all of its item :

Describe
NitroSession object should call the "miscKeepAlive" API call every 5mn to minimise the need of (automatically) re-login all the time

msiempy version:

• msiempy: 0.3.5

Additional context
This issue was open after the review of the following thread: https://community.mcafee.com/t5/Security-Information-and-Event/ESM-Rest-API-client-connection-session-best-practicies/m-p/671599

Describe
Test ESM is broken

Describe
We should be able to handle grouped queries as offed by the API

Additional context
We could access a new object: GroupedEventManager(FilteredQueryList)

GroupedEventManager(field, filters)

Reminder of the API documentation

qryExecuteGrouped
Description
Execute a grouped query giving the count and sum

Parameters
queryType
Type: EsmGroupedQueryType
Description: Query type
Accepted Values:
EVENT
FLOW
config
Type: EsmGroupedQueryConfig
Description: The parameters to apply to the query including filters, time range, field, etc
Return Value ("return" JSON root element IS returned)
Type: EsmRunningQuery
Description: The active query information, created as a result of executing the query successfully.
Example REST Call (with JSON if applicable)
https://URL:4443/rs/esm/qryExecuteGrouped?queryType=EVENT

Example JSON Content:

{"config": {
    "filters": [{
        "type": "EsmFieldFilter",
        "field": {"name": "(name)"},
        "operator": "IN",
        "values": [{
            "type": "EsmWatchlistValue",
            "watchlist": {"value": 0}
        }]
    }],
    "field": {"name": "(name)"},
    "timeRange": "CUSTOM",
    "customStart": "2020-03-11T11:30:12.595Z",
    "customEnd": "2020-03-11T11:30:12.596Z"
}}

This issue is from someone sending us emails, please directly create new issue in the future :)
Trying to filter data from a specific normalize ID.

here is the code:

import msiempy.event

events = msiempy.event.EventManager(
        time_range='CUSTOM',
 start_time='2019-10-16',
 end_time='2019-10-31',
        fields=['Name','EventCount','Category','UserIDSrc','IPSID'],
        filters=[('NormID', ['537919488']) ],
        limit=500,
 order=('DESCENDING','LastTime'),
        max_query_depth=2)
events.load_data(delta='2h', slots='10', workers=1)
print (events.json)

here is the output

Microsoft Windows [Version 10.0.17134.648]
(c) 2018 Microsoft Corporation. All rights reserved.

c:\msiempy>c:\SIEM_Reporter\venv\Scripts\python.exe test.py
INFO - New ESM session instance is created with : 10.28.0.50
Traceback (most recent call last):
  File "test.py", line 12, in <module>
    events.load_data(delta='2h', slots='10', workers=1)
  File "c:\msiempy\msiempy\event.py", line 351, in load_data
    sub_query = copy.copy(self)
  File "C:\Users\eaguilar.DOMINIO\AppData\Local\Programs\Python\Python37\lib\copy.py", line 88, in copy
    return copier(x)
  File "C:\Users\eaguilar.DOMINIO\AppData\Local\Programs\Python\Python37\lib\collections\__init__.py", line 1131, in __copy__
    inst.__dict__["data"] = self.__dict__["data"][:]
  File "C:\Users\eaguilar.DOMINIO\AppData\Local\Programs\Python\Python37\lib\collections\__init__.py", line 1096, in __getitem__
    return self.__class__(self.data[i])
  File "c:\msiempy\msiempy\event.py", line 111, in __init__
    self.fields=list(set(self.fields+fields))
TypeError: unhashable type: 'Event'

Can you help me to fix the issue?

Describe
When trying to call the GroupEventManager API with SIEM nicknames such as with below code an error message throws saying "EVENT Field not supported". Checking the code for event.py on line 566, I see that "self.get_field_nickname(field)" method is being called, shouldn't it be rather different and be trying to get the original internal name for Mcafee instead of the nickname?

Code link:

Full error message
Error with method (v2/qryExecuteGrouped?queryType=EVENT) and data : {'config': {'filters': [{'type': 'EsmFieldFilter', 'field': {'name': 'IPSID'}, 'operator': 'IN', 'values': [{'type': 'EsmBasicValue', 'value': '144125089401536512'}, {'type': 'EsmBasicValue', 'value': '144125084385148928'}, {'type': 'EsmBasicValue', 'value': '144125089418313728'}, {'type': 'EsmBasicValue', 'value': '144125089435090944'}]}], 'field': {'name': 'New_Value'}, 'timeRange': 'CURRENT_DAY'}}. From requests.HTTPError 400 Client Error: 400 for url: https://SIEM_IP/rs/esm/v2/qryExecuteGrouped?queryType=EVENT Field not supported: New_Value"

Code
query = GroupedEventManager(
time_range='CURRENT_DAY',
field="Alert.4259885",
filters=[
FieldFilter("IPSID", ["144125089401536512","144125084385148928","144125089418313728","144125089435090944"]),#'SrcIP', 'AlertID',
#FieldFilter("Alert.Action", ["11","12"]),
])
query.load_data()
results = list(reversed(sorted(query, key=lambda k: int(k['SUM(Alert.EventCount)']))))
top10=results[:10]
pprint.pprint(top10)

SIEM and msiempy versions:
SIEM and msiempy versions:

msiempy: 0.3.5
ESM version: 11.4.7

Is there an alternative to ipsAddAlertNote in the new v11.2.1 API ?

I'm unable to generate the documentation using Sphinx, and the docs on readthedocs appears to be missing.

Expected result: issuing "make " from ./msiempy/docs/ should generate docs.

Actual result: multiple errors in autodoc:
WARNING: autodoc: failed to import module 'alarm' from module 'msiempy'; the following exception was raised: No module named 'msiempy'

Attempts to resolve the issue include modifying the second parameter in conf.py file's "sys.path.insert" function call to "..", "../msiempy/".

More of a heads up than a current bug but looking at the 10.4 release notes, the notifyGetTriggeredNotification external API endpoint is being "removed to improve security", among some 77 others.

Looking for another way to get alarm details out of the API surfaces; I had hoped to use qryExecuteDetail with the TRIGGERED_ALARMS_QUERY type supplied but the API spits Unsupported query type: TRIGGERED_ALARMS_QUERY back at me, so go figure.

Describe
The whole library uses API V1 and the Private ESM API. Which is not a major issue, in any case the library will depend on private ESM methods.

Calls to ESM API should be edited inside PARAMS property ensuring the parameters stays the same (if they can)

See complete McAfee note about this: https://kc.mcafee.com/corporate/index?page=content&id=KB90289&locale=en_US

Also, not to forget, the session already handles "ESM API V1" changes across different SIEM versions with Session api_v property (can be 1 or 2) Not be confused with the ESM API v1 and v2 which are different. (It's confusing, should be changed). See #11 and #27