integraldefense / o365_log_fetch Goto Github PK

In the debug logs, you can see that the API calls have begun and the time range but you cannot tell which content types are being called (ex: Audit.Exchange, Audit.Sharepoint, etc.)

2018-08-21 17:59:15,402;INFO:root:JobId=8db3c592-58db-4829-a835-ece1cee1caec Begin run. Range: 2018-08-21T17:51:46 to 2018-08-21T17:59:14
2018-08-21 17:59:15,843;INFO:root:JobId=8db3c592-58db-4829-a835-ece1cee1caec Begin run. Range: 2018-08-21T17:51:46 to 2018-08-21T17:59:14
2018-08-21 18:00:08,795;INFO:root:JobId=8db3c592-58db-4829-a835-ece1cee1caec Begin run. Range: 2018-08-21T17:51:46 to 2018-08-21T17:59:14

Sequentially pulling down the content at each Blob location sometimes takes an aggregate longer time than the allotted window to pull down the aggregate logs. This could be remedied by using 'asyncio' to help switch between tasks instead of making a blob location request and then waiting/blocking before moving to the next blob location.

requests is a blocking module, however. So this would have to be replaced with a non-blocking version.

Current documentation leaves much to be desired. Need to have full usage instructions. Add links to Microsoft API information and focus on actual operation/setup of the script.

Error because logging.WARN (which is an int) is passed instead of 'logging.warn'.

in 'o365_api.handler.start_subscription_if_needed'

We get a log when there is no subscription, but there is no indication that the subscription was successfully started other than if the event logs show up.

Example of existing:
2018-08-21 17:51:46,983;WARNING:root:JobId=3efe0f6d-772f-42a3-ae81-5cafb6b886ec No subscription for Audit.Exchange.

Currently, there is no way to continue pulling down available log locations if the results are truncated and you are presented with a pagination header. Add handling of the pagination header to ensure no logs are missed for large log volumes:

https://docs.microsoft.com/en-us/office/office-365-management-api/office-365-management-activity-api-reference#pagination

It would be handy to see how many blobs / endpoints are going to be pulled down for each content type. This could also help troubleshoot Azure permissions issues.

Script was converted to utilize asyncio but does not have appropriate tests.

Currently, it is assumed that the input is in UTC time. The response from O365 is also in UTC. This gets confusing and you could be receiving logs from four or five hours ago if your in the Eastern time zone. It would be beneficial to have the ability to pick your Time Zone via the config file and then the script would convert your input to the appropriate UTC time.

The script exits with the error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0 invalid start byte.

This is on a Ubuntu 18.04.3 system using Python 3.6.8

I have successfully installed the specific requirements from the requirements.txt without error.