chrivand / firepower_o365_feed_parser

This is a Sample Script that can parse the O365 Web Service API and upload it to Firepower Management Center as Group Objects.

O365 Web Service API to Firepower Objects Parser [v4.1.2]

Note: you may now also use this SecureX orchestration workflow!

Note: please also check out if the new Dynamic Object Feed feature might solve your use case.

Note: there are reports that the script is incompatible with Python 3.12, please use 3.11. Feel free to do a Pull Request with a compatible version!

Note: for questions or remarks, please use the Cisco DevNet Community for Network Sceurity. The original author, Christopher Van Der Made ([email protected]), can also be contacted on a best-effort basis (different role in Cisco now). If you find any bugs or have enhancements, please create an issue in this GitHub repository or do a pull request.

This is a sample script that parses the NEW O365 Web Service API that Microsoft publishes with URL, IPv4 and IPv6 addresses. These addresses are used for the infrastructure of the Microsoft cloud applications (e.g., Office 365). The script will parse the NEW O365 Web Service API into 2 separate lists and use the FMC API to upload them into 2 Group Objects. These Group Objects can be used in a Firepower trust/prefilter rule. By doing so the traffic is excluded from further inspection, to prevent latency issues with the Microsoft O365 applications.

If you would like to see a demo of the script, please check out the video below:

https://youtu.be/nY9nWVrgO4I

Release notes v4.1.2

• VERSION 4.2 in BETA now, with ASA support: https://github.com/chrivand/Firepower_O365_Feed_Parser/tree/VERSION-4.2
• Minor (v4.1.2) update from version v4.0 that creates 4 group objects in instead of 2.
• Minor (v4.1.2) update from version v4.0 that adds proxy support. User is prompted if proxy is needed, and if authentication for the proxy is needed. If so, user will be prompted.
• Minor (v4.1.2) update from version v4.0 that fixed a problem with the Common Service Area. This was an important bug, and is now fixed.
• It now creates 2 URL group objects for Optimize+Allow and for Default category, and does the same for the IP group objects.
• [WARNING] It adds a dummy IP range (240.0.0.0/4) or dummy URL (example.com), if Microsoft does not return any addresses. Microsoft does not always return full lists. Sometimes they only return the Default URL’s for example, since they don’t own those IP ranges. This can cause a Policy Deploy failure, hence the dummy addresses.
• Some other changes include the optimization of the script (e.g. it will only do FMC requests if a new version is detected of the O365 script).
• Executing of the script is exactly the same as before, it now just creates 4 objects. The BYPASS objects are recommended to be bypassed (like in a prefilter rule), the DEFAULT objects are recommended to be treated just like other internet traffic.

Features

• Retrieving Wordlwide URLs and IPs from new O365 REST-based web service;
• Parsing these into 4 flat lists (URL and IP (mixed IPv4 and IPv6) for Optimize+Allow and Default category);
• Creating right JSON format for FMC API PUT requests;
• Uploading this JSON to FMC, overwriting the previous Group Object;
• If no objects have been created, 4 overridable objects will be created: 'O365_Web_Service_IPs' and 'O365_Web_Service_URLs';
• Checking if O365 file was updated, using the O365 Version API Endpoint;
• Automatic policy deploy using API when changes were made to Objects (optional, caution this will also deploy other, unrelated policy changes);
• Webex Teams alert when changes were made to Objects;
• Continuously checking for updates with a specified time interval (optional).

Potential next steps

• Create extra modules for other SaaS applications;
• Create extra modules for other Cisco solutions (WSA, Umbrella, etc.).

Solution Components

The script consists of 3 python files. The main script can run indefinitely, leveraging a function that is built in, to rerun the script every x amount of seconds (it can also just be executed once). Then, using the Version API Endpoint, the script checks if changes were made to the Web Service list. If changes were made, the Web Service list is parsed and uploaded using a PUT request to FMC. Microsoft updates the Office 365 IP address and FQDN entries at the end of each month and occasionally out of cycle for operational or support requirements. Therefore, Microsoft recommends you check the version daily, or at the most, hourly. This can be automated with the script.

Cisco Products / Services

• Cisco Firepower Management Center;
• Cisco Firepower Threat Defense NGFW.

Installation

These instructions will enable you to download the script and run it, so that the output can be used in Firepower as Group Objects. What do you need to get started? Please find a list of tasks below:

1. You need the IP address (or domain name) of the FMC, the username and password. These will be requested by the script the first time it is run. It is recommended to create a separate FMC login account for API usage, otherwise the admin will be logged out during every API calls. Add the IP/Domain of FMC, the username and password to the config.json file. If you do not add anything, you will be prompted to fill this in when executing the script.

2. The script will also prompt you which Service Areas (Exchange, SharePoint, Skype) you are using and which O365 plan you are using (Worldwide, Germany, USGovDoD, USGovGCCHigh, China). After this you will be prompted if you are using a proxy to go to the internet to make the API requests to Microsoft.

Note: Potentially you can run this script multiple times to create separate objects per Service Area (for example if a set of your end-users use SharePoint, but everyone uses Exchange). Please make sure to create a separate directory with it's own version of the config.json file.

3. In the FMC, go to System > Configuration > REST API Preferences to make sure that the REST API is enabled on the FMC.

4. A Network Group object and a URL Group object will be created automatically during the first run of the script. However, if you'd rather create the objects manually, you can follow the instructions below.

5. It is also recommended to download an SSL certificate from FMC and put it in the same folder as the scripts. This will be used to securely connect to FMC. In the config.json file, set the "SSL_VERIFY" parameter to true, and then set "SSL_CERT" to be the path to the FMC's certificate.

6. It is possible to integrate the script with Webex Teams. In order to do that, an API Access Token and a Room ID need to be entered in the config.json file. Please retrieve your key from: https://developer.webex.com/docs/api/getting-started. Then create a dedicated Webex Teams space for these notifications and retrieve the Room ID from: https://developer.webex.com/docs/api/v1/rooms/list-rooms. Please be aware that the personal token from the getting started page only works for 12 hours. Please follow these steps to request a token per request: https://developer.webex.com/docs/integrations. This is roadmapped for v5.0 of the script.

7. In this same Webex Teams room you can subscribe to an RSS feed from Microsoft regarding updates. Use this bot to integrate the RSS feed into your Webex space: https://apphub.webex.com/bots/rss-2739. The RSS feed URL can be found on the Microsoft website (link on top of this page).

8. If you do not have the needed Python libraries set up, you will get an error when executing the script. You will need to install the "requirements.txt" file like this (make sure you are in the same directory as the cloned files live):

pip install -r requirements.txt

9. After this is complete you need to execute the script (make sure you are in the same directory as the cloned files live). When you execute the Python script, you will be prompted for information if you did not fill this in step 1.

python3.6 O365WebServiceParser.py

10. Optionally you can let this script run periodically, by setting "SERVICE" to true in the config.json file. In line 374 of the O365WebServiceParser.py the time-period is set, per default it is set to an hour (Microsoft recommends you check the version daily, or at the most, hourly):

intervalScheduler(WebServiceParser, 3600) #set to 1 hour

11. Finally, if you want to automatically deploy the policies, you can set "AUTO_DEPLOY" to true in the config.json file. Be very careful with this, as unfinished policies might be deployed by doing so.

Manual Object Creation (optional, not recommended!!!)

1. Create 2 Group Objects in FMC: "O365_Web_Service_URLs" (URL Group Object) and "O365_Web_Service_IPs" (Network Group Object). At first you will have to put in a random URL/Network to create the group objects. No worries, we will override this later.

2. Use the FMC API Explorer to do a GET request for the Network Group Objects. This is done by going into the FMC API Explorer (can be reached at https://IP-addressOfFMC/api/api-explorer), and then clicking on "Object" in the left menu. The scroll down to "networkgroups" and click on "GET" and then again on "GET" in the right menu.

3. Now you will need to copy-paste the Object IDs of the Network Group Object ("O365_Web_Service_IPs"). The IDs will look like the following format: "000XXXX-YYYY-ZZZZ-0000-01234567890". This is displayed in the "Response Text" output box in the right menu. You will need these later in the PUT requests to update the objects. Below is an example of how this output would look for the "O365_Web_Service_IPs" Network Group Object:

"type": "NetworkGroup",
"name": "O365_Web_Service_IPs",
"id": "000XXXX-YYYY-ZZZZ-0000-01234567890"

4. Repeat the GET request of step 2 as well for "urlgroups", to obtain the ID for the URL Group Object ("O365_Web_Service_URLs"). You should now have two IDs copy-pasted, which you can put inside the config.json file as "IP_UUID" and "URL_UUID" to configure the script.

How to use the Group Objects in Firepower Management Center.

For better understanding of the packet flow in Firepower Threat Defense, and how the Fastpath action in the Prefilter Policy works, please review the following flow diagram:

After the successful PUT requests, the 2 Group Objects will have been updated with the new IP-addresses and URLs. Please find screenshots of the 2 Group Objects, after the API call:

These objects can be used in either Prefilter Policy Fastpath-rule (for the Network Object), or in an Access Control Policy Trust-rule (for the URL Object). This is an example of how to configure the Prefilter Policy rule in FMC:

This will result in the following rule:

Likewise, this can be done with a Trust Rule in the Access Control Policy for the URL Group Object:

As a final step you will need to do a Policy Deploy, each time that the Group Objects have been updated. This can be done from the FMC by clicking on "DEPLOY" and by selecting the device that need this Policy Deploy.

Using the Objects in AnyConnect Profiles for Split Tunneling (not tested yet):

I have not tested this yet but it is possible to use the created Network Group Objects for exclusion in your VPN tunnel. This can be done by creating or editing an AnyConnect Profile like below:

You can then add the Network Group Object as an ACL to be excluded from the VPN tunnel (split tunneling):

Please test this thorougly before using in production!

Please take caution on the following notes:

• Please be aware that a policy redeploy is needed to update the Group Objects in the used Policies. Currently there is an optional API call built in to do a policy redeploy, however please take caution in using this, since this might cause other, unrelated policies or objects to be deployed (e.g., if another network administrator is working on a Policy in the GUI).

• Important is to use SSL verification and to test the script before running this in a production environment. In the config.json file, set the "SSL_VERIFY" parameter to true, and then set "SSL_CERT" to be the path to the FMC's certificate.

• Please test this properly before implementing in a production environment. This is a sample script.

• In case the intervalScheduler is used: the running script should be hosted in a secure environment! For example: if a malicious actor can place additional IP-addresses or URL's in the list somehow, they will be put in a Firepower trust rule, and might cause the malicious actor to bypass security.

Author(s)

• Christopher van der Made (Cisco)
• Alan Nix (Cisco)

firepower_o365_feed_parser's Issues

Error Sending Commands to ASA

Hello,

first of all, I am really thankful for all the work you put into this project. So after preparing my linux system to run your script, I run into an error when the script tries to send the command to the asa. Here are the script output.

Chosen O365 plan: Germany, chosen applications: Exchange,SharePoint
Saving config data...

First time script runs, version of Office 365 Germany commercial service instance endpoints detected: 2020070800
Saving config data...

Connecting to ASA...
ASA Successfully Connected.

IP_DEFAULT list returned no IP's, empty list with dummy IP range (240.0.0.0/4) created (to avoid policy deploy failure)...

Sending show running-config | include object-group network O365_Service_IP_Adresses request to ASA.
The network-object object exists, removing before re-creating

Sending no object-group network O365_Service_IP_Adresses request to ASA.

Sending request to ASA:
['object-group network O365_Service_IP_Adresses', 'network-object 51.4.64.0 255.255.254.0', 'network-object 51.5.64.0 255.255.254.0', 'network-object 51.4.72.0 255.255.255.0', 'network-object 51.4.80.0 255.255.255.224', 'network-object 51.5.72.0 255.255.255.0', 'network-object 51.5.80.0 255.255.255.224', 'network-object 51.4.66.0 255.255.254.0', 'network-object 51.5.66.0 255.255.254.0', 'network-object host 51.4.144.200', 'network-object host 51.5.149.3', 'network-object 51.18.16.0 255.255.254.0', 'network-object host 51.4.2.10', 'network-object host 51.4.71.61', 'network-object 51.4.136.38 255.255.255.254', 'network-object 51.4.136.40 255.255.255.254', 'network-object host 51.4.136.42', 'network-object host 51.4.146.38', 'network-object host 51.4.146.206', 'network-object host 51.5.16.7', 'network-object host 51.5.71.22', 'network-object 51.5.136.32 255.255.255.252', 'network-object host 51.5.136.36', 'network-object host 51.5.145.29', 'network-object host 51.5.145.122', 'network-object 51.4.70.0 255.255.255.0', 'network-object 51.4.71.0 255.255.255.0', 'network-object host 51.4.226.115', 'network-object host 51.4.227.178', 'network-object host 51.4.230.178', 'network-object 51.5.70.0 255.255.255.0', 'network-object 51.5.71.0 255.255.255.0', 'network-object host 51.5.147.48', 'network-object host 51.5.242.163', 'network-object host 51.5.245.67', 'network-object host 51.4.144.41', 'network-object host 51.4.144.174', 'network-object host 51.4.145.38', 'network-object host 51.4.147.81', 'network-object host 51.4.147.233', 'network-object host 51.4.148.12', 'network-object host 51.4.150.145', 'network-object host 51.5.147.242', 'network-object host 51.5.149.100', 'network-object host 51.5.149.119', 'network-object host 51.5.149.123', 'network-object host 51.5.149.180', 'network-object host 51.5.149.186', 'network-object 51.18.0.0 255.255.248.0']
Error sending commands to ASA: Timed-out reading channel, data not available.

Exiting...

Any hints or tipps how to solve this issue myself?

Best regards
Christian

Split tunneling doesn't like IPv6 addresses

This is a great script, I appreciate you writing it. We're using it so that we can split tunnel our VPN users to go direct to O365. The only issue we encountered is that the Firepower doesn't like IPv6 addresses in the split tunnel ACL. Once we removed them from the network object your script created the policy would deploy successfully and split tunneling worked like a champ. This is on a 2130 running 6.3.0.5 code.

Wildcard (*) in URL List

Good afternoon!

My understanding was that the FMC didn't process wildcards, but utilized substring matching instead.

Has that changed?

The parser worked beautifully, and populated the object groups, but there's a number of *.domain listings.

Thank you!

Can you please add support for Python3.6?

I am getting a lot of dependencies errors and stuck with.
S@mate:~/scripts/Firepower_O365_Feed_Parser-asa_support# python3.6 O365_web_service_parser.py
Traceback (most recent call last):
File "O365_web_service_parser.py", line 24, in
import webexteamssdk
ModuleNotFoundError: No module named 'webexteamssdk'

tried using pip install webexteamssdk
$: command not found

then got installed
pip3 install webexteamssdk
Collecting webexteamssdk

again run the script
S@mate:~/scripts/Firepower_O365_Feed_Parser-asa_support# python3.6 O365_web_service_parser.py
Traceback (most recent call last):
File "O365_web_service_parser.py", line 29, in
from Asa import Asa
File "/root/scripts/Firepower_O365_Feed_Parser-asa_support/Asa.py", line 16, in
from netmiko import ConnectHandler
ModuleNotFoundError: No module named 'netmiko'

installed using
pip3 install -U netmiko

again run the script and got the following error

S@mate:~/scripts/Firepower_O365_Feed_Parser-asa_support# python3.6 O365_web_service_parser.py
Traceback (most recent call last):
File "O365_web_service_parser.py", line 29, in
from Asa import Asa
File "/root/scripts/Firepower_O365_Feed_Parser-asa_support/Asa.py", line 16, in
from netmiko import ConnectHandler
ModuleNotFoundError: No module named 'netmiko'

I run these commands and let worked like it should be.
pip install --upgrade pip
pip3 install --upgrade pip
pip3 install netmiko

then got it started work.

S@mate:~/scripts/Firepower_O365_Feed_Parser-asa_support# python3.6 O365_web_service_parser.py

Loading config data...
Config loading complete.

TESTTEST
Do you want to update an FMC [f] or ASA [a] [f/a]:

Thanks for the script, it helped me a lot.

Extended Access List Object Instruction not working

Hi,
thanks for the Great Script.
Just one Thing:
In your Screenshot of the "New Extended Access List Object" you put the Network into Destination, in the Previous Screenshot CISCO States "Configure the split tunnel networks in the source" of the Extended ACL, destination networks are ignored. So it needs to be put into Source, then it works fine.

Proxy not working

Hi there,

Thank you for writing the program; it is extremely useful to us.

We have started using a proxy and have noticed the program continues to send requests without going through a proxy first.

We see this whilst running tcpdump during program execution.

config.json settings:
"PROXY": "true",
"PROXY_USER": "",
"PROXY_PASSWD": "",
"PROXY_HOST": "...",
"PROXY_PORT": "8080"

The program hangs on "Saving config data..."
python3 O365_web_service_parser.py

Loading config data...
Config loading complete.

Chosen O365 plan: Worldwide, chosen applications: All
Saving config data...

If you have the time to look at it, it would be appreciated.

Many thanks

WebServiceParser -- URL_default_list empty check

Basically in the WebServiceParser the goal is to check both arrarys URL_List and URL_default_list for being empty. However, at line 478 the wrong variable list is used. URL_List is being checked instead of URL_default_list.

Emailed files excluded filtering?

Does Firepower_O365_Feed_Parser cause emailed files to be excluded from malware filtering?
Seems this might be the case.

URL default list references wrong var

Firepower_O365_Feed_Parser/O365_web_service_parser.py

Line 450 in 3a9ea6a

if not URL_List:

Firepower_O365_Feed_Parser/O365_web_service_parser.py

Line 451 in 3a9ea6a

URL_List.append("example.com")

Should be URL_default_list not URL_List

IP Default Category

I noticed in the code it is creating a default IP category object group. This step is really not necessary. Those items with the default category will never have IP addresses.
From Microsoft
https://learn.microsoft.com/en-us/microsoft-365/enterprise/microsoft-365-network-connectivity-principles?view=o365-worldwide#optimizing-connectivity-to-microsoft-365-services

Network endpoints classified under the “Default” category do not have IP addresses associated with them as they are more dynamic in nature and IP addresses change over time.

UnboundLocalError: local variable 'newVersion' referenced before assignment

Python Version 3.10.12:

Chosen O365 plan: , chosen applications: All
Saving config data...
Traceback (most recent call last):
File "/home/azureuser/Firepower_O365_Feed_Parser/O365_web_service_parser.py", line 597, in
WebServiceParser()
File "/home/azureuser/Firepower_O365_Feed_Parser/O365_web_service_parser.py", line 249, in WebServiceParser
bool_new_version = check_for_new_version(clientRequestId)
File "/home/azureuser/Firepower_O365_Feed_Parser/O365_web_service_parser.py", line 217, in check_for_new_version
sys.stdout.write(f"\nFirst time script runs, version of Office 365 {CONFIG_D ATA['O365_PLAN']} commercial service instance endpoints detected: {newVersion}\n ")
UnboundLocalError: local variable 'newVersion' referenced before assignment

IPv4 ONLY Option?

I thought I remember at one time in the beginning there was an option to only grab / import IPv4 IP addresses. At this time, I don't use IPv6 IP addresses. So, I updated the script to prompt from IPV4 only and proceed based on that information in the config file.

def isIPV4Only():
return CONFIG_DATA['IPV4_ONLY'] != ''

In the web parser fucntion

                    # if the IP hasn't already been appended, then append it
                    if ip not in IP_List:
                        if CONFIG_DATA['IPV4_ONLY']:
                            # check if IP is IPv4
                            if ipaddress.ip_network(ip).version == 4:
                                IP_List.append(ip)
                        elif not CONFIG_DATA['IPV4_ONLY']:
                            IP_List.append(ip)
            # make sure IPs exist in the item
            elif 'ips' in item and item['category'] == 'Default':

                # iterate through all IPs in each item
                for ip in item['ips']:
                    # if the IP hasn't already been appended, then append it
                    if ip not in IP_List:
                        if CONFIG_DATA['IPV4_ONLY']:
                            #check if IP is IPv4
                            if ipaddress.ip_network(ip).version == 4:
                                IP_default_list.append(ip)
                        elif not CONFIG_DATA['IPV4_ONLY']:
                            IP_default_list.append(ip)

In the START EXECUTION SCRIPT section

if not isIPV4Only():
print("IPV4_ONLY")
deviceAnswer=input("Set IPV4 Results Only YES [y] or NO [n]: ").lower().strip()[:1]
if deviceAnswer=="y":
CONFIG_DATA['IPV4_ONLY'] = True
elif deviceAnswer=="n":
CONFIG_DATA['IPV4_ONLY'] = False
else:
print("Invalid answer.")
sys.exit(1)

Empty Config String Check Issues in O365WebServerParser.py

There are multiple empty config option string checks in O365WebServerParser.py that do not implement the check properly, e.g.:

if CONFIG_DATA['WEBEX_ACCESS_TOKEN'] is '' or CONFIG_DATA['WEBEX_ROOM_ID'] is '':

With respect to this example, if neither of these two options are defined in config, the if statement will erroneously proceed to the "else" clause. In this case, in the "else" clause the script will attempt to use WebEx when no WebEx config options have been defined, causing the script to throw errors.

Example fix:

if CONFIG_DATA['WEBEX_ACCESS_TOKEN'] == '' or CONFIG_DATA['WEBEX_ROOM_ID'] == '':