ohagana / azure-collector

Alert Logic Azure Collector

Alert Logic Microsoft Office 365 Log Collector

AlertLogic Office 365 Log Collector

Overview

This repository contains the Microsoft Azure web application Node.js source code and an Azure Resource Manager (ARM) template to set up a data collector in Azure, which collects and forwards Microsoft Office 365 log data to Alert Logic Log Management.

Installation

To perform the set up required to grant Alert Logic permission access to collect Office 365 logs, you must have access to the following:

• A Microsoft Office 365 subscription with administrative privileges
• A Microsoft Azure account with administrative privileges
• An Alert Logic user account with administrative privileges

Register a New O365 Web Application in O365

In the Office 365 portal, you must register a new Office 365 web application to collect Office 365 logs.

To register an Office 365 web application to collect logs:

1. Log into the Office 365 portal as an Active Directory tenant administrator.
2. Navigate to Admin Centers and then Azure AD.
3. In the left navigation area, click Azure Active Directory, and then select App Registrations.
4. Click + New application registration and then provide the following information:
   • Name- Provide a name for the new application (For example alo365collector).
   • Select Single tenant option for supported account types.
   • Leave the Redirect URI blank.
5. Click Register
6. From the All applications tab on the App registration (Preview) blade, select All apps, and then click the application name you created.
7. Note the Application ID, for example, a261478c-84fb-42f9-84c2-de050a4babe3

Set Up the Required Active Directory Security Permissions

1. On the main panel under the new application, select View API Permissions, and then click + Add.
2. Locate the Office 365 Management APIs, and then click Select.
3. In Application permissions, expand then select ActivityFeed.Read, ActivityFeed.ReadDlp, ActivityReports.Read(both), ServiceHealth.Read, and ThreatIntelligence.Read(both).
4. Click Select, and then click Done.
5. Click Grant Permissions, and then click Yes. Note: Only the Active Directory tenant administrator can grant permissions to an Azure Active Directory application.
6. On the Settings panel for the application, select Certificated & Secrets.
7. Select the + New client secret button.
8. Type a key Description, and then set Duration to Never expires.
9. Click Add. Note: Save the key value, which you need during ARM template deployment.
10. From the Registered App blade, click the link under Managed application in local directory, and then click Properties.
11. Get the Service Principal ID associated with the application. (The Service Principal IDis labeled as Object ID on the properties page.) Caution: This ID is not the same Object ID found under the Registered app view or under the Settings.

Create an Alert Logic Access Key

Via the UI

We recommend creating an access key via the UI by following These Instructions. However, if you prefer you can use the command line instructions below.

Via the command line

From the Bash command line in Azure Cloud Shell run the following commands, where <username> is your Alert Logic user name and <password> is your Alert Logic password:

export AL_USERNAME='<username>'
auth=$(curl -SX POST -u $AL_USERNAME https://api.global-services.global.alertlogic.com/aims/v1/authenticate); export AL_ACCOUNT_ID=$(echo $auth | jq -r '.authentication.account.id'); export AL_USER_ID=$(echo $auth | jq -r '.authentication.user.id'); export AL_TOKEN=$(echo $auth | jq -r '.authentication.token'); if [ -z $AL_TOKEN ]; then echo "Authentication failure"; else roles=$(curl -SX GET -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/roles | jq -r '.roles[].name'); if [ "$roles" != "Administrator" ]; then echo "The $AL_USERNAME doesn’t have Administrator role. Assigned role is '$roles'"; else curl -SX POST -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys | jq .; fi; fi; unset AL_USERNAME;

For accounts with multi-factor authentication (MFA) enabled:

export AL_USERNAME='<username>'
auth=$(curl -SX POST -d '{"mfa_code": "<mfa_code_here>" }' -u $AL_USERNAME https://api.global-services.global.alertlogic.com/aims/v1/authenticate); export AL_ACCOUNT_ID=$(echo $auth | jq -r '.authentication.account.id'); export AL_USER_ID=$(echo $auth | jq -r '.authentication.user.id'); export AL_TOKEN=$(echo $auth | jq -r '.authentication.token'); if [ -z $AL_TOKEN ]; then echo "Authentication failure"; else roles=$(curl -SX GET -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/roles | jq -r '.roles[].name'); if [ "$roles" != "Administrator" ]; then echo "The $AL_USERNAME doesn’t have Administrator role. Assigned role is '$roles'"; else curl -SX POST -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys | jq .; fi; fi; unset AL_USERNAME;

An example of a successful response is:

{
  "access_key_id": "712c0b413eef41f6",
  "secret_key": "1234567890b3eea8880d292fb31aa96902242a076d3d0e320cc036eb51bf25ad"
}

Note: If the output is blank, verify the Alert Logic user account permissions. You must have administrator access. For more information about AIMS APIs, see Access and Identity Management Service.

Note the access_key_id and the secret_key values for use in the deployment steps below.

Note: An account can create only five access keys. If you receive a "limit exceeded" response, you must delete some keys to create more. Use the following command to list access keys:

curl -s -X GET -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys | jq

Then use the selected access_key_id in the following curl command to delete the key:

curl -X DELETE -H "x-aims-auth-token: $AL_TOKEN" https://api.global-services.global.alertlogic.com/aims/v1/$AL_ACCOUNT_ID/users/$AL_USER_ID/access_keys/<ACCESS_KEY_ID_HERE>

Download and Deploy the ARM Template

You can use either the Microsoft Azure portal or a command line to deploy the template. To perform either procedure, you must log into the Azure portal.

Note: The steps in this section require an active Azure subscription. To verify your Azure subscription, visit Azure subscriptions blade.

Note: This template will deploy a storage account in the active Azure subscription, alongside the function app, in order to successfully collect Office 365 logs.

If your organization uses multiple Active Directory tenants, log into the same tenant used to Register a New Office 365 Web Application. To find your Office 365 tenant ID, see Find your Office 365 tenant ID.

Deploy with the Custom ARM Template in an Azure Subscription

Click the button below to start deployment.

1. Provide the following required template parameters and click the Purchase button to start a deployment:
   • Name - Type the name of the log source to appear in the Alert Logic console.
   • Resource Group - We recommend that you create a new resource group for the collector.
   • Alert Logic Access Key ID - The access_key_id you created above
   • Alert Logic Secret Key - The secret_key you created above.
   • Alert Logic API endpoint - Leave the default value (api.global-services.global.alertlogic.com).
   • Alert Logic Data Residency - Leave the value as default.
   • Office365 Content Streams - The log types you want to collect. Valid values are: ["Audit.AzureActiveDirectory","Audit.Exchange","Audit.SharePoint","Audit.General"]
   • Service Principal ID - The Object ID of the application that created the subscription. Noe You can obtain this value from Azure -> AD -> App registrations -> Your app name -> Link under Managed application in local directory -> Properties -> Object ID.
   • App Client ID - The GUID of your application that created the subscription. Note You can obtain this value from Azure -> AD -> App registrations -> Your app name
   • App Client Secret - The secret key of your application from App Registrations
2. Click Purchase.

Deploy through the Azure CLI

If you want to deploy the template through the Azure command line (CLI), you can use either Azure Cloud Shell or a local installation of Azure CLI.

To deploy through the Azure CLI:

1. In the command line, type the following to create a new resource group. Note The example below creates a new resource group in the "Central US" location.)

   az group create --name <new-resource-group-name> --location "Central US"
   

2. In the Azure portal, access the Resource groups blade, and then select the resource group you created.
3. Select Access Control (IAM), and add Website Contributor role to the Active Directory application identity you created above.
4. In the command line, type the following command to deploy a template, and enter the required parameters when prompted.

   az group deployment create \
       --resource-group <new-resource-group-name> \
       --template-uri "https://raw.githubusercontent.com/alertlogic/azure-collector/master/template.json"
   

Verify the Installation

To verify successful installation of the template:

1. In the Azure portal, access Function Apps, and then choose the Alert Logic Office 365 collector function.
2. Click Functions -> Master -> Monitor and verify the recent log entry has the status of "OK" and contains no error messages. Example: O365 source checkin OK.
3. In the Alert Logic console, navigate to Configuration -> Deployments -> All Deployments -> Log Sources, and then filter the list by Push (Office 365, CloudWatch) collection method.
4. Verify a new Office 365 log source with the name provided during az group deployment create above appears with the source status as OK.

How the Office 365 Collector Works

Note: The following Azure functions use the Application/O365 tenant ID (APP_TENANT_ID web application setting) as a PublisherIdentifier during Office 365 management API requests. For more information about PublisherIdentifier, see Requesting content blobs and throttling.

Master Function

The Master function is a timer trigger function responsible for:

• Registering the Azure web app in Alertlogic backend
• Reporting health-checks to the backed
• Performing log source configuration updates, which happen via Alertlogic UI

Note: When you release a new version of the collector, remember to increment the version number in npm package.json file. To display the current version locally, issue npm run local-version

Updater Function

The Updater function is a timer triggered function that runs a deployment sync operation every 12 hours to keep web application up to date.

O365WebHook Function

The O365WebHook function exposes an HTTP API endpoint https://<app-name>/o365/webhook that is registered as an Office 365 webhook and processes Office 365 activity notifications

Notification Example

[
  {
    "contentType": "Audit.AzureActiveDirectory",
    "contentId": "20170721121608709004422$20170721121608709004422$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
    "contentUri": "https://manage.office.com/api/v1.0/bf8d32d3-1c13-4487-af02-80dba2236485/activity/feed/audit/20170721121608709004422$20170721121608709004422$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
    "notificationStatus": "Succeeded",
    "contentCreated": "2017-07-21T12:16:56.798Z",
    "notificationSent": "2017-07-21T12:16:56.798Z",
    "contentExpiration": "2017-07-28T12:16:08.709Z"
  },
  {
    "contentType": "Audit.AzureActiveDirectory",
    "contentId": "20170721121625590007449$20170721121625590007449$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
    "contentUri": "https://manage.office.com/api/v1.0/bf8d32d3-1c13-4487-af02-80dba2236485/activity/feed/audit/20170721121625590007449$20170721121625590007449$audit_azureactivedirectory$Audit_AzureActiveDirectory$IsFromNotification",
    "notificationStatus": "Succeeded",
    "contentCreated": "2017-07-21T12:16:56.798Z",
    "notificationSent": "2017-07-21T12:16:56.798Z",
    "contentExpiration": "2017-07-28T12:16:25.590Z"
  }
]

A notification contains a link to the actual data, which is retrieved by the O365WebHook, wrapped into a protobuf structure, and then sent to Alert Logic Ingest service.

Note: Audit content may not be available for up to 24 hours. Please follow this link to find the time it takes for the different services in Office 365.

Local Development

1. Clone the repo git clone [email protected]:alertlogic/azure-collector.git.
2. cd azure-collector
3. Run ./local_dev/setup.sh.
4. Edit ./local_dev/dev_config.js.
5. Run the Master function locally: npm run local-master.
6. Run the Updater function locally: npm run local-updater.
7. Run the O365WebHook function locally: npm run local-o365webhook.
8. Run npm test to perform code analysis and unit tests.

Please use the following code style as much as possible.

Setting environment in dev_config.js

• process.env.APP_TENANT_ID - The GUID of the tenant (such as alazurealertlogic.onmicrosoft.com)
• process.evn.APP_RESOURCE_GROUP - The name of the resource group where you deployed your application.
• process.env.CUSTOMCONNSTR_APP_CLIENT_ID - The GUID of your application that created the subscription. Note You can obtain this value from Azure -> AD -> App registrations -> Your app name
• process.env.CUSTOMCONNSTR_APP_CLIENT_SECRET - The secret key of your application from App Registrations.
• process.env.CUSTOMCONNSTR_APP_CI_ACCESS_KEY_ID - The access key returned from AIMs above.
• process.env.CUSTOMCONNSTR_APP_CI_SECRET_KEY- The secret key returned from AIMs above.

Known Issues/ Open Questions

• Sometimes deployments fail after siteSync action. We need better updater to handle that in order not to wait for 12 hours for the next update attempt.

Useful Links

• Node.js static code analysis tool
• How to monitor Azure functions?
• Server application to Web API authentication.
• Office 365 Management API reference.
• Azure Web apps API reference