grunticus03 / elasticflare

CloudFlare Enterprise Log Ingest to Elastic Stack

ElasticFlare

ElasticFlare pulls CloudFlare firewall and audit logs, then ingests and enriches the data using the Elastic Stack.

• Utilizes PowerShell pull down logs.
• Geo-IP enrichment on client and data-center IPs.
• Client user agent parsing
• Identify blocked requests and reason for blocks.
• Email notification when CloudFlare fields are added or removed.

10/6/2020:

• Combined the separate dictionary and log pull scripts into a single PowerShell script.
  • Built out the dictionary building to eliminate the HTTP calls made in the logstash pipeline. This was done to improve event processing speed and to reduce the number of calls to CloudFlare. Dictionary building appears to have had minimal impact on script processing time.
• Added retention flag check.
• General cleanup and scripting improvements

10/14/2019: Significant changes have been made. These changes were designed to provide uniform field naming and improve ease of use. Previous implementations will need to re-ingest all data to unify field names and data types for accurate representation on dashboards and in searches.

I recommend configuring two scheduled tasks in Windows to execute the scripts on a set schedule. See the CloudFlare ELS API documentation for additional support and limitations.

Built on Elastic Stack 7.2.0. Verified functional up to 7.9.2

Feedback and requests for additional features or enrichments is always welcome.

Overview Dashboard

Blocks Dashboard

Cache Dashboard

End User Dashboard

Endpoints & Queries Dashboard

Geo Dashboard

Response Times Dashboard

SSL Dashboard

elasticflare's Issues

Error in the Vega - Sankey - IP/Firewall Blocks dashboard

Hello,

First of all, thank you for this great effort.

I have an issue with the Vega - Sankey - IP/Firewall Blocks visualization. When I open the Blocks dashboard, I get this message at the visualization space:
Infinite extent for field "y1": [Infinity, -Infinity]

Do you have an idea why?

Thank you

CFLogPull.ps1 keeps throwing error during execution even with the right variables defined

Hi ,

Thanks for sharing this on the forum , I was redirected here from the Elastic forum and was interested in trying this out .

I was able to run the CFDictionaries script without any issues and pull the zones from my organization ID. But the other script for pulling CFlogs and audit logs keeps throwing error.

Here is the error I'm getting on PS :

Based on the number of api calls it is making , it looks like it is reaching all my zones under the org ID , but keeps pointing to the Invoke-RestMethod under errors. Is there something I'm missing here ? Appreciate any insight you could give me on this ? Thanks in advance.

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\anthonyp\Downloads\ElasticFlare-master\ElasticFlare\logstash\scripts\CFLogPull.ps1:55 char:10

• $R = Invoke-RestMethod @A "https://api.cloudflare.com/client/v4/z ...
  

•      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
  • FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Not able to find cloudflareauditzones.yaml under logstash foldeer

Hi ,

As i was looking for the same to collect cloudflare logs through logpull API , so am referring yours to implement this, but am not getting your steps here and no documentation how should go through to execute.

If you have a document please send .

As i configured my logstash and which is now throwing error as it couldnt able to find some files

[2020-09-14T15:43:44,180][ERROR][logstash.filters.translate] Invalid setting for translate filter plugin:

filter {
translate {
# This setting must be a path
# File does not exist or cannot be opened /dictionaries/cloudflareauditzones.yaml
dictionary_path => "/dictionaries/cloudflareauditzones.yaml"
...
}
}
[2020-09-14T15:43:44,276][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline

Logstash execute

[2022-03-01T12:17:35,501][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.9.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) Client VM 25.321-b07 on 1.8.0_321-b07 +indy +jit [mswin32-i386]"}
[2022-03-01T12:17:35,673][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-03-01T12:17:40,157][INFO ][org.reflections.Reflections] Reflections took 63 ms to scan 1 urls, producing 22 keys and 45 values
[2022-03-01T12:17:42,547][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create

, action_result: false", :backtrace=>nil}
[2022-03-01T12:17:42,860][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2022-03-01T12:17:47,846][INFO ][logstash.runner ] Logstash shut down.
[2022-03-01T12:17:47,861][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit