Coder Social home page Coder Social logo

carbonblack / cb-taxii-connector Goto Github PK

View Code? Open in Web Editor NEW
14.0 15.0 8.0 744 KB

Connector for pulling and converting STIX information from TAXII Service Providers into CB Feeds.

License: MIT License

Shell 3.95% Python 89.92% Kotlin 2.68% Dockerfile 0.66% HTML 2.80%

cb-taxii-connector's Introduction

VMware Carbon Black - STIX/TAXII 2 Connector (CentOS 6/7/8)

VMware Carbon Black EDR provides integration with STIX/TAXII version 2.0/2.1 servers.

To support this integration, Carbon Black provides an out-of-band bridge that communicates with the TAXII API. Built with python3!

The integration can be configured to retrieve STIX Indicators from a number of specified TAXII 2.0/2.1 servers. The integration will query the configured servers for SIX indicators, and then translate STIX-pattern indicators into EDR IOC format where possible to the produced a consolidated EDR threat intelligence feed. MD5/Sha256 hashes, IP addresses and domain names included in the available STIX Indicators patterns will be included, other indicators will be ignored.

Installation Quickstart

As root on your EDR or other RPM based 64-bit Linux distribution server:

cd /etc/yum.repos.d
curl -O https://opensource.carbonblack.com/release/x86_64/CbOpenSource.repo
yum install python-cb-taxii-connector

Once the software is installed via YUM, copy the /etc/cb/integrations/taxii/connector.conf.example file to /etc/cb/integrations/taxii/connector.conf. Edit this file and place your EDR API key into the carbonblack_server_token variable and your EDR server's base URL into the carbonblack_server_url variable.

Define a new section in the ini file for each Taxii server you wish to download STIX Indicators. url= is required, and must be set to the protocol prefixed url of the server version= is optional, and controls the TAXII version of the target server (v20 or v21) score= can be provided to score the retrieved indicators (1-100), with the default being 75

By default the integration will pull from all available collections, but you can specify collections= and indicate a comma delimited list of collection-ids to limit the integration's scope

By default, the ingegration will pull all MD5/SHA256 hashes, all ip address and domain name indicators You can specify which types of indicator in the server's section of the configuration to limit the types of indicators ioc_types= (hash,domain,address) as a comma delimited list to

Two forms of authentication username and password or token authentication can be configured, optionally: username= and password= can be set for the former and token for the later. cert= can be optionally provided to locate a .pem encoded certificate+key pair to use during TLS or set to a comma delimited list of the certificate file location followed by the key. verify= can be optionally set to control TLS verification using true or false as boolean values.

Once you have the connector configured with the desired TAXII servers:

service cb-taxii-connector start

Any errors will be logged into /var/log/cb/integrations/cb-taxii-connector/cb-taxii-connector.log.

Troubleshooting

If you suspect a problem, please first look at the Taxii connector logs found here: /var/log/cb/integrations/cb-taxii-connector/cb-taxii-connector.log (There might be multiple files as the logger "rolls over" when the log file hits a certain size).

If you need detail logging, set log_level=DEBUG in the core configuration.

Support

  • View all API and integration offerings on the Developer Network along with reference documentation, video tutorials, and how-to guides.
  • Use the Developer Community Forum to discuss issues and get answers from other API developers in the Carbon Black Community.
  • Report bugs and change requests to Carbon Black Support.

Reporting Problems

When you contact Carbon Black Support with an issue, please provide the following:

  • Your name, company name, telephone number, and e-mail address
  • Product name/version, CB EDR Server version, CB EDR Sensor version
  • Hardware configuration of the EDR Server or computer (processor, memory, and RAM)
  • For documentation issues, specify the version of the manual you are using.
  • Action causing the problem, error message returned, and event log output (as appropriate)
  • Problem severity

Building

To create a build for EL7, run:

FISH: ./gradlew build
BASH: ./gradlew build

To create a build for EL8, run:

FISH: env DOCKERIZED_BUILD_ENV=centos8 ./gradlew build
BASH: export DOCKERIZED_BUILD_ENV=centos8; ./gradlew build

Other common commands for ./gradlew:

  • runPyTest - Runs the python test suite
  • generatePepperReport - Generates a flake 8 based pepper report.
  • createVirtualEnv - Creates the appropriate python virtual environment to build and execute the connector. Can also be used for your IDE's virtual environment.
  • runSmokeTest - Runs the smoke tests available.

cb-taxii-connector's People

Contributors

zacharyestep avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cb-taxii-connector's Issues

Arbitrary STIX file

It would be useful to include the ability to import arbitrary STIX data from a file. Many orgs get this information by file and don't have the resources to host their own STIX server

IP address validation only validates lists, not individual entries

https://github.com/carbonblack/cb-taxii-connector/blob/master/cbopensource/connectors/taxii/cybox_parse.py#L169

The parsing for IPv4 addresses validates the format of lists of IPs via validate_ip_address() and fails cleanly on unexpected inputs, but does not validate individual entries. (e.g., the else block)

If an individual Address entry is malformed, it results in the bad IP address being included in the list of IOCs and a hard fail later in processing (e.g., on feed.dump()).

We should update the else block to call validate_ip_address()


Below is stack trace that triggered this, note the IP address we tried to parse in the feed validation contains an URL. That's a bug in the feed - our bug is we failed hard instead of cleanly.

2017-05-17 08:51:11,904 Sending Poll_Request to https://analysis.foo.com/taxii-data
2017-05-17 08:51:18,787 content blocks read: 2194
2017-05-17 08:51:18,788 current number of reports: 4701
2017-05-17 08:51:18,788 Found 4701 new reports.
2017-05-17 08:51:18,804 Total number of reports: 8860
2017-05-17 08:51:19,363 Traceback (most recent call last):
  File "cbopensource/connectors/taxii/bridge.py", line 468, in runner
  File "cbopensource/connectors/taxii/bridge.py", line 446, in perform
  File "cbopensource/connectors/taxii/bridge.py", line 344, in _import_collection
  File "cbopensource/connectors/taxii/cb_feed_util.py", line 116, in build_feed_data
  File "cbfeeds/feed.py", line 28, in dump
  File "cbfeeds/feed.py", line 91, in validate
  File "cbfeeds/feed.py", line 348, in validate
CbInvalidReport: Malformed IPv4 (1.2.3.4/zeco/panelnew/gate.php) addr in IOC list for report foo-Observable-eeb1d830-ac39-49f0-9317-57ae9b42eb3b

Does not recognize removed entries from a Taxii feed

I know it does not currently recognize reports (STIX) removed from a Taxii feed, but I wanted to suggest an option:

  • Request a pull from taxii with a start deep in the past (maybe leave the ability specify the start date), and end of now.
  • Hourly request an update for the last hour.
  • Daily, pull the entire feed again some time before the nightly CB full feed fetch
  • Let Carbon Black handle the feed management with the daily refresh

I think this will allow the Taxii feed integration handle deleted entries from Taxii.

TAXII connector re-enables deleted fees in Carbon Black

If I delete a feed int he Carbon Black GUI and the integration finds the same feed on the TAXII server again, it will re-import then enable both the deleted feed and the new feed so it appears I have double reports. I think the integration should honor the deleted flag in the database.

subscription_id support

Have feeds that require subscription_id support. Example requirement: "For each subscription, you will receive a subscription ID. This must be present in all Poll_Requests as the value of the “subscription_id” attribute".

Possible issue with README.md

Hello,
I might just be doing it wrong, but it appears some of the sample commands in the README.md are not correct.

Line 151: /usr/share/cb/integrations/cbtaxii/cbtaxii -c /etc/cb/integrations/cbtaxii/cbtaxii.conf
This does work for me after the latest upgrade, I get /usr/share/cb/integrations/cbtaxii/cb-taxii-connector: command not found. If instead, I use /usr/share/cb/integrations/cbtaxii/bin/cb-taxii-connector -c /etc/cb/integrations/cbtaxii/cbtaxii.conf then it works.

Line 166: /usr/share/cb/integrations/cbtaxii/cbtaxii -c /etc/cb/integrations/cbtaxii/cbtaxii.conf -l
Same issue as above, but if I try the alternative path get unrecognized argumets: -l.

Line 183: /usr/share/cb/integrations/cbtaxii/cbtaxii -c /etc/cb/integrations/cbtaxii/cbtaxii.conf --export
Same as the first issue.

If my change in pathing is correct, I would be happy to try and make the necessary changes.

Regards

Parse IP:port from IP addresses in input

Some data sources report IP addresses with a port number (IP:port) in the IP address field. Right now the entire record is rejected; we should strip off the port and add the IP address in the feed instead.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.