csirtgadgets / bearded-avenger

CIF v3 -- the fastest way to consume threat intelligence

Getting Started

Do NOT try to install from the master repo.

For installation instructions and various unix distribution guides, use the DeploymentKit.

https://github.com/csirtgadgets/bearded-avenger-deploymentkit/wiki

Integrations

Several integrations, plugins, and extensions have been written to bridge CIFv3 with other tools. Here are some examples to consider building an ecosystem:

• Python client
• Dockerized Python client
• PowerShell client
• MineMeld plugin
• Elastic threat intel integration
• Threatbus backbone
• Intelmq output bot

Getting Help

• the Wiki
• Known Issues
• FAQ

Getting Involved

There are many ways to get involved with the project. If you have a new and exciting feature, or even a simple bugfix, simply fork the repo, create some simple test cases, generate a pull-request and give yourself credit!

If you've never worked on a GitHub project, this is a good piece for getting started.

• How To Contribute
• Mailing List
• Project Page

Getting Started with Development

https://github.com/csirtgadgets/bearded-avenger/wiki

COPYRIGHT AND LICENSE

Copyright (C) 2017 the CSIRT Gadgets Foundation

Free use of this software is granted under the terms of the Mozilla Public License (MPLv2).

bearded-avenger's Issues

break storage into thread controlled by router

debian packaging

• clean up pyinstaller code, create binaries
• edit makefile to create debian packages based on binaries (somewhat done)
• create debian ppa
• create script to publish packages to ppa on release

https://packagecloud.io/

alienvault feeds

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/rules/example/alienvault.yml

cache feeds differently than in v2

in cifv2 we cache a feed appended with the feed name in the feed config. This causes us to download the same feed multiple times if we were to do something like this:

dga-feed-cryptolocker:
  remote: http://osint.bambenekconsulting.com/feeds/dga-feed.txt
  pattern: ^([^,]+)\,Domain used by (Cryptolocker[^,]+)\,([^,]+)\,([^\r\n]+)
dga-feed-p2p-gameover-zeus:
  remote: http://osint.bambenekconsulting.com/feeds/dga-feed.txt
  pattern: ^([^,]+)\,Domain used by (P2P Gameover Zeus[^,]+)\,([^,]+)\,([^\r\n]+)

This results in dga-feed.txt downloaded multiple times:

ls -lh /var/smrt/cache/osint.bambenekconsulting.com-dga*
-rw-rw-r-- 1 cif cif 83M Sep 20 20:15 /var/smrt/cache/osint.bambenekconsulting.com-dga-feed-cryptolocker
-rw-rw-r-- 1 cif cif 83M Sep 20 20:15 /var/smrt/cache/osint.bambenekconsulting.com-dga-feed-p2p-gameover-zeus

In v3 let's attempt to only download dga-feed.txt one time, cache it and then parsed the single cached feed.

default limit for largest cidr in a feed

1. Add the ability to limit the largest cidr found in a feed (e.g. /24, /16, /8)
2. Make this configurable so people can change the default

problem: missing cifv3 book

• bring over old wiki topics from https://github.com/csirtgadgets/massive-octo-spice/wiki/The-CIF-Book
• update with new information

ubuntu16 LTS deployment

cif-smrt feature: manipulate parsed data

How hard would it be to be able manipulate parsed feed data before mapping it to a cif parameter. Here's an example:

cybercrime rss feed:

<item>
<title>datatransfer.0xhost.net/25asejd14c2f5r7d4e5f63g2v5d41s/adsf544s2d5f778rg45j6hv32bn41h5j4f/solar/index.php?login</title>
<link>http://cybercrime-tracker.net/index.php</link>
<pubDate>22-07-2015</pubDate>
<description>Solar</description>
</item>

The date format is not one DateTime::Format::DateParse; likes.

It would be neat to be able to do something like this on parsed data:

pubDate:
  pattern: '(\S+)'
  subparse: (\d{1,2})-(\d{1,2})-(\d{4})
  new: $3-$2-$1
  values: lasttime

tokens support

problem: not doing multi-process, end to end testing

create advanced testing that spins up threads, etc

• can't do more than 1 thread at a time..
• mocker?
• test functions first...
• remove complex tests, push to complex folder?

https://github.com/zeromq/pyre/blob/master/tests/test_pyre.py
https://github.com/zeromq/pyre/blob/master/tests/test_zactor.py
https://github.com/zeromq/pyzmq/tree/master/zmq/tests

delim.py

https://docs.python.org/2/library/csv.html#csv.Sniffer

see if this might reduce some of our code.

p2p framework should restrict what's notified by notify flag

Multiple failures when following quickstart guide

• Failure because pip not present
• Failure pip install -r requirements.txt not run with sudo, Python.h unavailable

Solution:
sudo apt-get install python-pip python-dev
sudo pip install -r requirements.txt

bambenek feeds

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/rules/example/bambenekconsulting_com.yml

elasticsearch support

add elasticsearch support to cif.store

distro builds

https://build.opensuse.org/
http://openbuildservice.org/

travis deployment testing

• add deployment testing to circleci / travis

switch to pyzmq 15

improve tests for running components

• httpd
• router
• storage

etc...

clean up ping routes

• ping
• ping?write=true
• default ping should check for /help ?

problem: there is no stats forecasting

Look at using rrdtool + holt winters forecasting to monitor feed changes.

cif client output plugin to a local CIF instance

Create the ability for the cif client to pull a feed from one CIF server and push that data into another CIF server (localhost). Example command:

cif -C org_a_cif.yml --feed fqdn --tags botnet -f cif_api --recipient-config localhost_cif.yml

api throttle limiting

http://flask-limiter.readthedocs.io/en/stable/

problem: there is not stats interface (api, web ui)

• internal tables
• by provider
• by country / asn / etc
• provide msg type, interface, etc
• cif-store should provide this and keep track of.

aws deployment testing

• build aws deployment scripts
• create live api-testing scripts
  • populate database
  • test ping / query / submission

break gatherers out into threads

dockerize

https://github.com/Mazon/Jorgmundr

break hunters out into threads controlled by router

deb packaging

https://nylas.com/blog/packaging-deploying-python

abstract global ansible roles

• create development role that takes in local code base
• create role that's publishable to ansible galaxy

explore probability of risk

https://github.com/csirtgadgets/bearded-avenger/wiki/Probability

add gatherer support

garwarn blog

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/rules/example/garwarn.yml

archiving older data

• find and merge older data (configurable) using firsttime/lasttime/reporttime into weekly or monthly using count
• should run on router? (router should figure out if something's been merged or not if multiple routers are running and merging data)

searches are getting logged even if token is invalid

cif-smrt integration RESTful APIs (CIF API)

create the ability to cif-smrt to easily interact with RESTful APIs (CIF API).

• passing parameters
• keeping state of last ingested data

gatherer needs threading

py3 compliant

client feature parity with cif-sdk-py

easybutton

problem: wikidoc out of date, missing API doc, howto's

pull from https://github.com/csirtgadgets/massive-octo-spice/wiki

expand out arrow parser

2016-03-28 18:55:42,289 - DEBUG - cif.smrt.parser[66] - {"additional_data": null, "altid": "safebrowsing.clients.google.com", "altid_tlp": "white", "application": null, "asn": null, "asn_desc": null, "cc": null, "city": null, "confidence": 7, "description": "attackpage", "firsttime": "2016-03-28T22:55:41.916676Z", "group": "everyone", "indicator": "globalnursesonline.com", "itype": "fqdn", "lasttime": "1970-08-22T08:05:04.000000Z", "latitude": null, "longitude": null, "peers": null, "portlist": null, "protocol": null, "provider": "malwaredomains.com", "reference": null, "reference_tlp": null, "reporttime": "2016-03-28T22:55:41.916659Z", "tags": ["exploit", "malware"], "tlp": "green"}
2016-03-28 18:55:42,289 - DEBUG - cif.smrt.parser[58] - {'confidence': 7, 'tlp': 'green', 'description': 'attackpage', 'tags': ['exploit', 'malware'], 'altid_tlp': 'white', 'altid': 'safebrowsing.clients.google.com', 'indicator': 'gsiworld.neostrada.pl', 'provider': 'malwaredomains.com', 'lasttime': '20160304'}
2016-03-28 18:55:42,289 - DEBUG - cif.smrt.parser[66] - {"additional_data": null, "altid": "safebrowsing.clients.google.com", "altid_tlp": "white", "application": null, "asn": null, "asn_desc": null, "cc": null, "city": null, "confidence": 7, "description": "attackpage", "firsttime": "2016-03-28T22:55:41.916676Z", "group": "everyone", "indicator": "gsiworld.neostrada.pl", "itype": "fqdn", "lasttime": "1970-08-22T08:05:04.000000Z", "latitude": null, "longitude": null, "peers": null, "portlist": null, "protocol": null, "provider": "malwaredomains.com", "reference": null, "reference_tlp": null, "reporttime": "2016-03-28T22:55:41.916659Z", "tags": ["exploit", "malware"], "tlp": "green"}
2016-03-28 18:55:42,289 - DEBUG - cif.smrt.parser[58] - {'confidence': 7, 'tlp': 'green', 'description': 'attackpage', 'tags': ['exploit', 'malware'], 'altid_tlp': 'white', 'altid': 'safebrowsing.clients.google.com', 'indicator': 'guy-ouellette.net', 'provider': 'malwaredomains.com', 'lasttime': '20160304'}
2016-03-28 18:55:42,289 - DEBUG - cif.smrt.parser[66] - {"additional_data": null, "altid": "safebrowsing.clients.google.com", "altid_tlp": "white", "application": null, "asn": null, "asn_desc": null, "cc": null, "city": null, "confidence": 7, "description": "attackpage", "firsttime": "2016-03-28T22:55:41.916676Z", "group": "everyone", "indicator": "guy-ouellette.net", "itype": "fqdn", "lasttime": "1970-08-22T08:05:04.000000Z", "latitude": null, "longitude": null, "peers": null, "portlist": null, "protocol": null, "provider": "malwaredomains.com", "reference": null, "reference_tlp": null, "reporttime": "2016-03-28T22:55:41.916659Z", "tags": ["exploit", "malware"], "tlp": "green"}

api: feeds route

feeds logic should go from code to the API so you can do things like:

/feeds?itype=ipv4&confidence=35

and it does the timestamp logic for you vs query. get this out of the client code make them easier to maintain.

centos deployment

deployment/centos

pluggable route handlers

http://stackoverflow.com/questions/1057431/loading-all-modules-in-a-folder-in-python/1057534#1057534

make new types of routes a drop-in (storage?)

leveldb plugin

make deployment proxy friendly

problem: confidence scale is not documented

1. this is probably informational, it's usually adjacent to a primary indicator
   1. could be the MX or NS record of a FQDN name
8. i trust who observed this, it's probably an ip address or fqdn combined with a timestamp
9. i trust who saw this and what they think it was
   1. probably a url or binary hash
   2. could be an ip address when combined with a port, protocol and timestamp
10. i [as a human analyst, non-machine generated] saw this, i know what it is
    1. most likely a url, binary hash
    2. a search

Expand on the definition of --limit in the SDK

Consider having two "limit" knob's in the SDK:

1. --query-limit - this adjusts the default database query limit specified by the client.
   • max-query-limit is set on the server (e.g. 225k)
   • default-sdk-query-limit is set within the SDK (e.g. 50K)
   • --query-limit allows 0 - 225k effectively
2. --output-limit - this adjusts the output limit specified in the SDN
   • default-output-limit is NULL
   • --output-limit allows 0 through MAX records

inverse tag query

ability to do:

--tags botnet --not-tags dns,smtp,facilitator