intelowlproject / intelowl

IntelOwl: manage your Threat Intelligence at scale

Intel Owl

Do you want to get threat intelligence data about a malware, an IP address or a domain? Do you want to get this kind of data from multiple sources at the same time using a single API request?

You are in the right place!

IntelOwl is an Open Source solution for management of Threat Intelligence at scale. It integrates a number of analyzers available online and a lot of cutting-edge malware analysis tools.

Features

This application is built to scale out and to speed up the retrieval of threat info.

It provides:

• Enrichment of Threat Intel for files as well as observables (IP, Domain, URL, hash, etc).
• A Fully-fledged REST APIs written in Django and Python.
• An easy way to be integrated in your stack of security tools to automate common jobs usually performed, for instance, by SOC analysts manually. (Thanks to the official libraries pyintelowl and go-intelowl)
• A built-in GUI: provides features such as dashboard, visualizations of analysis data, easy to use forms for requesting new analysis, etc.
• A framework composed of modular components called Plugins:
  • analyzers that can be run to either retrieve data from external sources (like VirusTotal or AbuseIPDB) or to generate intel from internally available tools (like Yara or Oletools)
  • connectors that can be run to export data to external platforms (like MISP or OpenCTI)
  • pivots that are designed to trigger the execution of a chain of analysis and connect them to each other
  • visualizers that are designed to create custom visualizations of analyzers results
  • ingestors that allows to automatically ingest stream of observables or files to IntelOwl itself
  • playbooks that are meant to make analysis easily repeatable

Documentation

We try hard to keep our documentation well written, easy to understand and always updated. All info about installation, usage, configuration and contribution can be found here

Publications and Media

To know more about the project and its growth over time, you may be interested in reading the official blog posts and/or videos about the project by clicking on this link

Available services or analyzers

You can see the full list of all available analyzers in the documentation.

Type	Analyzers Available
Inbuilt modules	- Static Office Document, RTF, PDF, PE File Analysis and metadata extraction - Strings Deobfuscation and analysis (FLOSS, Stringsifter, ...) - PE Emulation with Qiling and Speakeasy - PE Signature verification - PE Capabilities Extraction (CAPA) - Javascript Emulation (Box-js) - Android Malware Analysis (Quark-Engine, ...) - SPF and DMARC Validator - Yara (a lot of public rules are available. You can also add your own rules) - more...
External services	- Abuse.ch MalwareBazaar/URLhaus/Threatfox/YARAify - GreyNoise v2 - Intezer - VirusTotal v3 - Crowdsec - URLscan - Shodan - AlienVault OTX - Intelligence_X - MISP - many more..

Partnerships and sponsors

As open source project maintainers, we strongly rely on external support to get the resources and time to work on keeping the project alive, with a constant release of new features, bug fixes and general improvements.

Because of this, we joined Open Collective to obtain non-profit equal level status which allows the organization to receive and manage donations transparently. Please support IntelOwl and all the community by choosing a plan (BRONZE, SILVER, etc).

🥇 GOLD

Certego

Certego is a MDR (Managed Detection and Response) and Threat Intelligence Provider based in Italy.

IntelOwl was born out of Certego's Threat intelligence R&D division and is constantly maintained and updated thanks to them.

The Honeynet Project

The Honeynet Project is a non-profit organization working on creating open source cyber security tools and sharing knowledge about cyber threats.

Thanks to Honeynet, we are hosting a public demo of the application here. If you are interested, please contact a member of Honeynet to get access to the public service.

Google Summer of Code

Since its birth this project has been participating in the Google Summer of Code (GSoC)!

If you are interested in participating in the next Google Summer of Code, check all the info available in the dedicated repository!

🥈 SILVER

ThreatHunter.ai

ThreatHunter.ai®, is a 100% Service-Disabled Veteran-Owned Small Business started in 2007 under the name Milton Security Group. ThreatHunter.ai is the global leader in Dynamic Threat Hunting. Operating a true 24x7x365 Security Operation Center with AI/ML-enhanced human Threat Hunters, ThreatHunter.ai has changed the industry in how threats are found, and mitigated in real time. For over 15 years, our teams of Threat Hunters have stopped hundreds of thousands of threats and assisted organizations in defending against threat actors around the clock.

🥉 BRONZE

Docker

In 2021 IntelOwl joined the official Docker Open Source Program. This allows IntelOwl developers to easily manage Docker images and focus on writing the code. You may find the official IntelOwl Docker images here.

DigitalOcean

In 2022 IntelOwl joined the official DigitalOcean Open Source Program.

Other collaborations

• LimaCharlie
• Tines

About the author and maintainers

Feel free to contact the main developers at any time on Twitter:

• Matteo Lodi: Author and principal maintainer
• Simone Berni: Backend Maintainer
• Daniele Rosetti: Frontend Maintainer
• Eshaan Bansal: Key Contributor

intelowl's Issues

Celery cannot connect to rabbitmq

Issue I am facing:
When I create a new analyze request using the pyintelowl client, the job gets created but it keeps saying `status: "running" and the command hangs, see here:

And if I visit the server's GUI to query the database, this is what I get:

The job just hangs there finally throws me time-out error.

What I tried:

I have tried with different analyzers, different files and observables as well. I am facing the same issue.
There are no errors in my docker-compose terminal.

Any help would be appreciated.

PEframe integration

We should add an analyzer to analyze files with PEframe

https://github.com/guelfoweb/peframe

Google WebRisk API analyzer

https://cloud.google.com/web-risk

We could create an analyzer for the WebRisk API. In this way, we can check a specific URL against the provided blacklists. This is the commercial version of the "GoogleSafeBrowsing" APIs.

Add support for external authentication methods (LDAP, GSuite, ...)

We should add support for some common authentication methods that could be leveraged to avoid the pain to create and manage new users directly in IntelOwl.

• LDAP support
• GSuite via SAML
• others, if requested

ThreatMiner API

We should add an analyzer to extract info about an IP/URL/hash from ThreatMiner.

https://www.threatminer.org/api.php

Suggestions Please @mlodic.

Custom fields in admin view

Custom column fields and searchable fields in admin view models for Job and Tag DB models.

OPSWAT Metadefender API

We should add an analyzer to extract info from opswat metadefender cloud

https://onlinehelp.opswat.com/mdcloud/Description_of_file_categories.html

Suggestions Please @mlodic

add indexes for searchable fields + change data retention

Matteo Lodi:

• We could add indexes for the fields that are being used for search purposes.
  Obviously, for testing and for a "classic" usage of this project this is not necessary, but to let it scale we should index the most important ones. https://docs.djangoproject.com/en/2.2/ref/models/indexes/

• Also, at the moment there is a cronjob in crons.py that performs a cleanup of old data. The retention days are hardcoded. We could move this value to the configuration and let the intel owl users decide the retention days.

[analyzer] screenshot to detect phishing campaigns

We could add a Manet analyzer to get a screenshot from a specified URL that we would like to analyze.

https://github.com/vbauer/manet

We should add the docker image as optional in the docker-compose configuration and use the Manet APIs to run the analysis.

SecurityTrials API Integration

We should add an analyzer to extract info from Securitytrials

https://securitytrails.com/corp/api

Suggestions Please @mlodic.

Authentication Service (DRF)

Task #1: (from eshaan's GSoC proposal)

• login, logout API endpoints in Django application
• add the django-cors-headers package to deal with preflight CORS requests.

Adding Auth0 API Module

We should add an analyzer to extract info from Auth0.

https://auth0.com/signals/docs/#get-full-ip-address-reputation-info

Suggestions Please @mlodic.

Malware Bazaar API

We should leverage the new service created by Abuse.ch to create an analyzer that checks a file hash against the Malware Bazaar DB.

https://bazaar.abuse.ch/api/#query_hash

Droidbox integration

We should integrate Droidbox as a new analyzer. As stated on the honeynet website,

Droidbox is a dynamic analysis platform for android applications. Droidbox was developed by Patrik Lantz as part of GSoc 2011. You can try it out by downloading Android Reverse Engineering virtual machine, which bundels droidbox as well as additional android malware analysis tools.

Team Cymru Hash analyzer

We could create an analyzer that checks a specific hash against the malware database of Team Cymru

https://www.team-cymru.com/mhr.html

Divide requirements.txt into development.txt and production.txt

We could separate the requirements.txt file into small files such as development/local.txt and requirements/production.txt.

Originally posted by @Andrew-Chen-Wang in #95 (comment)

Thug integration

We should integrate Thug as a new analyzer: https://github.com/buffer/thug

Even if Thug can be easily installed with pip for python3, it really requires a lot of dependencies to be installed in the machine (in particular https://github.com/area1/stpyv8/). We need to try to study how to add it to the project without "poisoning" the base docker image too much. For this purpose we could just add the official Thug docker image (https://hub.docker.com/r/buffer/thug/dockerfile) to the docker-compose file but, then, we should expose that container to the main container to let it run Thug via API.

Some other requirements:

• give the chance to setup custom proxy
• give the chance to setup custom browser
• give the chance to analyze only URLs or HTML files
• use all DOM events as default

Add GeoCity DB to Maxmind analyzer + update release

At the moment this analyzer (https://github.com/intelowlproject/IntelOwl/blob/master/api_app/script_analyzers/observable_analyzers/maxmind.py) leverages only the "Country" DB.

We could add the download of the GeoCity DB weekly and the related check.

Example usage of the library: https://geoip2.readthedocs.io/en/latest/

MISP connector

IntelOwl was designed to be integrated with other security tools very easily.
In particular, it is a perfect fit for Threat Intelligence Platforms because it can enrich the data available in those platform and can be customized on its own needs.

So, in order to help the community to achieve this result, we should add new connectors for the most used open source TIPs, for example: MISP

We should study their project and add a relative connector to integrate IntelOwl with MISP.
An idea could be to create a new expansion module (https://github.com/MISP/misp-modules) to allow MISP users to enrich an indicator via IntelOwl and integrate the results on the relative event.
We should leverage the official IntelOwl client for this purpose.

Analyze Chrome browser extensions (CRX)

Browser extensions in Chrome are identified by an alphanumeric identifier.

For example, Google Keep has the following URL:

https://chrome.google.com/webstore/detail/google-keep-chrome-extens/lpcaedmchfhocbbapmcbpinfpgnhiddi

which leads to the extension ID lpcaedmchfhocbbapmcbpinfpgnhiddi.

Extension IDs are used to determine the installation folder, e.g.

C:\Users\[login_name]\AppData\Local\Google\Chrome\User Data\Default\Extensions\lpcaedmchfhocbbapmcbpinfpgnhiddi

Also, in normal mode (not in incognito windows) local resources injected by an extension inside web pages is loaded by using urls of this form:

chrome-extension://lpcaedmchfhocbbapmcbpinfpgnhiddi/resource-name.js

Knowing whether an extension is malicious or not is a very useful task in many cases, and several public services/APIs already exist for this purpose, for example:

• https://crxcavator.io/
• https://robwu.nl/crxviewer/

Being able to analyze CRX IDs would be very useful IMHO.

URLscan API

We should add an analyzer to extract info from URLscan

https://urlscan.io/about-api/

Suggestions Please @mlodic.

Malpedia integration

We should add an analyzer to scan a given file against all the yara rules available in Malpedia.

https://malpedia.caad.fkie.fraunhofer.de/usage/api#apiscanbinary

VirusTotal Sandbox analyzer

We could add another analyzer for VirusTotal that extracts the sandbox reports available for a specific file.

https://developers.virustotal.com/v3.0/reference#file-behaviour-summary

ClamAV integration

We should add a new analyzer to allow users to analyze files with ClamAV.

https://github.com/Cisco-Talos/clamav-devel.

It's important to:

• if the license is permissive, add public-available rules by default

• give the chance to customize the rules used to scan a file with ClamAV, in particular to run its own rules

Discussion: Traefik for self-hosted deployment + serverless deployment with celery

Hey @eshaan7. I'm a contributor on cookiecutter-django, and I think the way you've set up this project could take some inspiration from that repository.

For one, the compose files could be labeled a little better. Other things like the .env files could be in their own directory, the Dockerfiles can be in their own, and the requirements could be separated to make the Docker builds smaller.

The one main part is Traefik: auto-renewal of certs and easy to configure with compose. You can take a look at production.yml and compose/production/traefik for how we configure it. It may come useful for those looking for self deployment.

If people want to scale, I made a complete-walkthrough repository for those looking to do AWS ECS deployment via git push to GitHub. It can be made into fargate if you don't integrate celery. (Which is the other thing: I don't think using Fargate with this project would be wise since you're using celery. The first two answers in this SO makes it clear why.).

First Use Clarity

It's not really clear on first use what analyzers are available, or how the platform is to be used. For example, I try to search an MD5 once logged in (eg. Mimikatz) and get no results. Do I need to enable some more analyzers, or is there something else I must do to populate a local DB?

Discussion: Env files should have .env extension

Some IDEs like having .env files because their extensions can easily digest those and show syntax errors. Additionally, it's just habit/conventional to have these "." files that are either unnecessary or "secretive" stay as . so that gitignore files can have something like:

.*
!.travis.yml
etc.

GreyNoise API Integration

Stumbled upon GreyNoise.
They have an "IP Context" API endpoint - https://docs.greynoise.io/#ip-context.

Should we integrate this ? @mlodic

Base class for analyzers run function

Under script_analyzers/, we have too much duplicate code in each analyzer's run() function. For example, getting additional_config_params, verifying them, setting up report, catching and handling errors. Can't we take more of an Object oriented approach here ?

This is just a very basic overview of what I have in mind:

from abc import ABC, abstractmethod
import requests

class AnalyzerRunNotImplemented(Exception):
    detail = "run() is not implemented for analyzer {}"

class Analyzer(ABC):
    analyzer_name = None
    md5 = None
    observable_classification = None
    # and more variables
    ...

    @abstractmethod
    def run(self, *args, **kwargs):
        # this should be overwritten in 
        # child class
        raise AnalyzerRunNotImplemented(self.analyzer_name)

    def __init__(self, *args, **kwargs):
        # initialize shared variables
    
    def get_config(self, *args, **kwargs):
        # fetch additional_config_params
        # verify params, API keys, etc

    def set_report_and_cleanup(self, *args, **kwargs):
        # sets report logic
    def handle_error(self, *args, **kwargs):
        # handles error logic


class Shodan(Analyzer):
    analyzer_name = "shodan"
    observable_classification = "ip"
    url = "https://api.shodan.io/shodan/host/"
    
    def run(self):
        try:
            data = {"ip": self.observable_value}
            resp = requests.post(self.url, json=self.data)
            return super(self).set_report(resp)
        except Exception:
            return super(self).handle_error(resp)

Advantages:

• reduce a lot of duplicate code
• code looks more clean
• makes it easier for new contributors to integrate an analyzer.
• don't have to pass a bunch of variables to the run function, since we can just store them as member variables of the class. def run(analyzer_name, job_id, observable_name, observable_classification, additional_config_params) - This does not look good.
• in some analyzers, the error/report handling is different - for those we can just overwrite the set_report_and_cleanup and handle_error functions too. But for all others, we don't need to.
• We already have code for handling error, setting up report and all in general.py, we would mostly need to refactor it work with objects/classes.

OpenCTI connector

IntelOwl was designed to be integrated with other security tools very easily.
In particular, it is a perfect fit for Threat Intelligence Platforms because it can enrich the data available in those platform and can be customized on its own needs.

So, in order to help the community to achieve this result, we should add new connectors for the most used open source TIPs, for example: OpenCTI

We should study their project and add a relative connector to integrate IntelOwl with OpenCTI. A related issue was already opened: OpenCTI-Platform/connectors#43.
We should leverage the official IntelOwl client for this purpose

Phishtank analyzer

We could add an analyzer that checks if a URL is in the Phishtank phish list.

https://www.phishtank.com/api_info.php

RIPE API analyzer

Get WHOIS for a specific IP with the RIPE API:

https://github.com/RIPE-NCC/whois/wiki/WHOIS-REST-API
https://www.ripe.net/manage-ips-and-asns/db/support/documentation/ripe-database-query-reference-manual#tables-of-query-types-supported-by-the-ripe-database
http://jaluther.blogspot.com/2016/04/python-for-network-engineers-part-2.html

Incorrect analyzer name passed for HybridAnalysis Observable tests

In the tests/test_observables.py, For HybridAnalysis I noticed that we are passing HA_Get as the analyzer name in all 3 instances, shouldn't it be HybridAnalysis_Get_Observable, as defined in the analyzers_configuration.json ? or am I wrong?

All the 3 instances referenced:
https://github.com/certego/IntelOwl/blob/af05c109e253100ae9a2b0087c96676bdb19dae4/tests/test_observables.py#L93
https://github.com/certego/IntelOwl/blob/af05c109e253100ae9a2b0087c96676bdb19dae4/tests/test_observables.py#L157
https://github.com/certego/IntelOwl/blob/af05c109e253100ae9a2b0087c96676bdb19dae4/tests/test_observables.py#L244

[New Analyzer] Drakvuf integration

We should add a new analyzer to allow users to analyze files using a Drakvuf instance:

https://github.com/tklengyel/drakvuf

ONYPHE API Integration

ONYPHE is a search engine for open-source and cyber threat intelligence data collected by crawling various sources available on the Internet or by listening to Internet background noise.

They have many scanning services available for an IP/Domain/URL.
Please take a look at API docs and suggest the ones we need.

There's a python client, pyonyphe available.
Plus, an expansion-plugin in MISP - MISP/....../expansion/onyphe.py

Should we try to integrate this with the MISP analzyer we have or use the python client available or just the API ?

Shodan analyzer

We should add an analyzer to extract info about an IP using Shodan APIs.

https://developer.shodan.io/api
https://shodan.readthedocs.io/en/latest/

Using black and flake8 effectively

Integrate psf/black and flake8 for better linting and consistent styling.

Dynamic configuration at time of requesting scan

ATM, there's no way to dynamically pass in a configuration for an analyzer at the time of requesting a scan. For any configuration changes, the user needs to edit the analyzer_config.json manually.

Breaking past this limit could be one of the most exciting and powerful features of Intel Owl.

Some use cases:

• an option that allows requesting an analysis with X instances of a single analyzer with different/dynamic configuration.
• ...

error in file (env_file_app) environment variable name 'MOCK_CONNECTIONS

Wigle.net integration

We can integrate wigle.net's API for wireless network’s analysis.

Interesting API endpoints:

• https://api.wigle.net/swagger#/Bluetooth_search_and_information_tools/detail
• https://api.wigle.net/swagger#/Cell_search_and_information_tools/mccMnc

Box-js integration

We should add an analyzer to analyze malicious Javascript files using Box-js

https://github.com/CapacitorSet/box-js

Tags Model

• A tags column in the Job table in the database. That way, a user can distinguish between or group different jobs.

Add Tranco API analyzer

https://tranco-list.eu/api_documentation

api_app_job"."errors", "api_app_job"."file"

URLhaus integration

https://urlhaus-api.abuse.ch

Check a URL, domain or IP against the URLhaus database

f-strings everywhere

Using f-strings increases code readability and are known to be faster than str.format()

APKiD analyzer for APKs

We could integrate APKiD which is an Android Application Identifier for Packers, Protectors, Obfuscators and Oddities - PEiD for Android.

GitHub: https://github.com/rednaga/APKiD

More Intel sources (API key required)

Adding to the awesome list already, we can also integrate the following sources to fetch intel:

intelx.io [Data Breaches] : https://blog.intelx.io/2020/04/13/new-python-api-wrapper-and-command-line-interface/
binaryedge.io [~Shodan]: https://docs.binaryedge.io/api-v2/
spiderfoot [Mega-Platform for many such sources]: https://www.spiderfoot.net/documentation/
phonebook.cz [A part of Intelx.io]: https://phonebook.cz/

Improve the web interface

At the moment, IntelOwl can be used mainly via APIs because its major intent is to be integrated with other security tools.
So now we can find only a very basic interface that allow the users:

• to authenticate
• to query the database for analysis results
• to view the results via a JSON file

We should start improving the web interface to allow the project to be more user friendly:

• Add the chance to analyze an observable from the web interface
• Add the chance to view the analyzer configuration and to change it
• etc...more ideas are welcome

FireHOL blocklists analyzer

http://iplists.firehol.org/

We could integrate an analyzer for the FireHOL blocklists. In this way, we could check if a specific IP was reported as malicious by one of those lists.

We should manage the download of these lists internally in the project. Then we should update these lists regularly to allow the IntelOwl users to get fresh results.