gacwr / openuba

A robust, and flexible open source User & Entity Behavior Analytics (UEBA) framework used for Security Analytics. Developed with luv by Data Scientists & Security Analysts from the Cyber Security Industry. [PRE-ALPHA]

Open User Behavior Analytics

A robust, and flexible open source User & Entity Behavior Analytics (UEBA) framework used for Security Analytics. Developed with luv by Data Scientists & Security Analysts from the Cyber Security Industry.

White Paper

source

This project is a work in progress and in a pre-alpha state; input and contributions are warmly welcome

Status Type	Status
Master Build
Development Build
Issues
Closed Issues
Last Commit
Server Docker Stars
Server Docker Pulls
Server Docker Automated
Server Docker Build
License
Releases
Latest Release
Top Language
Code Size
Chat

Problem

Many UBA platforms typically use a "black box" approach to data science practices, which may work best for security analysts who are not interested in the nuts and bolts of the underlying models being used to generate anomalies, baselines, and cases. These platforms view their models as IP.

Solution

OUBA takes an "open-model" approach, and is designed for the small subset of security analysts who have authentic curiosity about what models are doing, and how they work under the hood. We believe in the scientific computing community, and its contributions over the years (libraries, toolkits, etc). In security, rule/model transparency is key, for compliance, response/investigation, and decision making.

To take it a step further, OUBA also makes use of a community driven marketplace for models, similar to a plugin-store, where plugins are security models. This marketplace is where users of OUBA can install security models for their own use cases. Model developers can also upload their models, enabling other OUBA users to reuse them, whether for free, or compensation -- the choice is up to the model developer to make.

Architecture

Goals

To Build a lightweight, SIEM Agnostic, UEBA Framework focused on providing:

• Modeling
  • Model Management
  • Model Library (both community/internally driven)
  • Model Version Control
  • Ready-to-use model modules
  • Feedback Loop for continuous model training
  • "Shadow Mode" for model and risk score experimentation
  • Simple model configuration workflow
  • Model groups
  • Single-fire & Sequential models
  • "White-box" model standard
• Rule Engine
  • Single-fire & deviation-based rules
• Dashboard
  • Modern stack
  • Modular components
  • Live updating
  • Global state, and component state
• Features
  • Rule Storage/Management
  • Case Management
  • Peer-oriented/community intel
  • Lightweight, SIEM-agnostic architecture
  • Flexible/open dataset support
  • Alerting
  • Browser & desktop applications

Stack

• Client Dashboard
  • React
  • Bootstrap
  • Node JS
  • Express JS
  • D3.js
• Model Server (Remote or Local)
• API Server
  • Flask
• Visualization
  • Data Shader
  • Kibana
  • Matplotlib
  • NetworkX
• Modeling
  • Tensorflow
  • Scikit Learn
  • Keras
  • GP Learn
  • DEAP
  • Graphx
  • MLlib
• Compute Engine
  • Spark
  • Elastic Search
• Supported Data Formats (for now)
  • CSV
  • Parquet
  • Flat File

Interface (under active development)

Views

• Home
• Models
• Anomalies
• Cases
• Settings

Model Library

OpenUBA implements a model library purposed with hosting "ready-to-use" models, both developed by us, and the community. For starters, we host the default model repository, similar to any popular package manager (npm, cargo, etc). However, developers can host their own model repository for use in their own instance of OpenUBA.

Model Library Workflow:

Installation/Usage

Go to INSTALL.md

Get the updated code & documentation on XS code here

Our main development, and documentation branches are first pushed to our sponsorship repository, and then eventually pushed to our public free repository. To obtain the most updated code, and documentation for OpenUBA, subscribe to our XS Code repository.

Twitter

http://twitter.com/OpenUBA

Discord (Main Server, and Dev Chat)

Discord Server: https://discord.gg/Ps9p9Wy

Telegram (Backup server, other communications)

Telegram: https://t.me/GACWR

openuba's Issues

Data Ingestion & Testing

These link below are great! to get sample log data on proxy, weblog, dns logetc
https://www.secrepo.com/
https://log-sharing.dreamhosters.com/
We can start testing with Sample user data (user1, user2 etc) and Sample Proxy log data . I will create a new folder "data" folder update folder with sample proxy log data and user data.
Let me know your thoughts ? Or is it too early on data ingesting and testing ?

docker image is not runnable

The internal python version is 3.5, and the core.py is not in core folder. Is any other runnable docker image I can use?

Error in Makefile

No file named mypy in core directory
The output is attached below:

Finish docker integration for user interface

Realtime Alerts

Can we have a realtime alert mechanism via. email.

installation problems

Hi team,

i am trying to run the requirements.txt file using pip3 install and get the following error:

Collecting requirements.txt
Exception:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 342, in run
requirement_set.prepare_files(finder)
File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/usr/lib/python3/dist-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/usr/lib/python3/dist-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/usr/lib/python3/dist-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/usr/lib/python3/dist-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/usr/lib/python3/dist-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/usr/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/simple/requirements-txt/

Application error

Hi, OpenUBA development team!

I'm trying to run openuba locally on Ubuntu 20.04.
Could you please advise how to fix: i have an issue running the application, looks like some problems related to connecting to dev server
I had provided server and application logs and the screenshot from firefox
Will appreciate your help, thank you

logs.zip

Set up http://openuba.org

The quick way will be Github Hosting + Cloudflare. The README can be used as homepage for starters

TODO:

• Adding openuba.org to Cloudflare and set up DNS for Github hosting
• Add CNAME file to master branch

Issues with installation of OpenUBA

I've been examining OpenUBA to see its potential use in the future for our SOC. I have a few questions and issues:

1. Is this project a dissertation topic (M.S. or PhD)
2. Which flavor or version of Linux has OpenUBA been implemented on? I tested the installation instructions on SUSE and UBUNTU, both had the same issues.
3. What version of Python were you using to compile your code? Updated Python versions are notorious for redacting stuff from previous versions.
4. The "pip3 install requirements.txt" had issues with the versions listed in the file, so I manually installed them.
5. The make file crashed with the error that model.py was found twice under different modules (tried to attach an image, but got an error when trying to attach it).
6. Instead of using the make file, I manually tried to compile the modules under the core directory, and received syntax errors and dependency issues. This is why I was asking for the specific version of python you used.
7. Even though the make did not work correctly, I went ahead and tried to install Hadoop and it did not install on SUSE. I have yet to try it on UBUNTU.
   Any help would be appreciated. Thanks.
   Tom Allen

installation failed -- api display error

Hello, I am very interested in your project, but I installed it myself without success，I want to know
1.this project is still alive?
2.Is there a more detailed installation document, can you provide it?
3.I visit http://127.0.0.1:5000/display/teststring/, it shows api display error

4.I visit http://127.0.0.1:3000/, the page display does not look normal

Json installation problème

ERROR ModelEngine: ReadJSONFileFS failed: Expecting value: line 1 column 1 (char 0)

this issues happen when a start "make" step.

Improving documentation

1. Adding a project wiki
2. Improving the existing documentation and publishing using Github pages for starters
3. For the installation of ELK on platforms like Mac, Linux, Windows we can add an additional section or else we can use a shell script for automating the whole ELK installation process
4. This installation script can be added to docker too

URL not found on server

I renamed the .mypy_cache folder to mypy to successfully run the Makefile. While starting the server it gives out a 404error. The output screenshot are attached below

Add a continuous integration service

Adding a continuous integration service like TravisCI or CircleCI can help automate the software builds, tests, and deployments

Installation error.

Hi, After following the installation instructions I am getting an error while running make run_ui its giving me make: *** No rule to make target 'run_ui'. Stop. error.

I wonder how would I go about and solve this. I am not a programmer but very much interested in this project. Any kind of help will be appreciated regarding installation and setting it up.

Thank you.

could you please show me the install documents details

Api Display error

Hello,
When i start make file i got this type of error

can you please help me!

how to build baseline

Project abandoned/dead ?

Hi, last commit seems to be from last year, this project was in pre-alpha but right now there seems to no longer be any work being done on it ? Is there any new project that forks this one or build from scratch something that is alike ?

add badges

adding latest release badge from TravisCI, build status, issues, last commit, license, download, top language, and code size in bytes

saving docker for later

Make breaks - ModuleNotFoundError: No module named 'flask'

Hi,

Compiling breaks, here are the output.

root@20-c1:# pip3 install -r requirements.txt
Requirement already satisfied: flask==1.0.2 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 1)) (1.0.2)
Requirement already satisfied: keras==2.3.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 2)) (2.3.1)
Requirement already satisfied: pandas==0.25.3 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 3)) (0.25.3)
Requirement already satisfied: numpy>=1.9.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 4)) (1.19.5)
Requirement already satisfied: tensorflow==2.5.3 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 5)) (2.5.3)
Requirement already satisfied: coloredlogs==10.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 6)) (10.0)
Requirement already satisfied: pymongo==3.10.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 7)) (3.10.1)
Requirement already satisfied: mypy==0.761 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 8)) (0.761)
Requirement already satisfied: Jinja2>=2.10 in /usr/local/lib/python3.8/dist-packages (from flask==1.0.2->-r requirements.txt (line 1)) (3.1.2)
Requirement already satisfied: Werkzeug>=0.14 in /usr/local/lib/python3.8/dist-packages (from flask==1.0.2->-r requirements.txt (line 1)) (2.2.0)
Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.8/dist-packages (from flask==1.0.2->-r requirements.txt (line 1)) (2.1.2)
Requirement already satisfied: click>=5.1 in /usr/local/lib/python3.8/dist-packages (from flask==1.0.2->-r requirements.txt (line 1)) (8.1.3)
Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.8/dist-packages (from keras==2.3.1->-r requirements.txt (line 2)) (1.0.8)
Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.8/dist-packages (from keras==2.3.1->-r requirements.txt (line 2)) (1.1.2)
Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.8/dist-packages (from keras==2.3.1->-r requirements.txt (line 2)) (1.8.1)
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from keras==2.3.1->-r requirements.txt (line 2)) (5.3.1)
Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.8/dist-packages (from keras==2.3.1->-r requirements.txt (line 2)) (1.15.0)
Requirement already satisfied: h5py in /usr/local/lib/python3.8/dist-packages (from keras==2.3.1->-r requirements.txt (line 2)) (3.1.0)
Requirement already satisfied: python-dateutil>=2.6.1 in /usr/lib/python3/dist-packages (from pandas==0.25.3->-r requirements.txt (line 3)) (2.7.3)
Requirement already satisfied: pytz>=2017.2 in /usr/lib/python3/dist-packages (from pandas==0.25.3->-r requirements.txt (line 3)) (2019.3)
Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (3.19.4)
Requirement already satisfied: tensorboard~=2.5 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (2.9.1)
Requirement already satisfied: opt-einsum~=3.3.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (3.3.0)
Requirement already satisfied: astunparse~=1.6.3 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (1.6.3)
Requirement already satisfied: wrapt~=1.12.1 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (1.12.1)
Requirement already satisfied: grpcio~=1.34.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (1.34.1)
Requirement already satisfied: google-pasta~=0.2 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (0.2.0)
Requirement already satisfied: termcolor~=1.1.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (1.1.0)
Requirement already satisfied: tensorflow-estimator<2.6.0,>=2.5.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (2.5.0)
Requirement already satisfied: flatbuffers~=1.12.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (1.12)
Requirement already satisfied: keras-nightly~=2.5.0.dev in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (2.5.0.dev2021032900)
Requirement already satisfied: typing-extensions~=3.7.4 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (3.7.4.3)
Requirement already satisfied: absl-py~=0.10 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (0.15.0)
Requirement already satisfied: gast==0.4.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (0.4.0)
Requirement already satisfied: wheel~=0.35 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.5.3->-r requirements.txt (line 5)) (0.37.1)
Requirement already satisfied: humanfriendly>=4.7 in /usr/local/lib/python3.8/dist-packages (from coloredlogs==10.0->-r requirements.txt (line 6)) (10.0)
Requirement already satisfied: mypy-extensions<0.5.0,>=0.4.3 in /usr/local/lib/python3.8/dist-packages (from mypy==0.761->-r requirements.txt (line 8)) (0.4.3)
Requirement already satisfied: typed-ast<1.5.0,>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from mypy==0.761->-r requirements.txt (line 8)) (1.4.3)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from Jinja2>=2.10->flask==1.0.2->-r requirements.txt (line 1)) (2.1.1)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (0.6.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.8/dist-packages (from tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (0.4.6)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (1.8.1)
Requirement already satisfied: setuptools>=41.0.0 in /usr/lib/python3/dist-packages (from tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (45.2.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.8/dist-packages (from tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (3.4.1)
Requirement already satisfied: requests<3,>=2.21.0 in /usr/lib/python3/dist-packages (from tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (2.22.0)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.8/dist-packages (from tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (2.9.1)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (4.9)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (5.2.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (0.2.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.8/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (1.3.1)
Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.8/dist-packages (from markdown>=2.6.8->tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (4.12.0)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (3.8.1)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.8/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/lib/python3/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow==2.5.3->-r requirements.txt (line 5)) (3.1.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

root@20-c1:/home/sk/OpenUBA# make
cd core/ ; \
mypy \
anomaly.py \
case.py \
core.py \
database.py \
dataset.py \
entity.py \
entity_test.py \
encode.py \
encode_test.py \
model.py \
process.py \
process_test.py \
risk.py \
riskmanager.py \
display.py \
user.py \
user_test.py \
utility.py \
alert.py \
api.py \
hash.py \
hash_test.py \
--ignore-missing-imports ;
Success: no issues found in 22 source files
cd core/ ; python3.7 core.py ;
Traceback (most recent call last):
  File "core.py", line 21, in <module>
    from flask import Flask, jsonify
ModuleNotFoundError: No module named 'flask'
make: *** [Makefile:3: run] Error 1
root@20-c1:/home/sk/OpenUBA#

OS: Ubuntu 20.04 LTS

Finish docker integration for server