stratosphereips / stratospherelinuxips

Slips, a free software behavioral Python intrusion prevention system (IDS/IPS) that uses machine learning to detect malicious behaviors in the network traffic. Stratosphere Laboratory, AIC, FEL, CVUT in Prague.

Slips v1.0.13

---

Table of Contents

• Introduction
• Usage
• GUI
• Installation
• Extended Usage
• Configuration
• Features
• Contributing
• Documentation
• Troubleshooting
• License
• Credits
• Changelog
• Roadmap
• Demos
• Funding

Slips: Behavioral Machine Learning-Based Intrusion Prevention System

Slips is a powerful endpoint behavioral intrusion prevention and detection system that uses machine learning to detect malicious behaviors in network traffic. Slips can work with network traffic in real-time, PCAP files, and network flows from popular tools like Suricata, Zeek/Bro, and Argus. Slips threat detection is based on a combination of machine learning models trained to detect malicious behaviors, 40+ threat intelligence feeds, and expert heuristics. Slips gathers evidence of malicious behavior and uses extensively trained thresholds to trigger alerts when enough evidence is accumulated.

Introduction

Slips is the first free software behavioral machine learning-based IDS/IPS for endpoints. It was created in 2012 by Sebastian Garcia at the Stratosphere Laboratory, AIC, FEE, Czech Technical University in Prague. The goal was to offer a local IDS/IPS that leverages machine learning to detect network attacks using behavioral analysis.

Slips is supported on Linux and MacOS only. The blocking features of Slips are only supported on Linux

Slips is Python-based and relies on Zeek network analysis framework for capturing live traffic and analyzing PCAPs. and relies on Redis >= 7.0.4 for interprocess communication.

Usage

The recommended way to use Slips is on Docker.

Linux

docker run --rm -it -p 55000:55000 --net=host --cap-add=NET_ADMIN --name slips stratosphereips/slips:latest

./slips.py -f dataset/test7-malicious.pcap -o output_dir

cat output_dir/alerts.log

Macos M1

docker run --rm -it -p 55000:55000 --net=host --cap-add=NET_ADMIN --name slips stratosphereips/slips_macos_m1:latest

./slips.py -f dataset/test7-malicious.pcap -o output_dir

cat output_dir/alerts.log

Macos Intel

docker run --rm -it -p 55000:55000 --net=host --cap-add=NET_ADMIN --name slips stratosphereips/slips:latest

./slips.py -f dataset/test7-malicious.pcap -o output_dir

cat output_dir/alerts.log

For more installation options

For a detailed explanation of Slips parameters

Graphical User Interface

To check Slips output using a GUI you can use the web interface or our command-line based interface Kalipso

Web interface

./webinterface.sh

Then navigate to http://localhost:55000/ from your browser.

For more info about the web interface, check the docs: https://stratospherelinuxips.readthedocs.io/en/develop/usage.html#the-web-interface

Kalispo (CLI-Interface)

./kalipso.sh

For more info about the Kalipso interface, check the docs: https://stratospherelinuxips.readthedocs.io/en/develop/usage.html#kalipso

---

Installation

Slips can be run on different platforms, the easiest and most recommended way if you're a Linux user is to run Slips on Docker.

• Docker
  • Dockerhub (recommended)
    • On a linux host
      • Without P2P support
      • With P2P support
    • On MacOS M1 host
      • Without P2P support
    • On MacOS Intel processor
      • Without P2P support
      • With P2P support
  • Docker-compose
  • Dockerfile
• Native
  • Using install.sh
  • Manually
• on RPI (Beta)

---

Extended Usage

Linux

Analyse your own traffic without P2P

Analyse your own traffic with P2P

Analyse a pcap without using P2P

Macos M1

Analyse your own traffic without using P2P

MacOS Intel processors

Analyse your own traffic without using P2P

Analyse your own traffic with using P2P

Analyse a PCAP without using P2P

Configuration

Slips has a config/slips.conf that contains user configurations for different modules and general execution.

• You can change the timewindow width by modifying the time_window_width parameter

• You can change the analysis direction to all if you want to see the attacks from and to your computer

• You can also specify whether to train or test the ML models

• You can enable popup notifications of evidence, enable blocking, plug in your own zeek script and more.

More details about the config file options here

Features

Slips key features are:

• Behavioral Intrusion Prevention: Slips acts as a powerful system to prevent intrusions based on detecting malicious behaviors in network traffic using machine learning.
• Modularity: Slips is written in Python and is highly modular with different modules performing specific detections in the network traffic.
• Targeted Attacks and Command & Control Detection: It places a strong emphasis on identifying targeted attacks and command and control channels in network traffic.
• Traffic Analysis Flexibility: Slips can analyze network traffic in real-time, PCAP files, and network flows from popular tools like Suricata, Zeek/Bro, and Argus.
• Threat Intelligence Updates: Slips continuously updates threat intelligence files and databases, providing relevant detections as updates occur.
• Integration with External Platforms: Modules in Slips can look up IP addresses on external platforms such as VirusTotal and RiskIQ.
• Graphical User Interface: Slips provides a console graphical user interface (Kalipso) and a web interface for displaying detection with graphs and tables.
• Peer-to-Peer (P2P) Module: Slips includes a complex automatic system to find other peers in the network and share IoC data automatically in a balanced, trusted manner. The P2P module can be enabled as needed.
• Docker Implementation: Running Slips through Docker on Linux systems is simplified, allowing real-time traffic analysis.
• Detailed Documentation: Slips provides detailed documentation guiding users through usage instructions for efficient utilization of its features.

Contributing

We welcome contributions to improve the functionality and features of Slips.

Please read carefully the contributing guidelines for contributing to the development of Slips

You can run Slips and report bugs, make feature requests, and suggest ideas, open a pull request with a solved GitHub issue and new feature, or open a pull request with a new detection module.

The instructions to create a new detection module along with a template here.

If you are a student, we encourage you to apply for the Google Summer of Code program that we participate in as a hosting organization.

Check Slips in GSoC2023 for more information.

You can join our conversations in Discord for questions and discussions. We appreciate your contributions and thank you for helping to improve Slips!

Documentation

User documentation

Code docs

Troubleshooting

If you can't listen to an interface without sudo, you can run the following command to let any user use Zeek to listen to an interface not just root.

sudo setcap cap_net_raw,cap_net_admin=eip /<path-to-zeek-bin/zeek

You can join our conversations in Discord for questions and discussions.

Or email us at

• [email protected]
• [email protected],
• [email protected]

License

GNU General Public License

Credits

Founder: Sebastian Garcia, [email protected], [email protected].

Main authors: Sebastian Garcia, Alya Gomaa, Kamila Babayeva

Contributors:

• Veronica Valeros
• Frantisek Strasak
• Dita Hollmannova
• Ondrej Lukas
• Elaheh Biglar Beigi
• Martin Řepa
• arkamar
• Maria Rigaki
• Lukas Forst
• Daniel Yang

Changelog

https://github.com/stratosphereips/StratosphereLinuxIPS/blob/develop/CHANGELOG.md

Demos

The following videos contain demos of Slips in action in various events:

• 2022 BlackHat Europe Arsenal, Slips: A Machine-Learning Based, Free-Software, Network Intrusion Prevention System [web]
• 2022 BlackHat USA Arsenal, Slips: A Machine-Learning Based, Free-Software, Network Intrusion Prevention System [web]
• 2021 BlackHat Europe Arsenal, Slips: A Machine-Learning Based, Free-Software, Network Intrusion Prevention System [slides] [web]
• 2021 BlackHat USA Arsenal, Slips: A Machine-Learning Based, Free-Software, Network Intrusion Prevention System [web]
• 2021 BlackHat Asia Arsenal, Slips: A Machine-Learning Based, Free-Software, Network Intrusion Prevention System [web]
• 2020 Hack In The Box CyberWeek, Android RATs Detection With A Machine Learning-Based Python IDS [video]
• 2019 OpenAlt, Fantastic Attacks and How Kalipso can Find Them [video]
• 2016 Ekoparty, Stratosphere IPS. The free machine learning malware detection [video]

Funding

We are grateful for the generous support and funding provided by the following organizations:

• NlNet Foundation, https://nlnet.nl/
• Artificial Intelligence Centre at the Czech Technical University in Prague, https://www.aic.fel.cvut.cz/
• Avast, https://www.avast.com/
• CESNET, https://www.cesnet.cz/
• Google Summer of Code (2023, 2024), https://summerofcode.withgoogle.com/

Their funding has played a crucial role in the development and success of this project. We sincerely appreciate their commitment to advancing technology and their recognition of the value Slips brings to the community.

stratospherelinuxips's Issues

Redis - get_message(timeout=None)

The redis python function "get_message()" does not work with argument "timeout=None".
It throws following error:
"
Error in run() of unsupported operand type(s) for *: 'NoneType' and 'int'
"
The "get_message(timeout=None)" will wait (proccess is freezed) until new message arrives.

Read flows from suricata jsons

With the reading Argus and Bro flows read also the flows from Suricata.

Fix the read of Bro conn.log files

It should be possible to read as input the lines of Bro conn.log files that are separeted by TAB chars. But for some reason they are not working.

When reading a pcap the lines read are inconsistent. Check

Add an option to have multiple home networks, so computers roaming can use the tool continually.

Compute the application bytes on the output

When printing or using the flows, compute the applications bytes transferred compared to the total amount of bytes

Data retention

Any idea what is minimum number of network streams should be retained in practical deployments ? For example currently I see that Argus output is being used as input for SLIPS.py .

What is minimum duration or number of streams that argus should capture for SLIPS.py to start detecting attacks ? What is the maximum duration the streams should be retained ?

These parameters define whether it could be run inline mode ? or streams should be exported to offline server

Various small issues in README.md

1. Platforms: slips has also been tested in Ubuntu 16.04 LTS. Also is it OSX 10.9.5 (not IOS)?
2. Usage: In Debian and Ubuntu the command to install argus is sudo apt-get install argus-server argus-clients. Unfortunately this installs the previous version of argus (2.x). Maybe we should add the required version and put a note that it might be better to install from source.
3. Features: the '-m' switch is mentioned in two places but I think this is '-f'.
4. TODO: if these issues have been solved, maybe we should remove them or replace them with the list of wanted features.

Read nfdump flows

Be able to filter in only the flows of the current IP addresses of the host running slips

Create Whois module

We need a whois module that gets each IP address in the profiles, check to which network it belongs, check the local cache of whois data and if there is no match, ask the whois servers.

The data should be in the Redis database.
The module should run as an independent process in the background.

PortScanner detecting things that are not port scan

When this condition happens repeatedly:

• Same src ip connects to the same dst ip, in the same dst port, with not established flows
  Then we count those flows for the port scan detection.
  However, this can also be when a computer loses internet connection and is just retrying.

Since this does not qualifies as a port scan, the port scanner module should not consider these type of flows as part of the port scan count.

Create an alert when we don't know the port but Zeek recognizes it

The logsprocess wrongly computes the amount of modified TW at the beggining

Sometimes the first counter of TW modified is 1 too many.

Threshold calculation

Hi

From the blog, https://stratosphereips.org/stratosphere-ips-generation-of-the-behavioral-models.html
I see that special training methods were employed to determine the feature thresholds(size, duration, periodicity). Any reference to those methods shall be very helpful.

Thanks
Sachin

It seems that the timewindows are not created with Traffic-time, but with real-time

It seems that when creating the TW, we are using the real time of the machine and not the time of the flows. The TW should use the time of the flows.

Make each process use its own connection to the DB

The timeline re-creates the complete file each time. Modify to only add lines.

Find NATed networks heuristically

Implement an algorithm that can recognize which src IPs in the network are NATing other computers. This can be done by:

• Seeing that there are different operating systems using the same IP
• The amount of traffic compares with 2x, 3x, 4x more traffic that other computers alone.
• The timings of the traffic do not correspond with one human, but with many.

This is of course prone to errors because of the heuristics, but it will work in most cases with NATed networks of more than 5 computers.

This should be a module probably.

Create the main structure to load modules

Add to slips the capability to load new modules and run them as processes.

The idea is to read a folder and load each python file there as a separate process.

We should provide:

• An API that the modules can use to access the database.
• A template for a new module to be created, so people knows how to create new modules
• An API for the output: new file, add to profile, etc.

Migrate all the processes from time based to event based

Implement redis Publish–subscribe pattern to run specific code

Implement in the main code and database.py file a pubsub mechanism (https://redis.io/topics/pubsub)
This will allow part of slips and the modules to ask for specific events to happen in the DB to act. Instead of searching the DB for some data, you can ask the DB to wake you up when the data arrives or changes.

For example, if your module wants to check the country of the src IPS you can ask the DB to call you back for each new src IP received (even maybe each new unique src IP received). This would decrease the work of the modules.

Timeline: Some features are printed wrongly with respect to srcip and dstip

When dealing with features of inbound flows, print the srcip that sent the flow instead of the dst ip that received. For example in the profile of IP 34.253.54.22, it received flows from others computers to it, but the timeline says:

• https asked to 34.253.54.22 443/tcp, Size: 42.56Kb, Country: Unknown, ASN Org: Amazon.com, Inc.

The IP in the timeline should have been the srcip sending it. Not the dstip receiving it.

Fix the creation of letters for the tuples

The letters of the stratosphere behavior idea on the tuples are not correctly created. We need to fix how we are computing the time difference between flows, the size of flows and durations and use this to create the letters.

Now the implementation is half finished, but all the letters are 'a'.

Check if slips can run zeek (bro).

If zeek (bro) is not visilble for slips, it throws error. Test it on the beginning and say to user. Also add instruction to README how to do it.

Implement a local cache of IP well known to the user. Custom names

Implement whitelists

Implement a whitelist of IPs that should be completely ignored.

This should go in the main slips core files, not a module. And the ignoration of the flow should be as fast as possible in the processing of flows (probably the first thing done in the profiler)

We can have options for SRC IP whitelist, DST IP whitelist, or BOTH.
If it is BOTH:

• Ignore any flow/packet coming from or to this IP

If it is SRC IP:

• If a flow arrives coming from this SRC IP, ignore the flow.
• If a flow arrives coming for this DST IP, consider the flow

If it is DST IP:

• If a flow arrives coming from this SRC IP, consider the flow
• If a flow arrives coming for this DST IP, ignore the flow.

Add the detections to the timeline

Verbosity and debugging from config file is always ignored

StratosphereLinuxIPS/slips.py

Lines 55 to 77 in 7f48398

	# Get the verbosity, if it was not specified as a parameter
	if args.verbose == None:
	# No args verbose specified. Read the verbosity from the config
	try:
	args.verbose = int(config.get('parameters', 'verbose'))
	except (configparser.NoOptionError, configparser.NoSectionError, NameError):
	# There is a conf, but there is no option, or no section or no configuration file specified
	args.verbose = 1
	# Limit any verbosity to > 0
	elif args.verbose < 1:
	args.verbose = 1
	# Get the Debugging, if it was not specified as a parameter
	if args.debug == None:
	# No args debug specified. Read the debug from the config
	try:
	args.debug = int(config.get('parameters', 'debug'))
	except (configparser.NoOptionError, configparser.NoSectionError, NameError):
	# There is a conf, but there is no option, or no section or no configuration file specified
	args.debug = 0
	# Limit any debuggisity to > 0
	elif args.debug < 0:
	args.debug = 0

args.verbose and args.debug are set to theirs default value and comparison against None will always fail. Therefore, the code above is skiped and values from config file will never be used.

This issue refers to newslips branch

Realize that slips was not given a file to read or any file in the stdin, so stop slips.

If we are not given a file with -r, or a stdin file, we should stop the execution.

Stop Zeek if slips is stopped.

Bug in timeline

When a port scan is detected, the portscan is printed many times after it finished.

Threat Intelligence Feed

For each Dst ip , have some info on the threat level of the ip.

Two ways to implement:

1- Download files of Feed information from some sources. And check the ip on them. For example https://threatfeeds.io/
2- Ask directly to Threat intel sources like VirusTotal, etc.

Add in the profile of the IPs, the mac address and the vendor from the MAC

When Redis database does not run, say to user. Do not throw error.

PortScan not showing enough info to know what happened

The port scan module uses the feature ClientDstPortTCPNotEstablished to detect some type of port scan. However this feature is a sum for all the IPs that were destinations for these ports. Therefore we can not know which flows triggered the port scan.

Check if "zeek_files" folder exists.

Check if "zeek_files" folder exists if not create new one.

Make the logsprocess to read and use the slips.log file for errors.

Timeline: When dealing with UDP, recognize 1 packet flow and do not put 'not established' but 'Not answered'

In the timeline recognize multicast and special addresses

Creating "zeek_files" folder.

If "zeek_files" folder does not exist, slips does not create new "zeek_files" folder so it throws error. Is it for purpose?

It seems that the timeline does not put all the flows????? some are missing when reading a pcap

Implement some unittest to be sure the code does not broke with normal operations

Solve our processes eating 100% of the CPU when idle

Our processes do not block on the queues yet, so when there is no data to receive they eat the CPU.
But we need them to processes the input data as fast as possible.

Evaluate Travis to test and deploy

https://travis-ci.com/

Port scan detection not depending on the width of the time window

Now the port scan module detects amount of packets, but they are not related to the width of the time window. So 10 packets per millisecond are the same as 10 packets per day.

Design a strategy to stop the modules processes.

The problem is that data processing can take time and happen much after the input ended. Ideas:

• Each process realize that there is nothing more to do and just stop. (depends too much on the developer of the module)
• The main process realize that a module is getting out of plan and stops it. (central control)
• Ask each module?

Finish implementing these features

This list of features should be implemented:

clientDestinationPortTotalBytesUDPEstablished
clientDestinationPortNumberOfFlowsTCPEstablished
clientDestinationPortNumberOfFlowsUDPNotEstablished
clientDestinationPortTotalPacketsTCPEstablished
clientDestinationPortNumberOfFlowsUDPEstablished
clientDestinationPortTotalPacketsTCPNotEstablished
clientDestinationPortTotalBytesUDPNotEstablished
clientDestinationPortTotalBytesTCPEstablished
clientDestinationPortTotalPacketsUDPNotEstablished
clientDestinationPortNumberOfFlowsTCPNotEstablished
clientDestinationPortTotalBytesTCPNotEstablished
clientDestinationPortTotalPacketsUDPEstablished

Not sure which ones we are already reading from the flow file or are in the DB. Check

Use Zeek signatures to implement search for leaks

Add ARP port scan detection