iqtlabs / packet_cafe

A platform built for easy-to-use automated network traffic analysis

Overview

Background

Packet Café is a platform built for easy-to-use automated network traffic analysis. It is built to be modular and allow for a pipeline of tools that are triggered by different inputs and outputs. Currently the tools supplied are defined here.

This service accepts PCAP files and then processes them against the pipeline of tools providing automated analysis that gets returned in JSON format. That can then be consumed via the API directly and put into other systems such as SIEMs, searched and filtered through the included JSON viewer, or viewed through the included visualizations of the Packet Café frontend.

The hope and motivation behind the Packet Café platform is to enable lowering the barrier to understanding what is actually in a network traffic capture file (PCAP) and provide insight without having to be an expert in networking.

Ready to get started and take Packet Café for a spin already? Go here. Otherwise, we recommend reading through the design section to better understand the bigger picture.

Packet Café is open source and welcomes contributions to the project.

{% hint style="info" %} While the focus of the project is around network traffic and specifically processing PCAP files, the platform has been built such that the formats of what it can process are only dependent on the set of analytic processes (tools) and what inputs they can handle. The tools could be redefined to ones that handle different inputs while still leveraging the entire pipeline and feedback loop of results. {% endhint %}

packet_cafe's Issues

load balancer unhealthy

The lb container never becomes healthy because no response is received from

curl --silent --fail http://localhost/api/v1 || exit 1

it appears that the update to nginx on May 25 changed the way that configuration works, redulting in the config throwing the error:

nginx: [emerg] "server" directive is not allowed here in /etc/nginx/conf.d/default.conf:1

nginx was then falling through to the default config, causing the proper ports not to be listened on.

Fetching results after refreshing clears all results

Epic and Redux store entry for tool results

Create an epic to fetch tool results and store them locally in the redux store for ease of use.

Update cookies with appropriate SameSite value

based on:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite

it appears that the SameSite value should be Strict

Wildcard routes for the UI should redirect, not serve up the index

Docker error on Windows. packet_cafe_admin can't bind to port 5001

I've pulled the docker image onto a windows machine. Several other containers start without issue but the admin portion reports the following error: The main webpage does come up however. Windows 10 2009 and newest version of docker installed on windows.

(HTTP code 500) server error - Ports are not available: listen tcp 0.0.0.0:5001: bind: An attempt was made to access a socket in a way forbidden by its access permissions.

Update readme to reflect the current direction of the project

Auto "poll" updates from both status and results so that the current buttons are no longer needed

discard invalid uploads

Be able to refresh the browser page and not lose results of that session

Table with view results, should account for tools that have "viewableOutput: false"

Currently there's a show results for all tools, but some tools don't have results, they just modify files.

Add prose to landing page or perhaps in the top bar drop down to explain the service

Normalize identifiers related to Session IDs and Request IDs

As I've been coding a CLI that exercises the Packet Cafe API, I've noticed there are a number of inconsistencies in naming across components that causes some coding confusion and adds friction in coding and debugging and looking through source code to learn as-yet undocumented things, since you can't rely on one string to use for searching with grep -r, etc.

• Session IDs are called, in various places, sessionId, session_id, and sess_id. The API tends to use sess_id, but JSON keys in responses and internal code use the other two.

• Request IDs are called, in various places, req_id, id_dir, fileId, uid, and uuid. The API tends to use req_id, but again variables and dictionary keys use the others.

• What was most confusing to me when first coding to the API was that the API uses /id to mean sess_id, but in JSON responses and internally to source code id seemed to sometimes refer to either Session ID or Request ID. It was hard to learn the relationships between API calls and resulting file paths and JSON responses, etc. I had multiple commands using the column Id, but the column values were different in each table and started translating column headings (which just further confuses things.)

Here are a few comments in my code to remind me:

lim/packet_cafe/__init__.py:        # NOTE(dittrich): Beware: "sess_id" vis "sessionId".
lim/packet_cafe/__init__.py:            # NOTE(dittrich): Beware: "sess_id" vis "sessionId".
lim/packet_cafe/__init__.py:            # NOTE(dittrich): Don't forget: 'req_id' is 'uuid' in result
lim/packet_cafe/api/upload.py:            # NOTE(dittrich): Don't forget: 'req_id' is 'uuid' in result

It would be much cleaner and easier to read code, documentation, and API calls, if sess_id was used for all references to Session IDs and req_id was used for all references to Request IDs and generic alias terms like id, uid, etc. were not used anywhere.

I realize this might take a bit of work, as it requires a lot of careful searching through both Python and Javascript code, as well as requiring significant documentation changes (and API route changes, too). That could argue for the changes being part of a v2 API definition, though that adds its own complexity. Otherwise, these would be breaking changes in the v1 API. Regardless, I think these changes would be well worth making in terms of simplifying coding to the API, understanding internal workings and results, and in clarifying documentation.

Not sure if this is on the list already. Interested to get your thoughts.

Remove cookies popup

Add better file selector controls

Currently the active file selection process is highly unintuitive and can only be done from a single very specific place within the application. It would be beneficial to make this process more obvious and something that can be done from anypoint in the flow of the application.

Once 'submit' is clicked, a 'loading' icon should show until there are results

There could be a lengthy time between submitting and seeing anything "happen", particularly the first time, while Docker is pulling down all the images for the tools.

"delete" route in v1 API

I know that "stop" is a WIP per the API documentation. Is "delete" also on the list? Could it be added easily? (I started looking at the code to find how it could be implemented for a PR, but it looks like it may be in the Javascript code?)

In the mean time, it appears that simply deleting files from the Docker shared volume works. I don't see any (ASCII, at least) index files in that directory tree that contain the session or request IDs, though I suspect there may be state in the Redis container?

I haven't looked into container logs yet to see if I can narrow down where in the code it might be.

My workaround for development testing now is to bypass the API and manipulate the file system directly:

(lim) dittrich at ren in ~/git/packet_cafe on develop
$ lim cafe admin sessions
+--------------------------------------+
| SessionId                            |
+--------------------------------------+
| 30b9ce67-75a4-49e6-b484-c4646b72fbd9 |
| 2d222a53-5b01-4d5e-a659-7da7c21d3cf6 |
| 73d532d7-3b2b-4a93-9a68-ae7091af6a2f |
| 9a949fe6-6520-437f-89ec-e7af6925b1e0 |
| 58a922ac-5483-4502-85b0-84673c473467 |
| 7eedfd93-4f65-4422-8d70-a4edf47d7364 |
| b0cf794e-bd80-4355-bd58-e20ae0473075 |
| a42ee6ab-d60b-4d8e-a1df-cb3dc6985c81 |
+--------------------------------------+
(lim) dittrich at ren in ~/git/packet_cafe on develop
$ for sess_id in $(lim cafe admin sessions -f value | head -n 5)
> do
> rm -rf ~/packet_cafe_data/{files,ids}/$sess_id
> done
(lim) dittrich at ren in ~/git/packet_cafe on develop
$ lim cafe admin sessions
+--------------------------------------+
| SessionId                            |
+--------------------------------------+
| b0cf794e-bd80-4355-bd58-e20ae0473075 |
| a42ee6ab-d60b-4d8e-a1df-cb3dc6985c81 |
| 7eedfd93-4f65-4422-8d70-a4edf47d7364 |
+--------------------------------------+

That isn't ideal, I know. Is there any side-effect of doing this?

Remove ToS popup on upload

Refreshing resets the session ID

Using ISO 8601 timestamps

One of the major issues facing analysts is dealing with multiple data sources created in differing local times using timestamps that do not include timezone. Is the time in this timezone, or in UTC? Are timestamps shown This really (really!) reduces efficiency and accuracy.

The industry standard solution to this problem is using ISO 8601 timestamps.

I am converting dates returned from API calls in my own code, but I think this change would be well worth making (and relatively simple) in packet-cafe itself. You can do this by changing datetime.datetime.utcnow() calls with arrow.utcnow in Python code. I can send a PR for that code. I don't know how to best do this within the Javascript code and tests as I don't code in Javascript.

$ lim cafe status a42ee6ab-d60b-4d8e-a1df-cb3dc6985c81 deb9facbec7a4ccbb952b957b9ad73ff
+---------------+-------------+----------------------------------+
| Tool          | State       | Timestamp                        |
+---------------+-------------+----------------------------------+
| pcap-stats    | Complete    | 2020-05-08T18:03:01.823649+00:00 |
| pcap-splitter | Complete    | 2020-05-08T18:03:21.538080+00:00 |
| mercury       | Complete    | 2020-05-08T18:03:06.372628+00:00 |
| snort         | Complete    | 2020-05-08T18:03:11.875505+00:00 |
| pcapplot      | In progress | 2020-05-08T18:03:19.514207+00:00 |
| p0f           | In progress | 2020-05-08T18:03:23.474442+00:00 |
| pcap-dot1q    | Complete    | 2020-05-08T18:03:00.643334+00:00 |
| ncapture      | Complete    | 2020-05-08T18:02:56.485207+00:00 |
| networkml     | In progress | 2020-05-08T18:03:20.568981+00:00 |
+---------------+-------------+----------------------------------+

Push images to Docker Hub

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency react-router-dom to v6.5.0

Detected dependencies

docker-compose

docker-compose.ui-debug.yaml

docker-compose.yml

• iqtlabs/networkml v0.6.19
• iqtlabs/pcapplot v0.1.11

dockerfile

admin/Dockerfile

• alpine 3.17

lb/Dockerfile

messenger/Dockerfile

• rabbitmq 3-management-alpine

notebooks/Dockerfile

redis/Dockerfile

• redis 7-alpine

ui/Dockerfile

ui/Dockerfile.debug

ui/Dockerfile.test

web/Dockerfile

• alpine 3.17

web/Dockerfile.test

workers/Dockerfile

• alpine 3.17

github-actions

.github/workflows/docker.yml

• actions/checkout v3
• docker/setup-qemu-action v2
• docker/setup-buildx-action v2

.github/workflows/secrets-scan.yml

• actions/checkout v3
• actions/github-script v6
• actions/upload-artifact v3.1.1

.github/workflows/semgrep.yml

• actions/checkout v3
• actions/github-script v6
• actions/upload-artifact v3.1.1
• actions/checkout v3
• actions/github-script v6
• actions/upload-artifact v3.1.1
• actions/checkout v3
• actions/github-script v6
• actions/upload-artifact v3.1.1

.github/workflows/stale.yml

• actions/stale v6

.github/workflows/test.yml

• actions/checkout v3
• codecov/codecov-action v3.1.1

html

ui/public/index.html

• semantic-ui 2.5.0

npm

ui/dossier/package.json

• @fortawesome/fontawesome-svg-core 6.2.1
• @fortawesome/free-brands-svg-icons 6.2.1
• @fortawesome/free-solid-svg-icons 6.2.1
• @fortawesome/react-fontawesome 0.2.0
• apexcharts 3.36.3
• create-react-class 15.7.0
• react ^18.0.0
• react-apexcharts 1.4.0
• react-dom ^18.0.0
• react-router-dom 6.4.5
• semantic-ui 2.5.0
• semantic-ui-react 2.1.4

ui/package.json

• @fortawesome/fontawesome-svg-core 6.2.1
• @fortawesome/free-solid-svg-icons 6.2.1
• @fortawesome/react-fontawesome 0.2.0
• apexcharts 3.36.3
• body-parser 1.20.1
• cors 2.8.5
• d3 7.7.0
• d3-color 3.1.0
• eslint-config-react-app 7.0.1
• express 4.18.2
• multer 1.4.4
• postcss-flexbugs-fixes 5.0.2
• postcss-normalize 10.0.1
• prop-types 15.8.1
• ramda 0.28.0
• react 18.2.0
• react-apexcharts 1.4.0
• react-cookie 4.1.1
• react-data-table-component 7.5.3
• react-dom 18.2.0
• react-redux 8.0.5
• react-router-dom 6.4.5
• react-scripts 5.0.1
• react-tabs 6.0.0
• redux 4.2.0
• redux-actions 2.6.5
• redux-mock-store 1.5.4
• redux-observable 2.0.0
• request 2.88.2
• reselect 4.1.7
• semantic-ui-react 2.1.4
• serve 14.1.2
• styled-components 5.3.6
• uuid 9.0.0
• @babel/core 7.20.5
• @babel/plugin-proposal-nullish-coalescing-operator 7.18.6
• @babel/plugin-proposal-object-rest-spread 7.20.2
• @babel/plugin-transform-modules-commonjs 7.19.6
• @babel/preset-env 7.20.2
• @babel/preset-react 7.18.6
• @babel/register 7.18.9
• chai 4.3.7
• concurrently 7.6.0
• enzyme 3.11.0
• mocha 10.2.0
• nodemon 2.0.20
• nyc 15.1.0
• typescript 4.9.4

pip_requirements

admin/requirements.txt

• bjoern ==3.2.2
• falcon ==3.1.1
• falcon-cors ==1.1.7
• falcon-multipart ==0.2.0

web/requirements.txt

• bjoern ==3.2.2
• falcon ==3.1.1
• falcon-cors ==1.1.7
• falcon-multipart ==0.2.0
• jinja2 ==3.1.2
• pika ==1.3.1
• python-magic ==0.4.27
• redis ==4.4.0

workers/healthcheck/requirements.txt

• flask ==2.2.2
• healthcheck ==1.3.3

workers/requirements.txt

• arrow ==1.2.3
• docker ==6.0.1
• httpx ==0.23.1
• pika ==1.3.1
• redis ==4.4.0

travis

.travis.yml

• node 12

---

• Check this box to trigger a request for Renovate to run again on this repository

clicking 'refresh files' or 'refresh statuses' results in console errors

Uncaught TypeError: Cannot read property 'sessionId' of null
    at Object.e.fetchResults [as onClick] (Table.js:74)
    at e.exports (_apply.js:15)
    at e.exports (_baseInvoke.js:21)
    at e.exports (_apply.js:16)
    at _overRest.js:32
    at Button.js:77
    at Object.u (react-dom.production.min.js:14)
    at p (react-dom.production.min.js:14)
    at react-dom.production.min.js:14
    at m (react-dom.production.min.js:15)

tools after tcprewrite get written to the wrong location

results don't show up for these subsequent tools because of being written to the wrong location

add 'partial' to results until 'finished' is received

baseline-check_circle_outline-24px.svg seems to be missing?

Fix "tabs" view when multiple files are uploaded in the same session

Human readable timestamps in the table

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

convert redis hmset() calls to hset()

Remove Heatmaps

Remove the IP and Port heatmaps as they conflict with the newly implemented UI

Clear Results not working

Clicking the clear results button does not fire any events. This is because no non-null event handlers are being hooked up.

Pressing `clear results` should refresh the page after it's done.

Incorrect npm options

in the command npm install -g npm@latest --no-cache the flag --no-cache has the effect of setting NPM's cache path to false. This is an invalid option and throws a warning but is otherwise ignored. The correct option per: npm/cli#1159 (comment) is --prefer-online

URLs from tool results produce an 'undefined'

Here's an example: http://0.0.0.0/results/undefined/bde10797455b46ce9d64aec16558cf04/mercury