enteee / etherflows

L2 powered flow processing

Latest Blog Posts

• 19 Apr - Convert Video For Chromecast
• 24 Jan - Citation Needed
• 18 Oct - Count Beta Release
• 09 Aug - Disable reMarkable Touchscreen with evkill
• 06 May - Support Binary File Objects with pandas.DataFrame.to_csv
• 01 May - Citation Needed
• 24 Feb - Types for PlantUML Parser
• 07 Jan - reMarkable
• 07 Nov - Citation Needed
• 24 Oct - udev Rule Script Template

etherflows's Issues

Memory issues

I just discovered that the cluster stalls when there are too many documents. Thus I added the techniques described here:

• https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html
• https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html

now i try to verify that the cluster does not kill itself over time.

flowgen id

A user should be able to define the id of the flowgen (3 bytes log). The last 3 bytes of the source address of the generated frames (not vendor part) should be setted accordingly. If no ID was specified the flowgen id should be choosen by random. Also the flowworker should be able to extract the flowgen id from the src-MAC and append it as env.flowgen to the packet.
There is one problem left to solve: In the cluster configuration we have the following filter in place:

ether src b4:be:b1:6b:00:b5

Which ensures that we're only processing packets comming from a flowgen. I have to find a solution to do this with dynamic generated flowgen id. A 'filter all packets from a specific vendor'-bpf rule whould be handy. But I don't think this is possible.. investigating..

Pcaps submit

It would be cool if we could submit a pcap file for analysis purposes. Then the pcap gets analyzed by the flowworker.

flowworker: reset delay

if the flowworker does not process any traffic it will not call write_frame thus will not update Wroker.delay. Which means the last worker.delay will be transmitted to the flowgen, which might be quite high. So we've to find a way to reset the Worker.delay once the worker is in idle mode.

Filter alert

The idea is to write a packet filter for logstash which operates on json data.
The following configuration explains what this filter plugin does:

filter { 
     jqfilter {
         filter => [
             ' Json query string ' , rule uuid
         ]
         target => "field name"
     }
}

this plugin matches the Json query string against the json representation of the event and if matched adds the rule uuid to the field specified with target.

Remove netcat from startup.sh

Integrate tcp socket functionality into flowworker.py

Garbage collection

These days it can happen that a packet is beeing processed by multiple flow workers. This happens because a swtich brodacasts packets with destination MAC to all switchports. We've to eliminate duplicated packes in the ES-database.

Idea how we can do this:

1. Buffer one packet and only send packet to the DB when we get an other packet from the same flow. (flowworker.py ++)
2. Smart periodic cleanup on database with a query and delete script. (cronjob like) Delete packets which are dups.