Coder Social home page Coder Social logo

tklab-tud / id2t Goto Github PK

View Code? Open in Web Editor NEW
51.0 51.0 22.0 29.91 MB

Official ID2T repository. ID2T creates labeled IT network datasets that contain user defined synthetic attacks.

License: MIT License

Shell 1.50% Python 74.88% CMake 0.49% C++ 23.13%
dataset-generation intrusion-detection statistical-analysis

id2t's People

Contributors

jsonscorps avatar m-vasiloma avatar neproxx avatar pepper-jk avatar peppie123 avatar piratehideout avatar ra1nb0rn avatar saltyhash123 avatar thrimbor avatar xtikk avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

pepper-jk xtikk andreysfc trace-share hawk90 minkione tomasmadeja sastry17 gnebbia rorobig toukir-ahammed happyfield7 privacy-security g0wzancha cracer g-ampo alexanderkruse r0metheus m-vasiloma samkopmann abdiiir lemauanhphong

id2t's Issues

Extreme performance issues with PCAPs with many packets

Issue by carlos.garcia
Friday May 11, 2018 at 17:20 GMT
Originally opened as https://git.tk.informatik.tu-darmstadt.de/SPIN/ID2T-toolkit/issues/119

ID2T is extremely slow calculating statistics when there are many packets in a PCAP. This is due to the following code in statistics.cpp

    // Increment Degrees for sender and receiver, if Sender sends its first packet to this receiver
    std::vector<std::string>::iterator found_receiver = std::find(contacted_ips[ipAddressSender].begin(), contacted_ips[ipAddressSender].end(), ipAddressReceiver);
    if(found_receiver == contacted_ips[ipAddressSender].end()){
        // Receiver is NOT contained in the List of IPs, that the Sender has contacted, therefore this is the first packet in this direction
        ip_statistics[ipAddressSender].out_degree++;
        ip_statistics[ipAddressReceiver].in_degree++;

        // Increment overall_degree only if this is the first packet for the connection (both directions)
        // Therefore check, whether Receiver has contacted Sender before
        std::vector<std::string>::iterator sender_contacted = std::find(contacted_ips[ipAddressReceiver].begin(), contacted_ips[ipAddressReceiver].end(), ipAddressSender);
        if(sender_contacted == contacted_ips[ipAddressReceiver].end()){
            ip_statistics[ipAddressSender].overall_degree++;
            ip_statistics[ipAddressReceiver].overall_degree++;
        }  

        contacted_ips[ipAddressSender].push_back(ipAddressReceiver);
    }

The complexity of that piece of code is O(n^2). The call to std::find is extremely slow.

In StatsSatabase.py [line 279], extraction of results is done incorrectly

Issue by carlos.garcia
Wednesday Nov 02, 2016 at 14:36 GMT
Originally opened as https://git.tk.informatik.tu-darmstadt.de/SPIN/ID2T-toolkit/issues/10

If I understood correctly, the intention of the extraction mechanism is to obtain the result of a query, which returns only one value, as a single variable instead of a list.

In it's current form, results of the form:
[(10,'2.6.57.13',5)]
are also processed through the "extraction" mechanism and incorrectly retrieve the value "10", dropping everything else.

This incorrect behavior is observed when, for example, entering into query mode and issuing the query: "SELECT * FROM file_statistics;"

The PortscanAttack destination is not replying with the correct TCP flags

Issue by carlos.garcia
Wednesday Nov 02, 2016 at 17:15 GMT
Originally opened as https://git.tk.informatik.tu-darmstadt.de/SPIN/ID2T-toolkit/issues/17

In a SYN port scan, it is expected that the attacker sends a SYN request, and the victim responds with a SYN|ACK reply. Currently, the victim is responding with the RST|ACK flags set instead.

The parameters used for the PortscanAttack are as follows:
-a "PortscanAttack" ip.src="66.66.66.66" mac.src="32:08:24:DC:8D:27" inject.at-timestamp=1476301843

The following is what is observed in Wireshark in the resulting pcap file:
101 2016-11-02 16:44:36.718 66.66.66.66 192.168.178.13 TCP 60 0 8542→8080 [SYN] Seq=0 Win=8192 Len=0
102 2016-11-02 16:44:36.718 192.168.178.13 66.66.66.66 TCP 60 0 8080→8542 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

ID2T does not check for available disk space

Issue by jens.klein
Sunday May 27, 2018 at 12:23 GMT
Originally opened as https://git.tk.informatik.tu-darmstadt.de/SPIN/ID2T-toolkit/issues/127

ID2T should notify the user before generating, if there is no free space on disk.

This can be achieved by assuming the resulting .pcap size based on the original .pcap size and the estimated amount of generated packets.

At the very least it should not crash.

Crash if disk fills up during writing the result pcap:
$ ./id2t -i resources/201804041400.pcap -a SMBScan -T
Input file: resources/201804041400.pcap
Label file found. Loading labels...
Read 0 label(s) successfully.
Located statistics database at: /home/pepper-jk/.cache/id2t/db/186/185/9289ccfd2840.sqlite3
Loaded file statistics in 0.00 sec from statistics database.

Creating attack instance of SMBScanAttack
Validating and adding attack parameters.
Generating attack packets... done. (total: 1000 pkts in  339.02553367614746  seconds.)

POST INJECTION STATISTICS SUMMARY  --------------------------
Total packet count:	78330145 packets
Added packet count:	1000 packets
Share of added packets:	0.0013 %
Capture duration:	0.4295 seconds
------------------------------------------------------------
Merging base pcap with single attack pcap... Could not serialize base packet with timestamp 1522818221.90773
Could not serialize base packet with timestamp 1522818300.36396
Could not serialize base packet with timestamp 1522818389.45844
Could not serialize base packet with timestamp 1522818389.65043
Could not serialize base packet with timestamp 1522818467.90078
Could not serialize base packet with timestamp 1522818497.87013
Could not serialize base packet with timestamp 1522818565.43478
Could not serialize base packet with timestamp 1522818565.72621
Could not serialize base packet with timestamp 1522818565.72621
Could not serialize base packet with timestamp 1522818565.72626
done.
Deleting intermediate attack pcap... done.
Traceback (most recent call last):
  File "/home/pepper-jk/code/ID2T-toolkit/code/CLI.py", line 189, in <module>
    main(sys.argv[1:])
  File "/home/pepper-jk/code/ID2T-toolkit/code/CLI.py", line 184, in main
    cli.parse_arguments(args)
  File "/home/pepper-jk/code/ID2T-toolkit/code/CLI.py", line 84, in parse_arguments
    self.process_arguments()
  File "/home/pepper-jk/code/ID2T-toolkit/code/CLI.py", line 95, in process_arguments
    self.process_pcap()
  File "/home/pepper-jk/code/ID2T-toolkit/code/CLI.py", line 166, in process_pcap
    controller.process_attacks(self.args.attack, self.args.rngSeed, self.args.time, self.args.inject_empty)
  File "/home/pepper-jk/code/ID2T-toolkit/code/Core/Controller.py", line 158, in process_attacks
    self.label_manager.write_label_file(self.pcap_dest_path)
  File "/home/pepper-jk/code/ID2T-toolkit/code/Core/LabelManager.py", line 171, in write_label_file
    file.close()
OSError: [Errno 28] No space left on device

Crash on start with no free disk space left:

$ ./id2t -i resources/201804041400.pcap -T -a SMBScan target.count=30000 hosting.percentage=0.02
Input file: resources/201804041400.pcap
Label file found. Loading labels...
Read 0 label(s) successfully.
Located statistics database at:  /home/pepper-jk/.cache/id2t/db/186/185/9289ccfd2840.sqlite3
Loaded file statistics in 0.00 sec from statistics database.

Creating attack instance of SMBScanAttack
Traceback (most recent call last):
  File "/home/pepper-jk/code/ID2T-toolkit/code/CLI.py", line 189, in <module>
    main(sys.argv[1:])
  File "/home/pepper-jk/code/ID2T-toolkit/code/CLI.py", line 184, in main
    cli.parse_arguments(args)
  File "/home/pepper-jk/code/ID2T-toolkit/code/CLI.py", line 84, in parse_arguments
    self.process_arguments()
  File "/home/pepper-jk/code/ID2T-toolkit/code/CLI.py", line 95, in process_arguments
    self.process_pcap()
  File "/home/pepper-jk/code/ID2T-toolkit/code/CLI.py", line 166, in process_pcap
    controller.process_attacks(self.args.attack, self.args.rngSeed, self.args.time, self.args.inject_empty)
  File "/home/pepper-jk/code/ID2T-toolkit/code/Core/Controller.py", line 89, in process_attacks
    temp_attack_pcap, duration = self.attack_controller.process_attack(attack[0], attack[1:], time)
  File "/home/pepper-jk/code/ID2T-toolkit/code/Core/AttackController.py", line 139, in process_attack
    self.create_attack(attack, self.seed)
  File "/home/pepper-jk/code/ID2T-toolkit/code/Core/AttackController.py", line 116, in create_attack
    self.current_attack.set_statistics(self.statistics)
  File "/home/pepper-jk/code/ID2T-toolkit/code/Attack/BaseAttack.py", line 79, in set_statistics
    self.most_used_ttl_value = self.statistics.get_most_used_ttl_value()
  File "/home/pepper-jk/code/ID2T-toolkit/code/Core/Statistics.py", line 619, in get_most_used_ttl_value
    return self.process_db_query("SELECT ttlValue FROM (SELECT ttlValue, SUM(ttlCount) as occ FROM ip_ttl GROUP BY "
  File "/home/pepper-jk/code/ID2T-toolkit/code/Core/Statistics.py", line 803, in process_db_query
    return self.stats_db.process_db_query(query_string_in, print_results)
  File "/home/pepper-jk/code/ID2T-toolkit/code/Core/StatsDatabase.py", line 329, in process_db_query
    result = self.process_user_defined_query(query_string, sql_query_parameters)
  File "/home/pepper-jk/code/ID2T-toolkit/code/Core/StatsDatabase.py", line 137, in process_user_defined_query
    self.cursor.execute(query_string)
sqlite3.OperationalError: database or disk is full

pip installation fails when offline

Issue by jens.klein
Saturday May 26, 2018 at 18:07 GMT
Originally opened as https://git.tk.informatik.tu-darmstadt.de/SPIN/ID2T-toolkit/issues/126

The installation of pip dependencies fails while offline. This is due to pip requesting hashes for the packages in cache before installing, this requires an internet connection.

$ ./build.sh 
Updating SQLiteCpp
Detected OS: Arch Linux
Packages: Checking...
Packages: Found.
Additional Packages: Checking...
Additional Packages: Found.
Collecting coverage==4.5.1 (from -r resources/requirements.txt (line 1))
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fcb1a8ec160>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/coverage/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fcb1a8ec4a8>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/coverage/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fcb1a8ecba8>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/coverage/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fcb1a8ec0f0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/coverage/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fcb1a8ec128>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)': /simple/coverage/
  Could not find a version that satisfies the requirement coverage==4.5.1 (from -r resources/requirements.txt (line 1)) (from versions: )
No matching distribution found for coverage==4.5.1 (from -r resources/requirements.txt (line 1))
-- The C compiler identification is GNU 8.1.0
-- The CXX compiler identification is GNU 8.1.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
[...]

This results in the python-venv not being setup probably and ID2T to crash on start, because of the missing dependencies:

$ ./id2t -i resources/201804041400.pcap -a SMBScan-T                   
Traceback (most recent call last):
  File "/home/pepper-jk/code/ID2T-toolkit/code/CLI.py", line 5, in <module>
    from Core.Controller import Controller
  File "/home/pepper-jk/code/ID2T-toolkit/code/Core/Controller.py", line 8, in <module>
    import pyparsing as pp
ModuleNotFoundError: No module named 'pyparsing'

Port scans are not being inserted into the target in correct time sequence

Issue by carlos.garcia
Thursday Nov 03, 2016 at 13:32 GMT
Originally opened as https://git.tk.informatik.tu-darmstadt.de/SPIN/ID2T-toolkit/issues/21

When instructing to inject a PortscanAttack, the first injected packets are correctly placed at the timestamp specified by "inject.at-timestamp" but subsequent packets are wrongly placed using the current timestamp.

To test this problem, execute the command:
./CLI.py -i test_me_short.pcap -a "PortscanAttack" ip.src="66.66.66.66" mac.src="32:08:24:DC:8D:27" inject.at-timestamp=921506778

on the attached file. (cannot attach files right now, will attach later)

After loading a label's file, the application crashes

Issue by carlos.garcia
Wednesday Nov 02, 2016 at 17:04 GMT
Originally opened as https://git.tk.informatik.tu-darmstadt.de/SPIN/ID2T-toolkit/issues/16

The crash message is:

Input file: test_me_short_20161102-164436.pcap
Label file found. Loading labels...
Traceback (most recent call last):
  File "./CLI.py", line 113, in <module>
    main(sys.argv[1:])
  File "./CLI.py", line 108, in main
    cli.parse_arguments(args)
  File "./CLI.py", line 97, in parse_arguments
    self.process_arguments()
  File "./CLI.py", line 32, in process_arguments
    controller = Controller(self.args.input)
  File "/home/boy/Documents/CASED/Repos/id2t/code/ID2TLib/Controller.py", line 23, in __init__
    self.label_manager = LabelManager(self.pcap_src_path)
  File "/home/boy/Documents/CASED/Repos/id2t/code/ID2TLib/LabelManager.py", line 34, in __init__
    self._load_labels()
  File "/home/boy/Documents/CASED/Repos/id2t/code/ID2TLib/LabelManager.py", line 136, in _load_labels
    attack_note = a.childNodes[3].firstChild.data
AttributeError: 'NoneType' object has no attribute 'data'

There are no notes in the XML file, is this what is not correctly parsed?

Location of saved databases not adequate

Issue by carlos.garcia
Thursday Mar 15, 2018 at 14:55 GMT
Originally opened as https://git.tk.informatik.tu-darmstadt.de/SPIN/ID2T-toolkit/issues/71

ID2T is saving all cached statistics (the databases) in the folder ~/ID2T_DATA. This folder clutters the home directory of the user.

I see two ways to resolve this issue:

  • Save the databases in an XDG standard location such as ~/.config/id2t
  • Save it in the same directory as where the pcap is found

We can discuss this next time we meet.

Default parameters are always generated

Issue by jens.klein
Thursday May 24, 2018 at 13:01 GMT
Originally opened as https://git.tk.informatik.tu-darmstadt.de/SPIN/ID2T-toolkit/issues/125

When running an Attack ID2T will generate the default parameters before taking the user defined ones into account.

This results in unnecessary overhead.

One way to tackle this issue could be init_params() getting either the user defined parameters or flags representing them as an option. So that it only generates the needed parameters, which would be the wanted behavior.

Add version field to SQLite statistics database

Issue by patrick.jattke
Saturday Nov 05, 2016 at 10:33 GMT
Originally opened as https://git.tk.informatik.tu-darmstadt.de/SPIN/ID2T-toolkit/issues/24

The program should store a version number into the statistics database such that changes in the statistics calculation (e.g., extension of available statistics) or the scheme if the statistics database (e.g., renaming of columns, new tables) leads to an automatic recalculation. At the moment the user must force the recalculation by providing -r/--recalculate as program argument.

Improve query parsing by using an existing library

Issue by patrick.jattke
Friday Nov 04, 2016 at 10:51 GMT
Originally opened as https://git.tk.informatik.tu-darmstadt.de/SPIN/ID2T-toolkit/issues/23

The parsing process of queries is not robust and providing queries not recognized may lead to errors or unexpressive error messages.
For example, submitting the query most_used(macAddr); prints the error message An error occurred: near "None": syntax error.

Therefore I suggest re-implementing the query parsing functionality in StatsDatabase.process_db_query and StatsDatabase._process_named_query. This can be realized by using an existing library such as pyparsing.

Incorrect capture duration reported

Issue by carlos.garcia
Friday May 11, 2018 at 16:40 GMT
Originally opened as https://git.tk.informatik.tu-darmstadt.de/SPIN/ID2T-toolkit/issues/118

After finishing analyzing and storing the statistics of the pcap file in 1, the reported duration is wrong. The reported statistics are:

PCAP FILE STATISTICS SUMMARY  ------------------------------
Total packet count:     78329145 packets
Recognized packets:     78329145 packets
Unrecognized packets:   0 PDUs
% Recognized packets:   100.0 %
% Unrecognized packets: 0.0 %
Last unknown PDU:       None
Capture duration:       0.4295 seconds
------------------------------------------------------------

Examining the database in query mode, we can see:

SELECT timestampLastPacket FROM file_statistics;
Query returned 1 record:

+----------------------------+
| timestampLastPacket        |
+----------------------------+
| 2018-04-04 07:15:00.331786 |
+----------------------------+
> SELECT timestampFirstPacket FROM file_statistics;
Query returned 1 record:

+----------------------------+
| timestampFirstPacket       |
+----------------------------+
| 2018-04-04 07:00:00.448274 |
+----------------------------+

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.