Minimal nDPI Protocols about go-dpi

[nikofil]

Hello Warren,
It should certainly be possible to add new protocols. Disabling the other layers so that only nDPI is used might help.
Could you share the changes you made?
Cheers,
Nikos

[WarrenWillingham]

Oh man, was waiting for a notification, but you responded so quickly that I entirely missed it.
I have tried a bunch of different things but I have gone back to the most basic modifications that I could possibly make, so that I can be pointed in the right direction as easy as possible.

I appreciate your help!

Github response

My setup

• I used this pcap from nDPI: https://github.com/ntop/nDPI/blob/dev/tests/pcap/bittorrent.pcap
• I used your example_app.go with a few changes ot other files in the go-dpi package

I am running manjaro linux if that makes any difference at all

My modifications

I went through a bunch of different attempts, but this is the one that seemed to be closest to correct.
For an earlier attempt I was initializing the wrapper in the example, but even without that, my debugger shows it as still using 'ClassifyFlow()' under the nDPI_wrapper.go

nDPI_wrapper.go

added bittorrent line

var ndpiCodeToProtocol = map[uint32]types.Protocol{
	7:   types.HTTP,    // NDPI_PROTOCOL_HTTP
	5:   types.DNS,     // NDPI_PROTOCOL_DNS
	92:  types.SSH,     // NDPI_PROTOCOL_SSH
	127: types.RPC,     // NDPI_PROTOCOL_DCERPC
	3:   types.SMTP,    // NDPI_PROTOCOL_MAIL_SMTP
	88:  types.RDP,     // NDPI_PROTOCOL_RDP
	16:  types.SMB,     // NDPI_PROTOCOL_SMB
	81:  types.ICMP,    // NDPI_PROTOCOL_IP_ICMP
	1:   types.FTP,     // NDPI_PROTOCOL_FTP_CONTROL
	91:  types.SSL,     // NDPI_PROTOCOL_SSL
	64:  types.SSL,     // NDPI_PROTOCOL_SSL_NO_CERT
	10:  types.NetBIOS, // NDPI_PROTOCOL_NETBIOS
	67:  types.JABBER,  // NDPI_PROTOCOL_UNENCRYPTED_JABBER
	222: types.MQTT,    // NDPI_PROTOCOL_MQTT
	37:  types.BITTORRENT,
}

protocols.go

added bittorrent line

const (
	HTTP       Protocol = "HTTP"
	DNS        Protocol = "DNS"
	SSH        Protocol = "SSH"
	RPC        Protocol = "RPC"
	SMTP       Protocol = "SMTP"
	RDP        Protocol = "RDP"
	SMB        Protocol = "SMB"
	ICMP       Protocol = "ICMP"
	FTP        Protocol = "FTP"
	SSL        Protocol = "SSL"
	NetBIOS    Protocol = "NetBIOS"
	JABBER     Protocol = "JABBER"
	MQTT       Protocol = "MQTT"
	Unknown    Protocol = ""
	BITTORRENT Protocol = "BITTORRENT"
)

Output I'm getting

From my modified go-dpi

Packet #1: Could not identify (new flow)
Packet #2: Could not identify
Packet #3: Could not identify (new flow)
Packet #4: Could not identify
Packet #5: Could not identify
Packet #6: Could not identify
Packet #7: Could not identify
Packet #8: Could not identify (new flow)
........etc...etc.etc........ 

Lots more "could not identify" lines, didn't identify any

From nDPI

nDPI example program's output for that example pcap is:

[user@manjaro example]$ ./ndpiReader -c 1 -i /home/user/Downloads/bittorrent.pcap

-----------------------------------------------------------
* NOTE: This is demo app to show *some* nDPI features.
* In this demo we have implemented only some basic features
* just to show you what you can do with the library. Feel 
* free to extend it and send us the patches for inclusion
------------------------------------------------------------

Using nDPI (3.2.0-2218-60e3ac6b) [1 thread(s)]
Reading packets from pcap file /home/user/Downloads/bittorrent.pcap...
Unable to open file 1 [No such file or directory]
Running thread 0...

nDPI Memory statistics:
        nDPI Memory (once):      208.13 KB    
        Flow Memory (per flow):  2.20 KB      
        Actual Memory:           3.75 MB      
        Peak Memory:             3.75 MB      
        Setup Time:              10 msec
        Packet Processing Time:  0 msec

Traffic statistics:
        Ethernet bytes:        312904        (includes ethernet CRC/IFC/trailer)
        Discarded bytes:       0            
        IP packets:            299           of 299 packets total
        IP bytes:              305728        (avg pkt size 1022 bytes)
        Unique flows:          24           
        TCP Packets:           299          
        UDP Packets:           0            
        VLAN Packets:          0            
        MPLS Packets:          0            
        PPPoE Packets:         0            
        Fragmented Packets:    0            
        Max Packet size:       1472         
        Packet Len < 64:       21           
        Packet Len 64-128:     36           
        Packet Len 128-256:    35           
        Packet Len 256-1024:   21           
        Packet Len 1024-1500:  186          
        Packet Len > 1500:     0            
        nDPI throughput:       437.77 K pps / 3.41 Gb/sec
        Analysis begin:        14/Feb/2016 12:12:47
        Analysis end:          14/Feb/2016 12:13:02
        Traffic throughput:    20.12 pps / 164.51 Kb/sec
        Traffic duration:      14.859 sec
        Guessed flow protos:   0            


Detected protocols:
        BitTorrent           packets: 299           bytes: 305728        flows: 24           


Protocol statistics:
        Unsafe                      305728 bytes

If you need any more information from me, please just let me know (I actually have notifications on now)

Any help is appreciated, thanks so much!

[WarrenWillingham]

Also I am unsure of how to "disable other layers"
Could you point me to any documentation or if you have an example somewhere?
Couldn't seem to find anything about it in the wikis

[nikofil]

I added a wiki page for how to use only specific layers at https://github.com/mushorg/go-dpi/wiki/Using-only-specific-layers.
About the bittorrent protocol, what you've done is correct. I can confirm that ndpiReader also works for me but calling the library myself returns 0 (unknown), which I'm looking into.

[WarrenWillingham]

Thanks so much for looking into this for me.

And awesome, I figured the layer disabling / module selection out Friday night by just scrolling through your comments(which actually helped a TON, thank you for being descriptive) until I had a lead. The wiki page is excellent, thank you.

And ahh, I'm feeling better that it is doing the same for you as well. Also thank you for confirming that what I did was correct.
I did get that bittorrent example from their nDPI dev branch, I will now test the example pcap's from their 2.0-stable branch to try to figure out what the issue may be
I am having issues with a pcap of my own youtube traffic as well, and a few others. It is hit and miss with protocols, but i'm still investigating.
Thank you so much again.

[nikofil]

Well, the solution was simpler than I expected. Apparently BitTorrent is considered an app protocol, while the code I was using only considered "master" protocols (still not quite sure on the distinction, see ntop/nDPI#500).
I'm pushing a PR that should fix this now, I've also went ahead and added BitTorrent for you.