Document the API for using go-dpi as a library: What are the input formats? What are the output formats? We probably have to put some thought into this as we shouldn't change it after a certain point.

• Main lib file that runs the classifiers
• Directory of different classifiers for each protocol
• Readme, License, Makefile, glide.yaml
• Example app

There are a lot of *gopacket.Packet where it seems just using the bare interface would be simpler and just as effective as a pointer to an interface. The documentation specifically refers to gopacket.Packet as a structure instead of an interface, so I'm wondering if there's a misunderstanding that lead to the use of pointers instead.

#github.com/mushorg/go-dpi/modules/wrappers
....\pkg\mod\github.com\mushorg\[email protected]\modules\wrappers\nDPI_wrapper.go:61:24: could not determine kind of name for C.__time_t
cgo:
gcc errors for preamble:
In file included from C:/msys64/mingw64/include/ndpi_main.h:32,
from ./nDPI_wrapper_impl.h:3,
from ....\pkg\mod\github.com\mushorg\[email protected]\modules\wrappers\nDPI_wrapper.go:6:
C:/msys64/mingw64/include/ndpi_define.h:362:1: error: stray '@' in program
362 | @HANDLE_TLS_SIGS@#define TLS_HANDLE_SIGNATURE_ALGORITMS 1
| ^
C:/msys64/mingw64/include/ndpi_define.h:362:17: error: stray '@' in program
362 | @HANDLE_TLS_SIGS@#define TLS_HANDLE_SIGNATURE_ALGORITMS 1
| ^
C:/msys64/mingw64/include/ndpi_define.h:362:18: error: stray '#' in program
362 | @HANDLE_TLS_SIGS@#define TLS_HANDLE_SIGNATURE_ALGORITMS 1
| ^
C:/msys64/mingw64/include/ndpi_define.h:362:2: error: unknown type name 'HANDLE_TLS_SIGS'
362 | @HANDLE_TLS_SIGS@#define TLS_HANDLE_SIGNATURE_ALGORITMS 1
| ^~~~~~~~~~~~~~~
C:/msys64/mingw64/include/ndpi_define.h:362:26: error: expected '=', ',', ';', 'asm' or 'attribute' before 'TLS_HANDLE_SIGNATURE_ALGORITMS'
362 | @HANDLE_TLS_SIGS@#define TLS_HANDLE_SIGNATURE_ALGORITMS 1
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Hi Team,

I'm unable to build for windows platform.

Would you please help me to resolve the issue?

MacBook-Pro:go-dpi $ sudo GOOS=windows go build -v godpi_example/example_app.go
github.com/mushorg/go-dpi/modules/ml
go build github.com/mushorg/go-dpi/modules/ml: build constraints exclude all Go files in /Users/rajesh/Projects/go-dpi/modules/ml
github.com/mushorg/go-dpi/modules/wrappers
#github.com/mushorg/go-dpi/modules/wrappers
modules/wrappers/wrappers.go:51:3: undefined: NewLPIWrapper
modules/wrappers/wrappers.go:52:3: undefined: NewNDPIWrapper
MacBook-Pro:go-dpi $

Best Regards,
Rajesh~
iRxTx.com
+91-9600635770

We currently run the tests on travis both inside and outside the container, doubling the running time on travis.

Have a header file with defines for enabling each wrapper

I was analyzing the implementation of the GetFlowForPacket function and noticed that it only uses the source and destination port to identify flows.

Is there any reason to base the identification exclusively on these ports? Why can't the destination and source IP be used as a way to complement this process?

If we have two different pairs of machines using the same pair of ports to communicate, the current implementation will see these communications as a single flow and not two independent flows.

Probably the same as #57

Hello, When using nDPI as a wrapper, I am still only able to detect the same protocols as I can without the wrapper enabled. When using nDPI itself, I can find other protocols like Bittorrent and Youtube for example.
I have followed the wiki(hopefully correctly) on how to set up the mapping from nDPI to go-dpi still to no avail. I am wondering if I am making a mistake in my code or if I am misunderstanding the nDPI wrapper.

Is go-dpi able to receive all of the different protocols that nDPI supports, or is it just using nDPI to classify the same base protocols.

I apologize in advance if I am misusing any terms or being unclear. I would be more than happy to clarify or provide more information as needed.

Thanks so much!

Once a central API is created for using all layers

Hellp， when i run example_app.go. at GoLand occurred error,
how to fix this problem. i need installed ndpi?
env: MacOS 13.0.1 M1
go:1.19.2

github.com/mushorg/go-dpi/modules/ml
../modules/ml/linearsvc.go:18:11: fatal error: 'linear.h' file not found
#include <linear.h>
^~~~~~~~~~
1 error generated.
github.com/mushorg/go-dpi/modules/wrappers
In file included from ../modules/wrappers/nDPI_wrapper.go:6:
./nDPI_wrapper_impl.h:3:10: fatal error: 'ndpi/ndpi_main.h' file not found
#include <ndpi/ndpi_main.h>
^~~~~~~~~~~~~~~~~~
1 error generated.

Is there any way to find source IP in logs?

Is there any way to measure throughput for each protocols in go-dpi?
Thank you 🙂

domain wand.net.nz is down .. so i cant get packages of even neat support

I follow https://github.com/mushorg/go-dpi/wiki/Installation-guide

Step1 :
sudo apt-get install golang
#check version

go version go1.10.4 linux/amd64

Step2:
sudo apt-get install liblinear3 liblinear-dev

#check installation : liblinear3

• py@py:~$ apt list liblinear3
• Listing... Done
• liblinear3/bionic 2.1.0+dfsg-2 amd64

#check installation: liblinear-dev

• py@py:~$ apt list liblinear-dev
• Listing... Done
• liblinear-dev/bionic 2.1.0+dfsg-2 amd64

[Optional Part]

py@py:~$ echo "deb http://packages.wand.net.nz trusty main" | sudo tee -a /etc/apt/sources.list
[sudo] password for py: 
deb http://packages.wand.net.nz trusty main
py@py:~$ sudo apt-get update
Hit:1 http://tw.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://tw.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:3 http://tw.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 
Get:4 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]    
Get:5 http://tw.archive.ubuntu.com/ubuntu bionic-updates/main amd64 DEP-11 Metadata [294 kB]
Get:6 http://tw.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 DEP-11 Metadata [288 kB]
Get:7 http://tw.archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 DEP-11 Metadata [2468 B]
Get:8 http://tw.archive.ubuntu.com/ubuntu bionic-backports/universe amd64 DEP-11 Metadata [9292 B]
Get:9 http://security.ubuntu.com/ubuntu bionic-security/main amd64 DEP-11 Metadata [48.9 kB]
Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 DEP-11 Metadata [59.5 kB]
Get:11 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 DEP-11 Metadata [2460 B]
Get:12 http://packages.wand.net.nz trusty InRelease [3545 B]                   
Err:12 http://packages.wand.net.nz trusty InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 69A507877C4B94E8
Reading package lists... Done
W: GPG error: http://packages.wand.net.nz trusty InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 69A507877C4B94E8
E: The repository 'http://packages.wand.net.nz trusty InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

apt-get : Err

py@py:~$ sudo apt-get -y --force-yes install git gcc autoconf automake libtool libpcap-dev libtrace4 libtrace4-dev libprotoident libprotoident-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
W: --force-yes is deprecated, use one of the options starting with --allow instead.
E: Unable to locate package libtrace4
E: Unable to locate package libtrace4-dev
E: Unable to locate package libprotoident
E: Unable to locate package libprotoident-dev

Unable to locate package...
another :
sudo apt-get install autoconf

py@py:~$ git clone --branch 3.2-stable https://github.com/ntop/nDPI/ /tmp/nDPI
Cloning into '/tmp/nDPI'...
remote: Enumerating objects: 111, done.
remote: Counting objects: 100% (111/111), done.
remote: Compressing objects: 100% (100/100), done.
remote: Total 22437 (delta 38), reused 30 (delta 10), pack-reused 22326
Receiving objects: 100% (22437/22437), 110.44 MiB | 213.00 KiB/s, done.
Resolving deltas: 100% (15624/15624), done.
Checking out files: 100% (527/527), done.
py@py:~$ cd /tmp/nDPI/
py@py:/tmp/nDPI$ ls
autogen.sh    configure.seed   COPYING  example  INSTALL        m4           packages  README.md    README.protocols  tests  wireshark
CHANGELOG.md  CONTRIBUTING.md  doc      fuzz     libndpi.pc.in  Makefile.am  python    README.nDPI  src               utils

py@py:/tmp/nDPI$ ./autogen.sh && ./configure && make && sudo make install && cd -
libtool and libtoolize is missing: please install it and try again
bash: ./configure: No such file or directory
py@py:/tmp/nDPI$ ls
autogen.sh    configure.seed   COPYING  example  INSTALL        m4           packages  README.md    README.protocols  tests  wireshark
CHANGELOG.md  CONTRIBUTING.md  doc      fuzz     libndpi.pc.in  Makefile.am  python    README.nDPI  src               utils

no configure file
and i don't know what cd - means ?

root@py:/tmp/nDPI# go get github.com/mushorg/go-dpi
# github.com/mushorg/go-dpi/modules/ml
/root/go/src/github.com/mushorg/go-dpi/modules/ml/linearsvc.go:17:11: fatal error: linear.h: No such file or directory
 // #include <linear.h>
           ^~~~~~~~~~
compilation terminated.
# github.com/mushorg/go-dpi/modules/wrappers
In file included from /root/go/src/github.com/mushorg/go-dpi/modules/wrappers/nDPI_wrapper.go:6:0:
./nDPI_wrapper_impl.h:3:10: fatal error: ndpi/ndpi_main.h: No such file or directory
 #include <ndpi/ndpi_main.h>
          ^~~~~~~~~~~~~~~~~~
compilation terminated.

#github.com/mushorg/go-dpi/modules/wrappers
LPI_wrapper_impl.cpp:8:10: fatal error: libprotoident.h: No such file or directory
#include <libprotoident.h>
^~~~~~~~~~~~~~~~~
compilation terminated.

I was working on a MySQL classifier and was using a pcap from here. Since the ICMP classifier comes before the MySQL classifier in the classifierList, it gets detected before MySQL because there are packets which satisfy the conditions in the classifier. It never actually gets to the MySQL classifier since it breaks off after it has "identified" a protocol.

What approach should be taken to resolve this?

This issue happened sometimes (I had delay to display logs) .In my test just detected ICMP protocol. I had this log Packet #number: Could not identify for other protocols.

In order to avoid circular dependencies down the road

I want to add new protocol. I done according to installation but I don't know how to build new change in do-dpi. I used this command go test . ./modules/wrappers ./modules/classifiers ./types ./utils but it didn't work for me because I changed string Could not identify but it didn't changed.

Thanks

Need to write better classifiers for:

• NetBIOS
• RDP
• RPC
• SMB

Maybe add:

• RTP
• NTP
• MTP
• SIP
• IRC

I have a question about setup Go environment (according WiKi: Installation guide)
Error message is 'Unable to locate package golang-1.7-go'

Thank you for supporting a really useful module/library.

I would like to check if there is a specific reason that access to the cache (var flowTracker *cache.Cache) is not possible from a client module.

I would like to reference the flow cache from my module and it looks easy enough to do(will require a small change to go-dpi on my side) but I don't want to get caught out with a valid reason not to do it.

The intent is read-only and it will be to "list" a current set of known and identified flows.

Add a short description of the project

../modules/ml/linearsvc.go:18:21: fatal error: linear.h: No such file or directory
// #include <linear.h>
^
compilation terminated.
#github.com/mushorg/go-dpi/modules/wrappers
In file included from ../modules/wrappers/nDPI_wrapper.go:6:0:
./nDPI_wrapper_impl.h:3:28: fatal error: ndpi/ndpi_main.h: No such file or directory
#include <ndpi/ndpi_main.h>
^
compilation terminated.

github.com/mushorg/go-dpi/modules/wrappers

nDPI_wrapper_impl.c: In function 'ndpiInitialize':
nDPI_wrapper_impl.c:71:20: warning: implicit declaration of function 'ndpi_detection_get_sizeof_ndpi_id_struct'; did you mean 'ndpi_detection_get_sizeof_ndpi_flow_struct'? [-Wimplicit-function-declaration]
71 | size_id_struct = ndpi_detection_get_sizeof_ndpi_id_struct();
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| ndpi_detection_get_sizeof_ndpi_flow_struct
nDPI_wrapper_impl.c: In function 'packet_processing':
nDPI_wrapper_impl.c:210:30: error: too many arguments to function 'ndpi_detection_process_packet'
210 | ndpi_protocol detected = ndpi_detection_process_packet(ndpi_struct, ndpi_flow, (uint8_t *) iph, ipsize, time, src, dst);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from C:/msys64/mingw64/include/ndpi_main.h:35,
from nDPI_wrapper_impl.h:3,
from nDPI_wrapper_impl.c:5:
C:/msys64/mingw64/include/ndpi_api.h:326:17: note: declared here
326 | ndpi_protocol ndpi_detection_process_packet(struct ndpi_detection_module_struct *ndpi_struct,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I have this error many times.

Number of packets: 1336
Number of packets identified: 1053
Protocols identified:
map[DNS:6 SSL:1047]
panic: runtime error: slice bounds out of range
goroutine 1 [running]:
github.com/mushorg/go-dpi/vendor/github.com/google/gopacket/layers.(*DNSQuestion).decode(0xc422134ab8, 0xc4213981ea, 0x10, 0x10, 0xc, 0xb75dc0, 0xc4215c8de0, 0xc422134c08, 0xc422134a90, 0x64e1f6e1, ...)
/root/go/src/github.com/mushorg/go-dpi/vendor/github.com/google/gopacket/layers/dns.go:617 +0x1af
github.com/mushorg/go-dpi/vendor/github.com/google/gopacket/layers.(*DNS).DecodeFromBytes(0xc421189b60, 0xc4213981ea, 0x10, 0x10, 0xb75dc0, 0xc4215c8de0, 0xc42002f880, 0x7f681d2bd000)
/root/go/src/github.com/mushorg/go-dpi/vendor/github.com/google/gopacket/layers/dns.go:328 +0x292
github.com/mushorg/go-dpi/modules/classifiers.DNSClassifier.HeuristicClassify.func1(0xb7ab00, 0xc4211d9700, 0xb7ab00)
/root/go/src/github.com/mushorg/go-dpi/modules/classifiers/dns.go:17 +0xa5
github.com/mushorg/go-dpi/modules/classifiers.checkFlowLayer(0xc421164410, 0x2d, 0x7a0ae8, 0x41908d)
/root/go/src/github.com/mushorg/go-dpi/modules/classifiers/classifiers.go:110 +0xa5
github.com/mushorg/go-dpi/modules/classifiers.DNSClassifier.HeuristicClassify(0xc421164410, 0x7f681d230598)
/root/go/src/github.com/mushorg/go-dpi/modules/classifiers/dns.go:14 +0x43
github.com/mushorg/go-dpi/modules/classifiers.(*DNSClassifier).HeuristicClassify(0x10f1238, 0xc421164410, 0x10f1238)
:1 +0x3e
github.com/mushorg/go-dpi/modules/classifiers.(*ClassifierModule).ClassifyFlow(0xc42112aa80, 0xc421164410, 0xc422134af8, 0xc422134d80, 0x696115, 0xb7f140)
/root/go/src/github.com/mushorg/go-dpi/modules/classifiers/classifiers.go:78 +0xb2
github.com/mushorg/go-dpi.ClassifyFlow(0xc421164410, 0xc421173680, 0xc421164410, 0x1, 0x1)
/root/go/src/github.com/mushorg/go-dpi/godpi.go:94 +0x79
main.main()
/root/go/src/github.com/mushorg/go-dpi/godpi_example/example_app.go:75 +0x524

and sometimes I got this error.

Get https://raw.githubusercontent.com/wiki/mushorg/go-dpi/2grams_tcp.model: dial tcp: lookup raw.githubusercontent.com on 192.168.1.1:53: read udp 192.168.1.2:35056->192.168.1.1:53: i/o timeout
Get https://raw.githubusercontent.com/wiki/mushorg/go-dpi/2grams_tcp.model: dial tcp: lookup raw.githubusercontent.com on 192.168.1.1:53: read udp 192.168.1.2:37833->192.168.1.1:53: i/o timeout