ntop / ndpi Goto Github PK

Add nDPI support for TIM_WiFi application. Trace sent in a separate email.

un cliente mi ha segnalato che la sua connessione con la casa madre viene rilevata come bittorrent ma è una connessione Teredo.

i pcap li trovi in dropbox:
Dropbox/Wurth/baddpi-bittorrent-teredo.pcap.gz
Dropbox/Wurth/baddpi-bittorrent-teredo-small.pcap

Add nDPI support for TIM Beta Application, a chargeable application that allows the subscriber to pay accordingly their use of Voice, Internet and SMS service. The connection can be done via Facebook login or using a password that need to be generated via BETA site. Trace will be sent separately

Add nDPI support for TIM Menu Application, which gives subscribers access all Mobile Operator's apps and services. Trace will be sent separately

To work with >10Gps traffic I want to share the traffic detection in different threads. Shall I use one detection module for all threads or own detection module for each thread?

Add nDPI support for Recarga Application, an Application to recharge mobile credits

Hi,

I'm downloading some data using ftp and it hasn't detected how many bytes i've downloaded. Otherwise i can see it has been detected as Unknown protocol. If you need anything else please let me know.

Hello, folks!

Thanks for your brilliant work!

I have run nDPI on Network with 500 kpps load (per packet analitics mode) and expect significant performance issues, I can't achieve more than 250 kpps. Thus, I have tried to debug it but have some troubles with debugging symbols.

In perf top I saw:

Samples: 775K of event 'cycles', Event count (approx.): 49347133671
  19.00%  libc-2.19.so              [.] __memcmp_sse4_1
   4.31%  libc-2.19.so              [.] memset
   3.20%  libndpi.so.1.0.0          [.] 0x00000000000089ca
   3.20%  libndpi.so.1.0.0          [.] 0x00000000000089b0
   2.98%  libndpi.so.1.0.0          [.] 0x00000000000089c5
   2.58%  libndpi.so.1.0.0          [.] memcmp@plt
   1.81%  libndpi.so.1.0.0          [.] 0x00000000000089bf
   1.79%  libndpi.so.1.0.0          [.] 0x0000000000009d88
   1.75%  libc-2.19.so              [.] _int_free
   1.44%  libndpi.so.1.0.0          [.] ndpi_detection_process_packet
   1.39%  libc-2.19.so              [.] malloc
   1.34%  libndpi.so.1.0.0          [.] 0x0000000000009d98
   1.08%  libc-2.19.so              [.] _int_malloc
   0.96%  libndpi.so.1.0.0          [.] 0x000000000000a404
   0.90%  ndpicallback.so           [.] fastnetmon_parse_pkt

But my binary is not stripped:

file /opt/ndpi/lib/libndpi.so.1.0.0 
/opt/ndpi/lib/libndpi.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=3fbf0ef7daee47da17d55984d3184baf414340a2, not stripped

And have all important symbols:

nm /opt/ndpi/lib/libndpi.so.1.0.0 |grep process
000000000000fc10 T ndpi_detection_process_packet
0000000000007400 t ndpi_patricia_process

I have installed my nDPI binary with following reference:

cd /usr/src
git clone https://github.com/ntop/nDPI.git
cd nDPI/
./autogen.sh
./configure --prefix=/opt/ndpi
make install
echo "/opt/ndpi/lib" >  /etc/ld.so.conf.d/ndpi.conf
ldconfig

Maybe you could help me with this "unidentified" functions?

Currently nDPI is a monolithic library. Instead it would be desirable to split nDPI into engine + plugins where protocol dissectors are loaded at runtime (e.g. via shared libraries)

Add nDPI support for Som de Chamada Application, allowing Mobile Operator subscribers to personalize the ringtone with different tones and music. Trace will be sent separately.

Add nDPI support for Deezer Application, a Music Streaming service. Trace will be sent separately.

Add https://www.hsselite.com protocol dissector

Add nDPI support for MEU Application, allowing Mobile Operator's subscribers have access to all information regarding their account, bills, credits, activate or deactivate promotions, etc. Trace will be sent separately

There is application called Popcorn Time. It is very popular in our county (Serbia) and I believe in others too. Because of the local laws you can watch movies online but it is prohibited to download them, it is against the law. This app do exactely that. Popcorn Time streams movies, popular series and TV shows from torrents. So, this application is widely used. It can be used on Windows, Andoids etc. Popcorn Time uses, among others protocol, Bittorrent client for temporary downloading content (other P2P users downloads from you at the same time). And that is a problem. nDPI recognizes Popcorn Time as Bittorrent so it can not be properly prioritized. Most users wants it with higher priority while downloading other (P2P) stuff in backgroung. Can you make filter (protocol) for Popcorn Time? Aplication can be downloaded here https://popcorntime.io/ It is only 40-50MB...
here is pcap file https://drive.google.com/file/d/0B0SCwy1irn3qOGY4WFdxalQyVFk/view?usp=sharing and here is fiew screenshots of it:
http://www.dodaj.rs/f/3Y/2d/3SYEjWhH/popcorntimescshot.png
http://www.dodaj.rs/f/S/Gj/2MNcZwaG/popcorntimemv.png

I've used this DPI for many months; I always installed it in the past with the svn commands, as well as the git repository wasn't available yet.
So, for my new Ubuntu VMs as well as for my Ubuntu PC, I started installing it with the procedure here illustrated; since I have always used the json output, I installed the libraries before the autogen and the configuration, and the compilation was successful. But, when started the dpi (i.e. sudo ./ndpiReader -i eth0 -v 2 -j capture.json), the file isn't created at all. I tried it on many VMs and on my local computer, but nothing changed.
So I got back the last version from an older machine and repetead the previous (autogen, configure, make) and the same procedure gave me the output json file, so I thought it is a version problem!

In ndpi_detection_process_packet the packet parameter is defined as const unsigned char *packet:

ndpi_protocol ndpi_detection_process_packet(struct ndpi_detection_module_struct *ndpi_struct,
                    struct ndpi_flow_struct *flow,
                    const unsigned char *packet,
                    const unsigned short packetlen,
                    const u_int64_t current_tick_l,
                    struct ndpi_id_struct *src,
                    struct ndpi_id_struct *dst)

However on line https://github.com/ntop/nDPI/blob/dev/src/lib/ndpi_main.c#L3316 you cast this const away:

  flow->packet.iph = (struct ndpi_iphdr *)packet;

Which eventually gets use as a parameter to ndpi_network_ptree_match in line https://github.com/ntop/nDPI/blob/dev/src/lib/ndpi_main.c#L3423 :

struct ndpi_packet_struct *packet = &flow->packet;

if((ret.master_protocol = ndpi_network_ptree_match(ndpi_struct, (struct in_addr *)&packet->iph->saddr)) == NDPI_PROTOCOL_UNKNOWN)
  ret.master_protocol = ndpi_network_ptree_match(ndpi_struct, (struct in_addr *)&packet->iph->daddr);

Inside ndpi_network_ptree_match you write to this memory https://github.com/ntop/nDPI/blob/dev/src/lib/ndpi_main.c#L1673:

pin->s_addr = ntohl(pin->s_addr); /* Make sure all in network byte order otherwise compares wont work */

This causes a segfault if packet is in read-only memory. Surely you should not be writing anything to the library consumer packet buffer? I.e. flow->packet.iph should be read-only at all times?

Ability to return multi-protocol detection (e.g. IP, UDP, DNS, Twitter) instead of just Twitter for a DNS query for www.twitter.com

Add nDPI support for Globo.tv application. An application providing access to Globo, a TV channel in Brazil. Trace will be sent separately

The release notes for nDPI 1.6 mention "New API call for converting nDPI protocols IDs to names ndpi_protocol2name()", but the function name is not included in libndpi.sym and thus not exported in the resulting shared library.

natano@ketzer:~$ nm /usr/lib/libndpi.so.1.0.0|grep protocol2name
000000000000efd0 t ndpi_protocol2name

Notice the lower-case 't', which indicates that the symbol is local.

Is this on purpose?

Add nDPI support for Portas Abertas Application, a Mobile Operator Portal. Trace will be sent separately

Add nDPI support for OpenSignal, a Speed test application. Trace will be sent separately

Add nDPI support for HOOQ application. HOOQ is a Video Streaming service. https://www.hooq.tv/
Traces will be sent separately.

I'm glad to see ntop / nDPI coming to git!

I see you're starting with no history though, so I wanted to say feel free to take my svn mirror from https://github.com/nyov/ndpi and continue from there.
(I haven't watched it for a while now, but I hope it has all the ndpi branches and that no commits were missed by the mirror script.)
You can of course drop the glue and opendpi code after, if you wish.

It would be a boon to drop my mirror script if git becomes the canonical source.

Secure Socket Tunneling Protocol was introduced by Microsoft in Windows Vista SP1, and it is now available for Linux, RouterOS and SEIL. SSTP uses SSL v3, and therefore offers similar advantages to OpenVPN (such as the ability to use to TCP port 443 to avoid NAT firewall issues), and because it is integrated into Windows may be easier to use and more stable.
The following header structure is common to all types of SSTP packets:[8]

https://en.wikipedia.org/wiki/Secure_Socket_Tunneling_Protocol

here is pcap file https://drive.google.com/file/d/0B0SCwy1irn3qZE1kd2xPTWhOOGM/view?usp=sharing

Add nDPI support for Instagram (social network focused on photo publication).
Trace will be sent separately

Kakao Chat

Add nDPI support for Beta application. PCAP file with example sent separately.

Function ndpi_detection_process_packet hangs when process the packet mpegts.
Captured packet:
https://dl.dropboxusercontent.com/u/5135944/Github/mpegts.cap
This problem occur in version 1.6.1. Version 1.5.1 works normally.

I don't know why this is the case, but as shown at

For eg. It talks about using the demo pcapReader application, which is not present in the latest code.
Instead ndpiReader should be used.

Add nDPI support for EasyTaxi, an application to call taxis. Trace file will be sent separately.

Viber is no longer detected, identified as unknown. Will upload pcap file soon.

ICQ traffic (Oscar protocol) detects correct only when clients connects to icq server. Later, after client idle, when a flow was deleted as an old flow, when an icq-messages goes over the established connection, nDPI can't detect this flow as a oscar protocol.

Hello,
Please add nDPI support for Evernote (notes application generally in mobile devices)

Follows trace (tested login, synchronization, and changing a note):
http://www.netdeep.com.br/tmp/evernote.pcap

Thanks!

Hello guys. Congratulations for the work!
In my tests, the version 1.6 is not properly detecting/blocking Whatsapp messages.
Can confirm me if I have done wrong?

Hi,
Is it possible to work with ndpi to capture multiple requests and responses in a single http session (http pipeline support)?
The example code demonstrates only single request-response detection.

thanks,
Shirley.

Add nDPI support for Torcedor application. PCAP file with example sent separately.

Add nDPI support for Meu application. PCAP file with example sent separately.

Add nDPI support for EAQ application. PCAP file with example sent separately.

Please add nDPI support for TED (video conference platform application).

Trace is here:
http://www.netdeep.com.br/tmp/TED.pcap

Thanks!

Add nDPI support for Waze, community based traffic and navigation Application. Trace will be sent separately

Add nDPI support for 99 Taxis, an Application to call taxis. Trace will be sent separately.

I don't see it in /src/lib/protocols. Is it detected in some other way? Just in case pcap files for icmp and icmpv6 added... Thx

https://drive.google.com/file/d/0B0SCwy1irn3qTDlMVGhSUXc1dU0/view?usp=sharing

https://drive.google.com/file/d/0B0SCwy1irn3qZVJVRjd3X0x0Tm8/view?usp=sharing

I remember there was windows update protocol in the old days. What happened to it? Lots of people will migrate from windows 7, 8, 8.1 to windows 10... it will be 3GB update that will run in background. It will be handy to have WSUS filter to put it to low priority...

I'd like to request for Snapchat traffic to be classified by nDPI.

Server names should be the following:

feelsonice-hrd.appspot.com
feelsonice.appspot.com

Two pcap files of sample traffic are available at
https://drive.google.com/file/d/0B-7GNfxPaSADekxKUUdKdUlzWlE/view?usp=sharing

I am ddwrt user and some time ago ddwrt developers drop using l7 filters and started using nDPI with iptables (for marking-matching traffic for QoS and Access Restriction purposes). I am not developer nor the network guru. Just simple SOHO user with very basic understanding of routers.
ddwrt has big community and now we have problem because some nDPI protocols (filters) do not work. ddwrt devs says it is not ddwrt problem. Youtube, Steam do not work... my findings about this problem are here in bug report on ddwrt TRAC http://svn.dd-wrt.com/ticket/4117#comment:7
If I can provide more informations I would be happy, just to solve this issue. Thx for your work on nDPI it is already great!

Add nDPI support for SIMET, a speed test application. Trace will be sent separately

As a follow up to issue #77 , currently there is inconsistent usage of the byte order of the in_addr *pin parameter of ndpi_network_ptree_match

Based on the existing implementation of ndpi_network_ptree_match it appears ndpi_network_ptree_match expects in_addr *pin to be in host byte order, as you convert to network byte order looking at your comment here:

pin->s_addr = ntohl(pin->s_addr); /* Make sure all in network byte order otherwise compares wont work */

I'm assuming ndpi_patricia_search_best expects the prefix to be in network byte order then. However the problem is currently it's a bit of a mess how ndpi_network_ptree_match is called:

ndpi_host_ptree_match - calls ndpi_network_ptree_match with pin in host byte order
tor_ptree_match - calls ndpi_network_ptree_match with pin in network byte order
ndpi_detection_process_packet - calls ndpi_network_ptree_match with pin in network byte order
ndpi_guess_undetected_protocol - calls ndpi_network_ptree_match with pin in host byte order

Also related:

ndpi_init_ptree_ipv4 fills the prefix in network byte order
ndpi_add_host_ip_subprotocol fills the prefix in host byte order

Shouldn't this all be normalized so that whenever a struct in_addr is used that it should already be in network byte order, as the documentation and convention of struct in_addr is documented as network byte order: http://man7.org/linux/man-pages/man7/ip.7.html ?

This App is very popular in the Far East. Can you support it?

How can I send you traces for Kakao talk? I get a message 'unfortunately, we don't support that file type'

nDPI is required to differentiate between different types of WhatsApp media i.e. WhatsApp Voice vs. WhatsApp Messaging.

PCAP traces of WhatsApp Voice and WhatsApp Messaging sent by mail to [email protected]