kernel panic and build issue w/ current about ndpi-netfilter

[betolj]

You followed the steps in "ndpi.install" file?
Compile the nDPI project using the nDPI.tar.gz in this repository.
And... you do not need kernel patch anymore.

1. Install kernel sources
   apt-get install linux-source
2. Install nDPI
   apt-get install libtool
   apt-get install autoconf
   apt-get install pkg-config
   apt-get install subversion
   cd /usr/src
   svn co https://github.com/ntop/nDPI
   cd nDPI
   ./autogen.sh
   #./configure –with-pic
   make
   make install
3. ndpi-netfilter
   apt-get install iptables-dev
   cd /usr/src/
   #tar xvf ndpi-netfilter.patch.tar.gz
   cd ndpi-netfilter
   NDPI_PATH=/usr/src/nDPI make
   make modules_install
   cp /usr/src/ndpi-netfilter/ipt/libxt_ndpi.so /lib/xtables/

[syadnom]

trying now, FYI, instruction #2 says to pull nDPI from svn.....

[betolj]

Ok... i update the ndpi.install. Its better.

[betolj]

And now, the build works?

[syadnom]

yes. this works. I'm only matching a fraction of some protocols though, specifically bittorrent. ntopng seems to match much more. is this to be expected?

my test is downloading a torrent on the same box as ndpi-netfilter, here are my iptables rules:
iptables -t mangle -A INPUT -m ndpi --bittorrent -j MARK --set-mark 15
iptables -t mangle -A INPUT -m mark --mark 15 -j CONNMARK --save-mark
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -m mark --mark 15 -j DSCP --set-dscp 15
iptables -t mangle -A OUTPUT -m ndpi --bittorrent -j DSCP --set-dscp 15

[betolj]

Try with:
iptables -t mangle -I PREROUTING -m ndpi --dpi_check
iptables -t mangle -I POSTROUTING -m ndpi --dpi_check

2015-06-16 15:56 GMT-04:00 syadnom [email protected]:

yes. this works. I'm only matching a fraction of some protocols though,
specifically bittorrent. ntopng seems to match much more. is this to be
expected?

my test is downloading a torrent on the same box as ndpi-netfilter, here
are my iptables rules:
iptables -t mangle -A INPUT -m ndpi --bittorrent -j MARK --set-mark 15
iptables -t mangle -A INPUT -m mark --mark 15 -j CONNMARK --save-mark
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -m mark --mark 15 -j DSCP --set-dscp 15
iptables -t mangle -A OUTPUT -m ndpi --bittorrent -j DSCP --set-dscp 15

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[syadnom]

What does --dpi_check do? here is my output by the way, it's not matching anything. This is with a completely flushed iptables mangle table

Chain PREROUTING (policy ACCEPT 155K packets, 218M bytes)
pkts bytes target prot opt in out source destination
0 0 all -- any any anywhere anywhere protocol CHECK

Chain INPUT (policy ACCEPT 155K packets, 218M bytes)
pkts bytes target prot opt in out source destination
0 0 all -- any any anywhere anywhere protocol CHECK

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 77243 packets, 4704K bytes)
pkts bytes target prot opt in out source destination
0 0 all -- any any anywhere anywhere protocol CHECK

Chain POSTROUTING (policy ACCEPT 77246 packets, 4704K bytes)
pkts bytes target prot opt in out source destination
0 0 all -- any any anywhere anywhere protocol CHECK

[betolj]

This rule will not match anything, but force ndpi check all flows for any protocol.
So, need only PREROUTING and POSTROUTING.

[syadnom]

ok, what can I learn from that? I see no matching etc, or is that fact that it hasn't errored what we care about?

when I do something like -m ndpi --http or --ssl DROP, it does work like 90+%. Chrome doesn't immediate fail to connect because something gets through but it definitely is matching that. Bittorrent on the other hand is barely matching (20%?) vs 90%+ on ntopng's interface.

[betolj]

The dpi_check type doesn't exist internally in nDPI API. When you use this rule, all packets will be analyzed internally and dpi flows will be created too (in any direction). This increases chances of inspection of the flow through the nDPI. Maybe, I can force a valid return for a packet account statistic, but dpi_check is not a protocol.

Try only this rule:
iptables -I INPUT -m ndpi --youtube -j DROP

This will not work because you will haven't the full dpi flow and not "match" in the best moment.
The dpi flow will be created only on the first event into this rule - so, It won't be created on output.

And, for ntopng... everything is analized. It doesn't depend on a flow or trigger rule.

But, if you add "iptables -t mangle -I POSTROUTING -m ndpi --dpi_check", this rule will drop your conections - because you already have the any output flow mapped.

It's a little bit confusing, but the goal was this.

I will do more tests and try to improve detection.

[syadnom]

ok, so --dpi_check is to help increase the matches when use with other ndpi
matches.

So, I start with the
iptables -t mangle -I PREROUTING -m ndpi --dpi_check
iptables -t mangle -I POSTROUTING -m ndpi --dpi_check

then add additional rule like
iptables -A INPUT -m ndpi --youtube -j DROP

which should make it so that I drop inbound youtube videos..

now, I've done this and I'm only matching a small number of packets and I'm
able to watch youtube:

Generated by iptables-save v1.4.21 on Tue Jun 16 17:20:14 2015

*mangle
:PREROUTING ACCEPT [217739:172284508]
:INPUT ACCEPT [217667:172280972]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [230886:165472667]
:POSTROUTING ACCEPT [230908:165473818]
-A PREROUTING -m ndpi--dpi_check
-A POSTROUTING -m ndpi--dpi_check
COMMIT

Completed on Tue Jun 16 17:20:14 2015

Generated by iptables-save v1.4.21 on Tue Jun 16 17:20:14 2015

*filter
:INPUT ACCEPT [99175:66534830]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [121136:104990247]
-A INPUT -m ndpi--youtube -j DROP
COMMIT

On Tue, Jun 16, 2015 at 4:47 PM, Humberto Jucá [email protected]
wrote:

The dpi_check type doesn't exist internally in nDPI API. When you use this
rule, all packets will be analyzed internally and dpi flows will be created
too (in any direction). This increases chances of inspection of the flow
through the nDPI. Maybe, I can force a valid return for a packet account
statistic, but dpi_check is not a protocol.

Try only this rule:
iptables -I INPUT -m ndpi --youtube -j DROP

This will not work because you will haven't the full dpi flow and not
"match" in the best moment.

The dpi flow will be created only on the first event into this rule - so,
It won't be created on output.

And, for ntopng... everything is analized. It doesn't depend on a flow or
trigger rule.

But, if you add "iptables -t mangle -I POSTROUTING -m ndpi --dpi_check",
this rule will drop your conections - because you already have the any
output flow mapped.

It's a little bit confusing, but the goal was this.

I will do more tests and try to improve detection.

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[syadnom]

yeah, I'm not matching much at all. Can't drop youtube or netflix. Some
packets are getting dropped, but not enough to even stop playback.

iptables:

Generated by iptables-save v1.4.21 on Tue Jun 16 17:51:20 2015

*mangle
:PREROUTING ACCEPT [1003655:1048859972]
:INPUT ACCEPT [1003339:1048843044]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [897868:553095231]
:POSTROUTING ACCEPT [897972:553108438]
-A PREROUTING -m ndpi--dpi_check
-A POSTROUTING -m ndpi--dpi_check
COMMIT

Completed on Tue Jun 16 17:51:20 2015

Generated by iptables-save v1.4.21 on Tue Jun 16 17:51:20 2015

*filter
:INPUT ACCEPT [12395:13161190]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [11402:8897774]
-A INPUT -m ndpi--netflix -j DROP
-A FORWARD -m ndpi--netflix -j DROP
-A OUTPUT -m ndpi--netflix -j DROP
COMMIT

Completed on Tue Jun 16 17:51:20 2015

iptables -L

Chain INPUT (policy ACCEPT 68198 packets, 73M bytes)
pkts bytes target prot opt in out source
destination
57 32886 DROP all -- any any anywhere
anywhere protocol NETFLIX

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination
0 0 DROP all -- any any anywhere
anywhere protocol NETFLIX

Chain OUTPUT (policy ACCEPT 63217 packets, 50M bytes)
pkts bytes target prot opt in out source
destination
18 10242 DROP all -- any any anywhere
anywhere protocol NETFLIX

iptables -L -t mangle

Chain PREROUTING (policy ACCEPT 1070K packets, 1120M bytes)
pkts bytes target prot opt in out source
destination
0 0 all -- any any anywhere
anywhere protocol CHECK

Chain INPUT (policy ACCEPT 1070K packets, 1120M bytes)
pkts bytes target prot opt in out source
destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source
destination

Chain OUTPUT (policy ACCEPT 960K packets, 602M bytes)
pkts bytes target prot opt in out source
destination

Chain POSTROUTING (policy ACCEPT 960K packets, 602M bytes)
pkts bytes target prot opt in out source
destination
0 0 all -- any any anywhere
anywhere protocol CHECK

On Tue, Jun 16, 2015 at 5:20 PM, dan [email protected] wrote:

ok, so --dpi_check is to help increase the matches when use with other
ndpi matches.

So, I start with the
iptables -t mangle -I PREROUTING -m ndpi --dpi_check
iptables -t mangle -I POSTROUTING -m ndpi --dpi_check

then add additional rule like
iptables -A INPUT -m ndpi --youtube -j DROP

which should make it so that I drop inbound youtube videos..

now, I've done this and I'm only matching a small number of packets and
I'm able to watch youtube:

Generated by iptables-save v1.4.21 on Tue Jun 16 17:20:14 2015

*mangle
:PREROUTING ACCEPT [217739:172284508]
:INPUT ACCEPT [217667:172280972]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [230886:165472667]
:POSTROUTING ACCEPT [230908:165473818]
-A PREROUTING -m ndpi--dpi_check
-A POSTROUTING -m ndpi--dpi_check
COMMIT

Completed on Tue Jun 16 17:20:14 2015

Generated by iptables-save v1.4.21 on Tue Jun 16 17:20:14 2015

*filter
:INPUT ACCEPT [99175:66534830]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [121136:104990247]
-A INPUT -m ndpi--youtube -j DROP
COMMIT

On Tue, Jun 16, 2015 at 4:47 PM, Humberto Jucá [email protected]
wrote:

The dpi_check type doesn't exist internally in nDPI API. When you use
this rule, all packets will be analyzed internally and dpi flows will be
created too (in any direction). This increases chances of inspection of the
flow through the nDPI. Maybe, I can force a valid return for a packet
account statistic, but dpi_check is not a protocol.

Try only this rule:
iptables -I INPUT -m ndpi --youtube -j DROP

This will not work because you will haven't the full dpi flow and not
"match" in the best moment.

The dpi flow will be created only on the first event into this rule - so,
It won't be created on output.

And, for ntopng... everything is analized. It doesn't depend on a flow or
trigger rule.

But, if you add "iptables -t mangle -I POSTROUTING -m ndpi --dpi_check",
this rule will drop your conections - because you already have the any
output flow mapped.

It's a little bit confusing, but the goal was this.

I will do more tests and try to improve detection.

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[kong156]

I made some tests with youtube and it looks like all ssl connections are not recognized. I presume that the same thing happens also with netflix and bittorent protocols for encrypted connections. Also noticed that blocking youtube comes together with unecrypted google and other destinations like github?!

[syadnom]

upon further testing, I think kong's guess on ssl being an issue holds up, and for many protocols.

For example, I could only ID about 20% of an ubuntu ISO torrent, but 0% of a Fedora torrent. Inspecting the packets, the Fedora torrent was 100% SSL and the ubuntu torrent had a lot of unencrypted packets.

This issue seems specific to ndpi-netfilter because ntopng will identify these as torrents right away.

I'm wondering if the code that checks SSL certs is playing a more significant role in identifying packets and is only functional in ntopng/ndpi proper...

[kong156]

"It can be detected by analyzing the SSL client certificate and checking the name that does not match to a real host in addition of begin a bit weird. As doing DNS resolution is not a task for nDPI we let applications do and then recognize SSL-tunnelled connections". It's a quote from nDPI/README.protocols.

[syadnom]

no change apparent... I flushed iptables first so all transfers are either the torrent, or the SSH session. 313M DL vs 800k dropped.

Chain INPUT (policy ACCEPT 228K packets, 313M bytes)
pkts bytes target prot opt in out source destination
720 800K DROP all -- any any anywhere anywhere protocol BITTORRENT

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 117K packets, 7037K bytes)
pkts bytes target prot opt in out source destination
1442 136K DROP all -- any any anywhere anywhere protocol BITTORRENT

transmission:
Progress: 33.4%, dl from 23 of 29 peers (2.93 MB/s), ul to 0 (0 kB/s) [0.00]

[syadnom]

fyi, i did an svn co and got revision 41, and the module built and loaded fine:
lsmod|grep dpi
xt_ndpi 294580 4
nf_conntrack 96976 2 xt_ndpi,nf_conntrack_ipv4
x_tables 34059 4 xt_ndpi,ip_tables,iptable_filter,iptable_mangle

[65472.777851] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[65472.778598] xt_ndpi: module verification failed: signature and/or required key missing - tainting kernel
[65472.778988] xt_ndpi 2.0 (nDPI wrapper module).
[66203.002307] ip_tables: (C) 2000-2006 Netfilter Core Team

[syadnom]

this specific match:
iptables -A OUTPUT -m ndpi --bittorrent -j DROP

this rule set even drops ICMP now:
root@mtspace-server1:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted

it drops dns lookups etc. it's matching almost everyhting.....except bittorrent which still gets through.

It takes a while for the bittorrent to work around this block, but I just downloaded the ubuntu ISO in about 5 minutes with this block enabled while ever other network activity except SSH is funky.

Generated by iptables-save v1.4.21 on Wed Jun 17 12:51:59 2015

*filter
:INPUT ACCEPT [75090:102997125]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [36309:2131821]
-A INPUT -m ndpi--bittorrent -j DROP
-A INPUT -m ndpi--youtube -j DROP
-A INPUT -m ndpi--netflix -j DROP
-A OUTPUT -m ndpi--bittorrent -j DROP
-A OUTPUT -m ndpi--youtube -j DROP
-A OUTPUT -m ndpi--netflix -j DROP
COMMIT

Completed on Wed Jun 17 12:51:59 2015

Generated by iptables-save v1.4.21 on Wed Jun 17 12:51:59 2015

*mangle
:PREROUTING ACCEPT [491401:663352459]
:INPUT ACCEPT [491348:663349781]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [256604:22727118]
:POSTROUTING ACCEPT [255107:22538716]
-A PREROUTING -m ndpi--dpi_check
-A POSTROUTING -m ndpi--dpi_check
COMMIT

Completed on Wed Jun 17 12:51:59 2015

[betolj]

You remove xt_ndpi before?

[syadnom]

yes. I did this twice just to make sure. rmmod xt_ndpi (after flushing iptables). I even deleted all the source files and redownloaded, both a ZIP from github, and by svn and used the included nDPI archive.

[betolj]

You don't build this with nDPI svn.
The number of app protocols are different... only works when i change the "src/xt_ndpi.h".
Im my tests, the https requests are being filtered. But still not so good.

[kong156]

This is a brief description of how nDPI treats SSL. Not much, because you can do it without nDPI, matching the known urls.
"When using encrypted communications, the only part of the data exchange that can be decoded is the initial key exchange. nDPI contains a decoder for SSL that extracts the host name from the server certificate. This information is placed in the nDPI flow metadata in a similar way as the
server name from the ‘Host:’ HTTP header. With this approach we can identify known services and tag them according to the server name. For instance, an encrypted communication towards a server named ‘api.twitter.com’ is marked as Twitter, ‘maps.google.com’ as Google maps, and ‘*.whatsapp.net’ as
the WhatsApp messaging protocol."

[syadnom]

I'm building with the nDPI archive included in the ndpi-netfilter sources.

[betolj]

Hi,
Try again, please!
I done many code changes and fix ct collisions.

It has worked well in my tests.

[betolj]

And, FYI... you dont need make firewall marks.
This fix remember the last detection.

[kong156]

Looks much better. Seems to work with youtube over SSL and also torrents... looks promising, but needs more testing. Good job by the way :)

[syadnom]

looks good, the matchers are basically functioning. bittorrent though,
still isn't being identified anything near ntopng/ndpi. A torrent takes
longer to get starter, but only a fraction of the packets are identified.
I'm transferring 5.6MB/s in transmission-cli with

iptables -t mangle -I PREROUTING -m ndpi --dpi_check
iptables -t mangle -I POSTROUTING -m ndpi --dpi_check
iptables -A INPUT -m ndpi --bittorrent -j DROP

On Mon, Jun 22, 2015 at 12:43 PM, kong156 [email protected] wrote:

Looks much better. Seems to work with youtube over SSL and also
torrents... looks promising, but needs more testing. Good job by the way :)

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[syadnom]

slight update, if I setup the block on bittorrent before starting a
torrent, it takes ages if ever to start. if I start dropping after the
torrent is running, very little effect. It's like it is detecting the
session initiation but not detecting the peer-to-peer data stream.

again, ntopng doesn't detect this data, so it's either the pull of ndpi we
are building against, or something in the ndpi-netfilter process.

Thanks.

On Mon, Jun 22, 2015 at 1:28 PM, dan [email protected] wrote:

looks good, the matchers are basically functioning. bittorrent though,
still isn't being identified anything near ntopng/ndpi. A torrent takes
longer to get starter, but only a fraction of the packets are identified.
I'm transferring 5.6MB/s in transmission-cli with

iptables -t mangle -I PREROUTING -m ndpi --dpi_check
iptables -t mangle -I POSTROUTING -m ndpi --dpi_check
iptables -A INPUT -m ndpi --bittorrent -j DROP

On Mon, Jun 22, 2015 at 12:43 PM, kong156 [email protected]
wrote:

Looks much better. Seems to work with youtube over SSL and also
torrents... looks promising, but needs more testing. Good job by the way :)

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[kong156]

Tested the torrents, but they started to download only after I removed the rules from firewall. Maybe more testing is needed

[syadnom]

kong, try starting the torrents and then once they are going add the drop
rules.

this isn't exactly what I'm after though, I'm setting DSCP
iptables -t mangle -A POSTROUTING -m ndpi --bittorrent -j DSCP --set-dscp 15

I have a separate router that I want to queue the torrents in. The problem
is that it's only matching some torrent traffic and so only marking DSCP on
some of the traffic. I add a 1Mbps queue on the router and it fills up,
but then a ton of other torrent traffic slips by because it wasn't market
with DSCP from ndpi-netfliter.

On Mon, Jun 22, 2015 at 1:31 PM, kong156 [email protected] wrote:

Tested the torrents, but they started to download only after I removed the
rules from firewall.

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[kong156]

I think this point is irrelevant. After some time these torrents will stop and others will start just with the firewall rules up.

[kong156]

This behaviour is normal, i think, because nDPI doesn't decrypt, just marks the begining of a conversation and tracks it until dies.

[syadnom]

ntopng+ndpi matches these packets, ndpi-netfilter doesn't. ndpi-netfilter
should ideally match everything nDPI does, which it's not.

On Mon, Jun 22, 2015 at 1:42 PM, kong156 [email protected] wrote:

This behaviour is normal i think because nDPI doesn't decypt, just marks
the beginig of a conversation and tracks it until dies.

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[kong156]

maybe you're right, but what tests did you conduct to make a relevant comparison?

[syadnom]

I have 2 vms, 1 with ntopng+ndpi and 1 with ndpi-netfilter.

on each vm I start an ubuntu ISO download.

the ntopng box matches essentially 100% of these packets as bittorrent.
The only other data on this box is the ntopng web interface an an ssd
session.

the ndpi-netfilter box marks maybe 1/6 of the traffic. I can see this from
iptables -t mangle -L -v, showing a big descrepancy between marked packets
and interface packets.

In my firewall, I make a 1Mbps queue for the marked packets. I see the
same disparity between transmission-cli's reported speed and the queue.

I can block that traffic on the router, or on the ndpi-netfilter box
successfully if I turn the drops on before starting a torrent, but if any
of the torrent's announce gets through, the torrent takes off.

so, I can block the torrents, but I can't shape them. I can't stick
torrents in a queue to say "only let 5Mbps of torrents though" because I
can only match the announcement or initial connection.

On Mon, Jun 22, 2015 at 1:47 PM, kong156 [email protected] wrote:

maybe you're right, but what tests did you conduct to make a relevant
comparision?

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[kong156]

Did you serialize these machines? Same traffic passes the both machines? It seems enough to me to detect the beginning of the connection, if keeping tracks is implemented right. Anyway, I don't say that you are wrong, just suspicious, maybe more testing is needed...

[syadnom]

the vms are identical except for the installation of ndping and
ndi-netfilter and dependancies. Same torrent, same router in front of them.

just start a torrent, then add the drop rule. You'll see that it doesn't
drop the subsequent connections the torrent makes. This, by logical
derivation, means that it can't be used to queue an application, only
completely drop, because it's really only matching the initial connection,
the announce.

ndpi matches like a pro though.

On Mon, Jun 22, 2015 at 2:06 PM, kong156 [email protected] wrote:

Did you serialize these machines? Same traffic passes the both machines?
It seems enough to me to detect the begining of the connection, if keeping
tracks is implemented right. Anyway, I don't say that you are wrong, just
suspicious, may more testing is needed...

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[betolj]

Include a ACCEPT rule at beginning, and after run ktorrent... in my tests, i can see all packets. I do more tests later.

[syadnom]

On Mon, Jun 22, 2015 at 2:14 PM, Humberto Jucá [email protected]
wrote:

Include a ACCEPT rule at beginning, and after run ktorrent... in my tests,
i can see all packets. I do more tests later.

can you explain this a little more?

[kong156]

A torrent doesn't mean a single session. A torrent is composed of multiple sessions without a single affinity between them. So any open connection after isn't related with other at all, just the trackers instruct the clients about the peers arround.

[kong156]

I'm curious if the connections are multipling after you raise the bittorrent rule or just dropping.

[betolj]

I'm downloading the ubuntu server with torrent too.
I run ktorrent only for this... and without other downloads
Take a look in this results:

iptables -I OUTPUT -m ndpi --bittorrent -j ACCEPT
iptables -I INPUT -m ndpi --bittorrent -j ACCEPT

iptables -nL INPUT -v | head -3
Chain INPUT (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
523K 694M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 protocol BITTORRENT

ls -lah /home/humberto/ubuntu-14.04.2-server-amd64.iso
-rw-rw-r-- 1 humberto humberto 595M Jun 22 16:20 /home/humberto/ubuntu-14.04.2-server-amd64.iso

Works for me

[kong156]

is this torrent encrypted?

[betolj]

DROP works too

[betolj]

I can see all torrent traffic when download.
And now i can stop or allow when include or delete drop rule.

[kong156]

have to wait for an encrypted torrent to check this...

[kong156]

so far you're right :))

[betolj]

Yes... a encrypted torrent i cant stop

[betolj]

Maybe, i need to enable NDPI_PROTOCOL_SSL_NO_CERT with NDPI_PROTOCOL_BITTORRENT to drop this.

[betolj]

More tests... :)

[kong156]

12% missed

[kong156]

should be an improvement :)

[betolj]

👍

[kong156]

good job anyway, appreciate...

[betolj]

Thanks alot

[kong156]

for nothing

[kong156]

Next days, I will test the shaping to verify syadnom's assertions. Right now, I don't have the right conditions to do this, but they will be. Matching and shaping rules are different things...

[kong156]

I've just downloaded a 1.9GB encrypted torrent with the following rules set:
#iptables -nvL -t filter

Chain INPUT (policy ACCEPT 59057 packets, 64M bytes)
pkts bytes target prot opt in out source destination
1617K 2301M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x1

#iptables -nvL -t mangle

Chain PREROUTING (policy ACCEPT 1677K packets, 2365M bytes)
pkts bytes target prot opt in out source destination
1677K 2365M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore

Chain POSTROUTING (policy ACCEPT 35840 packets, 3065K bytes)
pkts bytes target prot opt in out source destination
692K 41M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore
656K 38M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x0
1270 85068 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0 protocol BITTORRENT MARK set 0x1
35840 3065K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save

The counters exceeds the download but it seems to be all right because we also have to take into account the extra traffic induced by proto/encryption overhead.

[syadnom]

kong, can you dump your iptables commands for me so I can test identical
settings?

On Tue, Jun 23, 2015 at 2:49 PM, kong156 [email protected] wrote:

I've just downloaded a 1.9GB encrypted torrent with the following rules
set:
#iptables -nvL -t filter

Chain INPUT (policy ACCEPT 59057 packets, 64M bytes)
pkts bytes target prot opt in out source destination

1617K 2301M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x1

#iptables -nvL -t mangle

Chain PREROUTING (policy ACCEPT 1677K packets, 2365M bytes)
pkts bytes target prot opt in out source destination

1677K 2365M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore

Chain POSTROUTING (policy ACCEPT 35840 packets, 3065K bytes)
pkts bytes target prot opt in out source destination

692K 41M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore
656K 38M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x0
1270 85068 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0 protocol
BITTORRENT MARK set 0x1
35840 3065K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save

The counters exceeds the download but it seems to be all right because we
also have to take into account the extra traffic induced by
proto/encryption overhead.

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[kong156]

*mangle
-A PREROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A POSTROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A POSTROUTING -m mark ! --mark 0x0 -j ACCEPT
-A POSTROUTING -m mark --mark 0x0 -m ndpi--bittorrent -j MARK --set-xmark 0x1/0xffffffff
-A POSTROUTING -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
COMMIT
*filter
-A INPUT -m mark --mark 0x1 -j ACCEPT
COMMIT

make abstraction of masks which are defaults

[syadnom]

I must be marking in the wrong spot... See my add below. In about 250MB
of downloaded, I've only marked 6M of DSCP. Thoughts?

*mangle
:PREROUTING ACCEPT [93389:131296225]
:INPUT ACCEPT [93385:131296029]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [48137:2912071]
:POSTROUTING ACCEPT [8204:769477]
-A PREROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask
0xffffffff
-A POSTROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask
0xffffffff
-A POSTROUTING -m mark ! --mark 0x0 -j ACCEPT
-A POSTROUTING -m mark --mark 0x0 -m ndpi--bittorrent -j MARK --set-xmark
0x1/0xffffff$
-A POSTROUTING -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
0xffffffff
############################################################################

added DSCP setting for the 0x1 mark from 2 commands up

##########################
-A POSTROUTING -m mark --mark 0x1 -j DSCP --set-dscp 0x0f
############################################################################
COMMIT

Completed on Tue Jun 23 15:15:02 2015

Generated by iptables-save v1.4.21 on Tue Jun 23 15:15:02 2015

*filter
:INPUT ACCEPT [282992:400158777]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [488206:29709563]
-A INPUT -m mark --mark 0x1 -j ACCEPT
COMMIT

On Tue, Jun 23, 2015 at 3:02 PM, kong156 [email protected] wrote:

*mangle
-A PREROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask
0xffffffff
-A POSTROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask
0xffffffff
-A POSTROUTING -m mark ! --mark 0x0 -j ACCEPT
-A POSTROUTING -m mark --mark 0x0 -m ndpi--bittorrent -j MARK --set-xmark
0x1/0xffffffff
-A POSTROUTING -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask
0xffffffff
COMMIT
*filter
-A INPUT -m mark --mark 0x1 -j ACCEPT
COMMIT

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[kong156]

DSCP doesn't preserve across the whole session, so you have to figure how to fix this issue.

[kong156]

You set DSCP for egress and must fit egress only. The receiver doesn't preserve it so that you souldn't find it on ingress.

[betolj]

What exactly you want to do?

Are you trying to set DSCP into your firewall and shape in a border router?
You could set DSCP and shape directly in your firewall box.
You could set DSCP and shape in your bittorrent box too - if its acceptable.

So, its depends on how and when you want to filter and shape it.

[kong156]

You should shape your traffic with flow-based classifiers (ex. flow-based WFQ in case of a Cisco router or something similar for other vendors) or use a linux based router, brocade, vyatta, edgeos, maybe mikrotik if connmark or similar techniques are supported.
You could use DSCP if you are in control of the both sides.

[kong156]

Thanks to nDPI and betolj, now we can classify the traffic at L7 which is way better for lazy admins like me :(. You are a life saver, betolj :). Hope that my joy isn't premature.

[betolj]

For the next release i'll work on IPv6 support too.
Thanks... 👍

[syadnom]

ok, so I tested very thoroughly. It seems that nDPI is matching the
torrent downloads fine, but it's not matching some other things like DHT
and peer exchange etc, which open NEW connections thus making the
connection mark ineffective.

I changed my iptables log file to a file and did this:
iptables -A INPUT -m ndpi --bittorrent -j LOG --log-prefix="bittorrent: "

then I pushed the src ip addresses from the match to my router into an
address list. I match those packets to my queue and it is very accurate.
This isn't efficient as you can imagine, but I needed to see if the
matching was actually effective.

I'm going to work on matching connections related to the conn-marked ones
and see if that improves the matching. That said, it looks like
ndpi-netfilter is doing a reasonable job of matching now.

########################

What I'm trying to do is mark torrents that are downstream of the router,
and push that data back up to the router for queuing.

I was hoping that just marking DSCP on the return packets would be enough
to mark the connections on the router (mikrotik) so they could be queued,
but it looks like torrents are just too good at evading connection marks
for this to work.

My next path to this is to put the device upstream of the router and make
it a 'bump-in-the-wire' bridge, still marking DSCP but doing so on the
forward chain. This will let me queue the downloads as well as match the
src address for the upload queue. I originally didn't want to do this
because it adds another device in the chain that could fail.

On Wed, Jun 24, 2015 at 9:38 AM, Humberto Jucá [email protected]
wrote:

For the next release i'll work on IPv6 support too.
Thanks... [image: 👍]

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[kong156]

Your DSCP approach is hopeless. It won't work.

[syadnom]

On Wed, Jun 24, 2015 at 12:16 PM, kong156 [email protected] wrote:

Your DSCP approach is hopeless. It won't work.

why?

[kong156]

see the reason above.

[syadnom]

On Wed, Jun 24, 2015 at 12:17 PM, kong156 [email protected] wrote:

see the reason above.

maybe I'm blind, but I see no reason above.

[kong156]

:)) OK. You could use DSCP if you are in control of the both sides. The receiver doesn't preserve it so that you shouldn't find it on ingress. Source address is ephemeral.

[kong156]

I'm worried about youtube... this one, at least, isn't functional for connmark. It's still ok for DROP rule.

[kong156]

It seems that nDPI identifies it as Quic proto. That's fine for me :) Older versions find it as youtube.

[betolj]

May be a true proto detection!
https://groups.google.com/a/chromium.org/forum/#!topic/proto-quic/_7EzqwFzu7I

[kong156]

If I understand right, you have a ndpi-netfilter machine as wire-bump behind a mikrotik router. In this case, I think you can restore the connmarks on mikrotik using DSCP marks from your ndpi-netfilter for both, upstream and downstream. Am I right?

[kong156]

Also checked against facebook and it seems to work well. Can't tell if it catches till the last bit, but it performs very nice.

[syadnom]

I'm closing this as the 'issues' have basically been fixed or determined to be protocol specific oddities. Thanks.