p-quic / pquic

The PQUIC implementation, a framework that enables QUIC clients and servers to dynamically exchange protocol plugins that extend the protocol on a per-connection basis

PQUIC

The PQUIC implementation, a framework that enables QUIC clients and servers to dynamically exchange protocol plugins that extend the protocol on a per-connection basis.

The current PQUIC implementation supports the draft-29 version of the QUIC specification.

Using Docker

Docker builds exist on (Docker Hub)[https://hub.docker.com/r/pquic/pquic/]. They contain a build of the master branch, along with the necessary tools to build PQUIC.

    docker run -it pquic/pquic
    ./picoquicdemo -h

Building PQUIC

More detailed instructions are available at: https://pquic.org

PQUIC is developed in C, and is based on picoquic (https://github.com/private-octopus/picoquic). It can be built under Linux (the support of Windows is not provided yet). Building the project requires first managing the dependencies, Picotls, uBPF, michelfralloc, libarchive and OpenSSL.

PQUIC on Linux

To build PQUIC on Linux, you need to:

• Install and build Openssl on your machine

• Install libarchive. It is usually found in distribution packages (e.g., apt install libarchive-dev) or on (the LibArchive page)[http://libarchive.org/]

• Clone and compile Picotls (https://github.com/p-quic/picotls), using cmake as explained in the Picotls documentation.

• Clone and compile PQUIC with both its uBPF and michelfralloc dependencies:

   git submodule update --init
   cd ubpf/vm
   make
   cd ../..
   cd picoquic/michelfralloc
   make
   cd ../..
   cmake .
   make

Documentation

Generate doc with

doxygen

pquic's Issues

error while compiling

The last make produces error. How can I compile?

$ make
[ 45%] Built target picoquic-core
[ 47%] Linking C executable picoquic_ct
/usr/bin/ld: cannot find -lprofiler
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/picoquic_ct.dir/build.make:448: picoquic_ct] Error 1
make[1]: *** [CMakeFiles/Makefile2:114: CMakeFiles/picoquic_ct.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

How multiple path can exist?

While running as container, only one network adapter can be connected to the container. How to achieve multipath then?

Difference between picoquic and p-quic

@mpiraux what is the main difference between picoquic and p-quic
seems to me both has same options

Server exit with code = -1 immediately

HI,
When I make pquic, there is an error reported: it said IPV6_DONTFRAG is not defined. Then I modified all this macro as 0. It compiled successfully. But when I ran ./picoquicdemo . The terminal shows Starting PicoQUIC server on port 4443, server name = ::, just_once = 0, hrr= 0, 0 local plugins and 0 both plugins Server exit, ret = -1 Server exit with code = -1
I used CentOS 7 as the environment. How should I proceed?

Thanks and regards,
Xiangrui

Invalid original_destination_connection_id

Hi,

During testing commit 841c822, we discovered that the PQUIC server always updates/overwrites its original_destination_connection_id when processing the client's Initial packet.

According to RFC 9000, original_destination_connection_id is the value of the Destination Connection ID field from the first Initial packet sent by the client. However, when sending the packet sequence below, PQUIC will overwrite the value every time it receives a new Initial packet:

1. Initial packet that carries a Ping frame
2. Initial packet that carries a Ping frame

Using pquic with chrome or chromium.

Hi,
I wanted to know if it is possible to use pquic with Chrome or Chromium.

-lprofiler

getting an error of 'cant find -lprofiler' when trying to make pquic

Segmentation Fault on Ubuntu 16.04

After compiling pquic on Ubuntu 16.04 using the guide on https://pquic.org/, picoquicdemo quits with a segmentation fault.

The used versions of openssl and libarchive are as follows:
OpenSSL_VERSION: 1.0.2g
libarchive_version: 3.4.0

Server:
$ ./picoquicdemo
Starting PicoQUIC server on port 4443, server name = ::, just_once = 0, hrr= 0, 0 local plugins and 0 both plugins
2701ba8be43457fc: Receiving packet type: 2 (initial), S0, Version ff00000e,
2701ba8be43457fc: <2701ba8be43457fc>, <5d1d9d49b00c3ee9>, Seq: 0, pl: 1187
2701ba8be43457fc: Decrypted 1187 bytes
2701ba8be43457fc: Crypto HS frame, offset 0, length 279: 01000113030373e9...
2701ba8be43457fc: padding, 904 bytes

2701ba8be43457fc: 0.001671 : Connection established, state = 11, from length: 28
2701ba8be43457fc: Client address: ::1, port 53473
2701ba8be43457fc: SNI not received.
2701ba8be43457fc: Received ALPN: hq-14
2701ba8be43457fc: Received transport parameter TLS extension (62 bytes):
2701ba8be43457fc: Proposed version: ff00000e
2701ba8be43457fc: Extension list (56 bytes):
2701ba8be43457fc: Extension type: 0, length 4 (0x0000 / 0x0004), 00200000
2701ba8be43457fc: Extension type: 1, length 4 (0x0001 / 0x0004), 00100000
2701ba8be43457fc: Extension type: 2, length 2 (0x0002 / 0x0002), 4000
2701ba8be43457fc: Extension type: 3, length 2 (0x0003 / 0x0002), 000f
2701ba8be43457fc: Extension type: 5, length 2 (0x0005 / 0x0002), 05a0
2701ba8be43457fc: Extension type: 8, length 2 (0x0008 / 0x0002), 4000
2701ba8be43457fc: Extension type: 10, length 4 (0x000a / 0x0004), 00010063
2701ba8be43457fc: Extension type: 11, length 4 (0x000b / 0x0004), 0000ffff
Segmentation fault (core dumped)

Client:
$ ./picoquicdemo ::1 4443
Starting PicoQUIC connection to server IP = ::1, port = 4443 and 0 local plugins
No server name specified, certificate will not be verified.
2701ba8be43457fc: Sending packet type: 2 (initial), S0, Version ff00000e,
2701ba8be43457fc: <2701ba8be43457fc>, <5d1d9d49b00c3ee9>, Seq: 0, pl: 1187
2701ba8be43457fc: Prepared 1187 bytes
2701ba8be43457fc: Crypto HS frame, offset 0, length 279: 01000113030373e9...
2701ba8be43457fc: padding, 904 bytes

2701ba8be43457fc: Sending 1232 bytes to 0:0:0:0:0:0:0:1 at T=0.001408 (5974cb77c67e0)
pquic/picoquic/sender.c:1331 [retransmit_needed]: 328 <= 1232
pquic/picoquic/sender.c:1400 [retransmit_needed]: Retransmit packet type 2, pc=2, seq = 0, is_client = 1

2701ba8be43457fc: Sending packet type: 2 (initial), S0, Version ff00000e,
2701ba8be43457fc: <2701ba8be43457fc>, <5d1d9d49b00c3ee9>, Seq: 1, pl: 1187
2701ba8be43457fc: Prepared 1187 bytes
2701ba8be43457fc: Crypto HS frame, offset 0, length 279: 01000113030373e9...
2701ba8be43457fc: padding, 904 bytes

2701ba8be43457fc: Sending 1232 bytes to 0:0:0:0:0:0:0:1 at T=1.002702 (5974cb78baf2e)
pquic/picoquic/sender.c:1331 [retransmit_needed]: 328 <= 1232
pquic/picoquic/sender.c:1400 [retransmit_needed]: Retransmit packet type 2, pc=2, seq = 1, is_client = 1

Post request

Hi, I have a question about how to handle post requests. There is a struct named stream_ctx whose member variable path_callback is given callback functions to handle events such as picohttp_callback_provide_data, but I don't see this callback function anywhere in the code.

However, I saw demoserver_post_callback() in picoquicdemo.c in the picoquic implementation. Did you delete or move it somewhere else?

Question: Can plugins modify the congestion control behavior?

I am sorry to ask a meta question here, but I was just wonder whether or not you are considering plugins that have the capability to modify the congestion control algorithms in quic? There are protocols that do this type of thing (UDT3, https://en.wikipedia.org/wiki/UDP-based_Data_Transfer_Protocol) and I thought it might be really cool to have quic be able to do something similar.

I think that the work on making quick "pluginizable" is awesome and incredibly valuable. Thank you for working so hard on it! I apologize if the question is stupid, naive, or off topic. Please tell me if it is!

Thanks again!

Will

QUIC_ prefix in keylog is deprecated

Your interop runner image still creates keylog files with the QUIC_ prefix, which has been long deprecated and is causing wireshark decryption failures.

Uniflows are ACTIVE before address validation is completed

Part of the discussion in #18

Limitless active_connection_id_limit

Hi,

During testing commit 841c822, we discovered that the PQUIC server does not set a limit for its active_connection_id_limit transport parameter and able to process up to 1000 NEW_CONNECTION_ID frame.

According to RFC 9000, if the value of active_connection_id_limit is absent, a default limit of 2 is assumed. This means that if the number of active connection IDs exceeds the value advertised in its active_connection_id_limit transport parameter (which is 2 in this case), the server MUST close the connection with an error of type CONNECTION_ID_LIMIT_ERROR.

Malformed Packets captured in Wireshark

I an getting Malformed Packets in Wireshark whenever I run the picoquicdemo.

You can get the .pacpng file in this link --> https://drive.google.com/drive/folders/1-_XI2ipoXmqqj4ukiXUZWWhNjGP2-_Yb?usp=sharing

Please help me rectify this problem.

Unable to send my own data using PQUIC

Hi,
I tried sending data using the client-server implementation by changing index.html but I am unable to send any data by doing so. The index.hltm file keeps resetting itself.
Am I doing something wrong? If yes then please let me know the correct way to do that.

Multipath Plugin does not establish subflows

I'm using the picoquicdemo with the multipath_rr.plugin in a mininet-wifi network. The network consists of two stations with two Mesh-Interfaces for each Station. To transmit data, two mesh-networks are established between the stations:
sta1-mp0 (10.0.0.1) <--> sta2-mp0 (10.0.0.2) as mesh1
sta1-mp1 (10.0.1.1) <--> sta2-mp1 (10.0.1.2) as mesh2

On the server i start: "./picoquicdemo -P plugins/multipath/multipath_rr.plugin -p 4443"
and on the client : "./picoquicdemo -P plugins/multipath/multipath_rr.plugin -G 10000000 10.0.0.2 4443"
The transmission is done without any error.

I used wireshark to see if everything worked as expected and saw that the QUIC-Protocol is used bidirectional between 10.0.0.1 and 10.0.0.2 (as expected), but there is no connection between 10.0.1.1 and 10.0.1.2.
There are also Frames using the UDP-Protocol from:
10.0.0.2 to 10.0.1.1
10.0.0.1 to 10.0.1.2

I can post the python code for the network if necessary.

how to run pquic with multipath plugins

I try to run pquic with multipath, but meet many problems, could you pls give the basic command on how to run mpquic?

I have tried following cmd but it crash:

1. server side:
   ./picoquicdemo -P plugins/multipath/multipath_rr_cond.plugin -P plugins/multipath/multipath_rr.plugin

2. client side:
   ./picoquicdemo -p 4443 -P plugins/multipath/multipath_rr.plugin -P plugins/multipath/multipath_rr_cond.plugin 127.0.0.1

==crash log==
Starting PicoQUIC server on port 4443, server name = ::, just_once = 0, hrr= 0, 2 local plugins and 0 both plugins
local plugin plugins/multipath/multipath_rr_cond.plugin
local plugin plugins/multipath/multipath_rr.plugin
include multipath_cond.plugin...
create memory manager for plugin be.qdeconinck.multipath.rtt
create fixed block size memory manager
Successfully inserted preplugin plugins/multipath/multipath_rr_cond.plugin
include multipath.plugin...
create memory manager for plugin be.qdeconinck.multipath.rr
create dynamic memory manager
Successfully inserted local plugin plugins/multipath/multipath_rr.plugin
13cdaad11dd1d5ac: Receiving packet type: 2 (initial), S0, Version ff00001d,
13cdaad11dd1d5ac: <13cdaad11dd1d5ac>, <8d0a8603a507a0a8>, Seq: 0, pl: 1206
13cdaad11dd1d5ac: Decrypted 1206 bytes
13cdaad11dd1d5ac: Crypto HS frame, offset 0, length 275: 0100010f030393df...
13cdaad11dd1d5ac: padding, 927 bytes

quic/picoquic/tls_api.c:386 [picoquic_client_hello_call_back]: ALPN Selection call back selects 0 (out of 8)
quic/picoquic/tls_api.c:401 [picoquic_client_hello_call_back]: Client Hello call back returns 0 (0x0)
Register plugin be.qdeconinck.multipath.rtt for full registration
include multipath_cond.plugin...
Replace pluglet already inserted!
Failed to insert pluglet for parametrable protocol operation parse_frame with param 64
Trying to unplug pluglet for non-existing proto op id process_transport_parameter...
Segmentation fault (core dumped)

I think maybe I didn't get the right way to use it, could you pls show me how to run? thank you very much.

Segmentation fault (core dumped)

Program received signal SIGSEGV, Segmentation fault.
__strlen_sse2 () at ../sysdeps/x86_64/multiarch/strlen-vec.S:126
126     ../sysdeps/x86_64/multiarch/strlen-vec.S: No such file or directory.
(gdb)
(gdb)
(gdb)
(gdb) bt
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/strlen-vec.S:126
#1  0x000055555556e629 in quic_client (ip_address_text=ip_address_text@entry=0x7fffffffe641 "snort-automation1.org",
    server_port=server_port@entry=5443, sni=<optimized out>, sni@entry=0x0, root_crt=root_crt@entry=0x0,
    proposed_version=proposed_version@entry=4278190091, force_zero_share=force_zero_share@entry=0, mtu_max=0,
    F_log=0x7ffff7a59780 <_IO_2_1_stdout_>, F_tls_secrets=0x0, local_plugin_fnames=0x7fffffffde00, local_plugins=1,
    qlog_filename=0x0, plugin_store_path=0x0, stats_filename=0x0, alpn=0x5555555ca1d3 "hq-29",
    client_scenario_text=0x5555555fe300 "0:/10000\n", no_disk=1, out_dir=0x0) at /root/pquic/picoquicfirst/picoquicdemo.c:820
#2  0x000055555556b5dc in main (argc=<optimized out>, argv=<optimized out>) at /root/pquic/picoquicfirst/picoquicdemo.c:1252
(gdb) q

What are the differences between plugins with & without "cond"

For provided plugins, there are some with "cond", maybe for "conditional"? but no docs indicated this. What are the differences?
E.g.
plugins/multipath/multipath_rtt_cond.plugin
plugins/multipath/multipath_rtt.plugin

fatal error: ptmalloc3/malloc-private.h: No such file or directory

Hi,
I am trying to run PQUIC but while running make command in pquic directory I am getting this error:

fatal error: ptmalloc3/malloc-private.h: No such file or directory
5 | #include "ptmalloc3/malloc-private.h"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

I am following instructions given on https://pquic.org/

Handling 'interface down' events

It seems mp-quic doesn't handle changes in interface status. Case 1 and 2 (below) are almost identical, but one of the client interfaces is turned down at 2s in Case 2. Additionally, the available link capacity is always higher in Case 2 than Case 1. Case 1 finishes in 11s, but Case 2 throws an error at 14.3s.

For sake of reproducibility, the end of the bug report contains a small modification of minitopo.

Thanks!

case 1

topo

topoType:MultiIf
leftSubnet:10.0.
rightSubnet:10.1.
path_c2r_0:40,40,2
path_c2r_1:40,40,2

xp

xpType:pquic
clientPcap:yes
serverPcap:yes
snaplenPcap:100
pquicPlugins:~/pquic/plugins/multipath/multipath_rr_cond.plugin
pquicSize:5120000
pquic_ifdown:0

logs

$ mprun -t topo -x ./xp_norm; tail pquic_client.log 
2021-05-06 13:11:33,634 [INFO] __init__: Extract parameters from file topo
2021-05-06 13:11:33,635 [WARNING] load_parameter_file: Got error 'need more than 1 value to unpack' for line '
'; ignore it
2021-05-06 13:11:33,638 [INFO] __init__: {'path_c2r_0': '40,40,2', 'path_c2r_1': '40,40,2', 'rightSubnet': '10.1.', 'topoType': 'MultiIf', 'leftSubnet': '10.0.'}
Link type: c2r
Link id: 0
  Delay: 40.0
  Queue Size: 40
  Bandwidth: 2.0
  Loss: 0.0
  Backup: 0
      
Link type: c2r
Link id: 1
  Delay: 40.0
  Queue Size: 40
  Bandwidth: 2.0
  Loss: 0.0
  Backup: 0
      
2021-05-06 13:11:33,643 [INFO] __init__: Initializing MultiInterfaceTopo...
2021-05-06 13:11:33,646 [INFO] apply_topo: Using topo Simple multiple interface topology 
/-sw---bl---sw-\ 
c                r                s
\-sw---bl---sw-/ \-sw---bl---sw-/

2021-05-06 13:11:33,649 [INFO] apply_topo_config: Using topo config <topos.multi_interface.MultiInterfaceConfig object at 0x7f8ea9af0710>
2021-05-06 13:11:34,234 [INFO] disable_tso: Disable TSO, GSO and GRO on all interfaces of all nodes
2021-05-06 13:11:34,374 [INFO] configure_interfaces: Configure interfaces using MultiInterfaceConfig...
2021-05-06 13:11:34,375 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_1-eth1 root; tc qdisc del dev bs_c2r_0_1-eth1 ingress 
2021-05-06 13:11:34,396 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_1-eth2 root; tc qdisc del dev bs_c2r_0_1-eth2 ingress 
2021-05-06 13:11:34,410 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_2-eth1 root; tc qdisc del dev bs_c2r_0_2-eth1 ingress 
2021-05-06 13:11:34,423 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_2-eth2 root; tc qdisc del dev bs_c2r_0_2-eth2 ingress 
2021-05-06 13:11:34,444 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_1-eth2 root handle 10: netem  delay 40.0ms limit 50000
2021-05-06 13:11:34,453 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_2-eth2 root handle 1:0 tbf rate 2.0mbit burst 15000 limit 78000.0
2021-05-06 13:11:34,466 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_2-eth1 root handle 10: netem  delay 40.0ms limit 50000
2021-05-06 13:11:34,475 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_1-eth1 root handle 1:0 tbf rate 2.0mbit burst 15000 limit 78000.0
2021-05-06 13:11:34,495 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_1-eth1 root; tc qdisc del dev bs_c2r_1_1-eth1 ingress 
2021-05-06 13:11:34,510 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_1-eth2 root; tc qdisc del dev bs_c2r_1_1-eth2 ingress 
2021-05-06 13:11:34,525 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_2-eth1 root; tc qdisc del dev bs_c2r_1_2-eth1 ingress 
2021-05-06 13:11:34,542 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_2-eth2 root; tc qdisc del dev bs_c2r_1_2-eth2 ingress 
2021-05-06 13:11:34,558 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_1-eth2 root handle 10: netem  delay 40.0ms limit 50000
2021-05-06 13:11:34,574 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_2-eth2 root handle 1:0 tbf rate 2.0mbit burst 15000 limit 78000.0
2021-05-06 13:11:34,582 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_2-eth1 root handle 10: netem  delay 40.0ms limit 50000
2021-05-06 13:11:34,596 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_1-eth1 root handle 1:0 tbf rate 2.0mbit burst 15000 limit 78000.0
2021-05-06 13:11:34,664 [INFO] __init__: Extract parameters from file ./xp_norm
2021-05-06 13:11:34,670 [INFO] __init__: Extract parameters from file ./xp_norm
2021-05-06 13:11:34,683 [INFO] ping: ping -c 5 -I 10.0.0.1 10.1.0.1 >> ping.log
2021-05-06 13:11:38,799 [INFO] ping: ping -c 5 -I 10.0.1.1 10.1.0.1 >> ping.log
2021-05-06 13:11:42,959 [INFO] run_tcpdump: Activating tcpdump, waiting for it to run
2021-05-06 13:11:47,963 [INFO] run_netem_at: No need to change netem
2021-05-06 13:11:47,975 [INFO] get_pquic_server_cmd: ~/pquic/picoquicdemo  -P ~/pquic/plugins/multipath/multipath_rr_cond.plugin  -c ~/pquic/certs/cert.pem -k ~/pquic/certs/key.pem &> pquic_server.log &
2021-05-06 13:11:49,983 [INFO] get_pquic_client_cmd: ~/pquic/picoquicdemo  -P ~/pquic/plugins/multipath/multipath_rr_cond.plugin  -4 -G 5120000 10.1.0.1 4443 &> pquic_client.log
2021-05-06 13:12:03,797 [INFO] <module>: cleanup mininet
bb3da03cb78ee84e: Receiving packet type: 6 (1rtt protected phi0), S1,
bb3da03cb78ee84e:     <ccdb1e8a6bfc3bec>, Seq: 5 (5)
bb3da03cb78ee84e:     Decrypted 4 bytes
bb3da03cb78ee84e:     connection_close, Error 0x0, Reason length 0

Processed 33 bytes, state = 18 (disconnected), return 0

5120000 bytes received
11022.279 ms
Client exit with code = 0

case 2

topo

topoType:MultiIf
leftSubnet:10.0.
rightSubnet:10.1.
path_c2r_0:40,40,4
path_c2r_1:40,40,4

xp

xpType:pquic
clientPcap:yes
serverPcap:yes
snaplenPcap:100
pquicPlugins:~/pquic/plugins/multipath/multipath_rr_cond.plugin
pquicSize:5120000
pquic_ifdown:2

logs

$ mprun -t topo2 -x ./xp_down; tail pquic_client.log 
2021-05-06 13:13:41,012 [INFO] __init__: Extract parameters from file topo2
2021-05-06 13:13:41,013 [INFO] __init__: {'path_c2r_0': '40,40,4', 'path_c2r_1': '40,40,4', 'rightSubnet': '10.1.', 'topoType': 'MultiIf', 'leftSubnet': '10.0.'}
Link type: c2r
Link id: 0
  Delay: 40.0
  Queue Size: 40
  Bandwidth: 4.0
  Loss: 0.0
  Backup: 0
      
Link type: c2r
Link id: 1
  Delay: 40.0
  Queue Size: 40
  Bandwidth: 4.0
  Loss: 0.0
  Backup: 0
      
2021-05-06 13:13:41,017 [INFO] __init__: Initializing MultiInterfaceTopo...
2021-05-06 13:13:41,021 [INFO] apply_topo: Using topo Simple multiple interface topology 
/-sw---bl---sw-\ 
c                r                s
\-sw---bl---sw-/ \-sw---bl---sw-/

2021-05-06 13:13:41,021 [INFO] apply_topo_config: Using topo config <topos.multi_interface.MultiInterfaceConfig object at 0x7fad6ef68710>
2021-05-06 13:13:41,616 [INFO] disable_tso: Disable TSO, GSO and GRO on all interfaces of all nodes
2021-05-06 13:13:41,758 [INFO] configure_interfaces: Configure interfaces using MultiInterfaceConfig...
2021-05-06 13:13:41,759 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_1-eth1 root; tc qdisc del dev bs_c2r_0_1-eth1 ingress 
2021-05-06 13:13:41,777 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_1-eth2 root; tc qdisc del dev bs_c2r_0_1-eth2 ingress 
2021-05-06 13:13:41,793 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_2-eth1 root; tc qdisc del dev bs_c2r_0_2-eth1 ingress 
2021-05-06 13:13:41,814 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_0_2-eth2 root; tc qdisc del dev bs_c2r_0_2-eth2 ingress 
2021-05-06 13:13:41,831 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_1-eth2 root handle 10: netem  delay 40.0ms limit 50000
2021-05-06 13:13:41,849 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_2-eth2 root handle 1:0 tbf rate 4.0mbit burst 15000 limit 97500.0
2021-05-06 13:13:41,857 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_2-eth1 root handle 10: netem  delay 40.0ms limit 50000
2021-05-06 13:13:41,870 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_0_1-eth1 root handle 1:0 tbf rate 4.0mbit burst 15000 limit 97500.0
2021-05-06 13:13:41,886 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_1-eth1 root; tc qdisc del dev bs_c2r_1_1-eth1 ingress 
2021-05-06 13:13:41,932 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_1-eth2 root; tc qdisc del dev bs_c2r_1_1-eth2 ingress 
2021-05-06 13:13:41,945 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_2-eth1 root; tc qdisc del dev bs_c2r_1_2-eth1 ingress 
2021-05-06 13:13:41,968 [INFO] configure_bottleneck: tc qdisc del dev bs_c2r_1_2-eth2 root; tc qdisc del dev bs_c2r_1_2-eth2 ingress 
2021-05-06 13:13:41,986 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_1-eth2 root handle 10: netem  delay 40.0ms limit 50000
2021-05-06 13:13:41,996 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_2-eth2 root handle 1:0 tbf rate 4.0mbit burst 15000 limit 97500.0
2021-05-06 13:13:42,015 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_2-eth1 root handle 10: netem  delay 40.0ms limit 50000
2021-05-06 13:13:42,030 [INFO] configure_bottleneck: tc qdisc add dev bs_c2r_1_1-eth1 root handle 1:0 tbf rate 4.0mbit burst 15000 limit 97500.0
2021-05-06 13:13:42,102 [INFO] __init__: Extract parameters from file ./xp_down
2021-05-06 13:13:42,104 [INFO] __init__: Extract parameters from file ./xp_down
2021-05-06 13:13:42,119 [INFO] ping: ping -c 5 -I 10.0.0.1 10.1.0.1 >> ping.log
2021-05-06 13:13:46,232 [INFO] ping: ping -c 5 -I 10.0.1.1 10.1.0.1 >> ping.log
2021-05-06 13:13:50,383 [INFO] run_tcpdump: Activating tcpdump, waiting for it to run
2021-05-06 13:13:55,387 [INFO] run_netem_at: No need to change netem
2021-05-06 13:13:55,399 [INFO] get_pquic_server_cmd: ~/pquic/picoquicdemo  -P ~/pquic/plugins/multipath/multipath_rr_cond.plugin  -c ~/pquic/certs/cert.pem -k ~/pquic/certs/key.pem &> pquic_server.log &
2021-05-06 13:13:57,409 [INFO] get_pquic_client_cmd: ((sleep 2; ip link set dev Client_0-eth1 down)&);~/pquic/picoquicdemo  -P ~/pquic/plugins/multipath/multipath_rr_cond.plugin  -4 -G 5120000 10.1.0.1 4443 &> pquic_client.log
2021-05-06 13:14:14,587 [INFO] <module>: cleanup mininet
6e586e5ba058d0cc: Receiving 33 bytes from 10.1.0.1:4443 at T=14.314553 (5c1a916532b20)
6e586e5ba058d0cc: Receiving packet type: 6 (1rtt protected phi0), S1,
6e586e5ba058d0cc:     <2533849e9bd76492>, Seq: 8 (8)
6e586e5ba058d0cc:     Decrypted 4 bytes
6e586e5ba058d0cc:     connection_close, Error 0x0, Reason length 0

Processed 33 bytes, state = 18 (disconnected), return 0

-1.0
Client exit with code = 0

modifications of minitopo

diff --git a/experiments/pquic.py b/experiments/pquic.py
index 3ebe491..53acee0 100644
--- a/experiments/pquic.py
+++ b/experiments/pquic.py
@@ -8,6 +8,7 @@ class PQUICParameter(ExperimentParameter):
     CLIENT_PLUGINS = "pquicClientPlugins"
     SERVER_PLUGINS = "pquicServerPlugins"
     SIZE = "pquicSize"
+    IF_DOWN = "pquic_ifdown"
 
     def __init__(self, experiment_parameter_filename):
         super(PQUICParameter, self).__init__(experiment_parameter_filename)
@@ -16,6 +17,7 @@ class PQUICParameter(ExperimentParameter):
             PQUICParameter.CLIENT_PLUGINS: "",
             PQUICParameter.SERVER_PLUGINS: "",
             PQUICParameter.SIZE: 10240000,
+            PQUICParameter.IF_DOWN: 0,
         })
 
 
@@ -40,6 +42,7 @@ class PQUIC(Experiment):
         self.client_plugins = self.experiment_parameter.get(PQUICParameter.CLIENT_PLUGINS)
         self.server_plugins = self.experiment_parameter.get(PQUICParameter.SERVER_PLUGINS)
         self.size = int(self.experiment_parameter.get(PQUICParameter.SIZE))
+        self.if_down = int(self.experiment_parameter.get(PQUICParameter.IF_DOWN))
 
     def prepare(self):
         super(PQUIC, self).prepare()
@@ -62,7 +65,10 @@ class PQUIC(Experiment):
         return s
 
     def get_pquic_client_cmd(self):
-        s = "{} {} -4 -G {} {} 4443 &> {}".format(PQUIC.BIN, self.get_plugin_cmd(client=True), self.size,
+        s = ""
+        if self.if_down:
+            s += "((sleep {}; ip link set dev Client_0-eth1 down)&);".format(self.if_down)
+        s += "{} {} -4 -G {} {} 4443 &> {}".format(PQUIC.BIN, self.get_plugin_cmd(client=True), self.size,
             self.topo_config.get_server_ip(), PQUIC.CLIENT_LOG)
         logging.info(s)
         return s

error: ‘IPV6_DONTFRAG’ undeclared (first use in this function)

have this error
my machine centos7：
3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

thanks

-n sni is not opening on specified server

./picoquicdemo -n snort-automation1.org -p 4443

Starting PicoQUIC server on port 4443, server name = ::, just_once = 0, hrr= 0, 0 local plugins and 0 both plugins

Incomplete frames being send during MultiPath-supported connection

While handling a connection that supports the multipath extension, the server sends MP-frames, such as ADD_ADDRESS and MP_NEW_CONNECTION_ID, which lack certain fields. E.g.: there is no retire_prior_to-field being wirtten to the frame MP_NEW_CONNECTION_ID.

I will update this issue with other frames that are incomplete when I inspect the source-code of the multipath plugin.

Problem about using multipath plugin

The connection always closed with error when I try to establish the connection with multipath_rtt_cond.plugin.
like that:
`Successfully inserted preplugin plugins/multipath/multipath_rtt_cond.plugin
Register plugin be.qdeconinck.multipath.rtt for full registration
include multipath_cond.plugin...
Successfully injected post plugins for be.qdeconinck.multipath.rtt
Negotiated ALPN: hq-29
Almost ready!

ERROR: trying to get input 5, but there are only 5 inputs available
Connection established. Version = ff00001d, I-CID: 3598041ec8f8c14b
No progress for 10 seconds. Closing.
-1.0
Client exit with code = 0`
What should I do to solve this problem?

Receiving mp_new_connection_id-frame for non-existing uniflow id

Upon setting up a connection where my client sets the transport parameter "max_sending_uniflow_id" to 0 it still receives a mp_new_connection_id-frame where the uniflow_id is set to 1, resulting in an error because the client does not have a uniflow with id 1 for the connection.

/usr/bin/ld: cannot find -lprofiler: No such file or directory

I'm using Ubuntu 22.04 Linux. I followed the steps of https://pquic.org/
for compile pquic when I run "make" command, I got this error :

user@quic-proj:~/pico/pquic$ make
[ 45%] Built target picoquic-core
[ 51%] Built target picohttp-core
[ 52%] Linking C executable picoquicdemo
/usr/bin/ld: cannot find -lprofiler: No such file or directory
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/picoquicdemo.dir/build.make:124: picoquicdemo] Error 1
make[1]: *** [CMakeFiles/Makefile2:147: CMakeFiles/picoquicdemo.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

ps: i installed google-perftools

Interop with aioquic decryption issue

This issue was mentioned previously on the pquic-channel in the quicdev slack

I also created an issue on the aioquic github page

I seem to have stumbled on a small decryption issue that seems to exist on the regular implementations of aioquic and pquic, and therefore also exists on other implementations based on aioquic and pquic. I will explain my observations across multiple tests here.

I have the following cases:
Case 1: A local aioquic client that tries to download 999999 bytes from a local PQUIC server
(Default aioquic implementation, the PQUIC server has enabled the QLOG plugin)

Case 2: An online interop result from interop.seemann.io between an aioquic client and a PQUIC server
(Test: transfer, flag: DC)

Case 3: My modified local aioquic client that tries to download 999999 bytes from a local pquic server
(Modified aioquic implementation that enables multipath, the PQUIC server has enabled the QLOG plugin, as well as the multipath_rr_cond and multipath_qlog plugins)

In all three cases both endpoints seem to have some issues with decrypting certain packets that are being transmitted while the handshake is being performed.

In Case 1 I generated a QLOG-file on both endpoints (Ac_large.qlog and Ps_large.qlog), and I also stored the console output of the PQUIC server (Ps_large.txt) All files are stored in logs.zip. By loading both QLOG files in qvis.edm.uhasselt.be and selecting the actual traces for both the client and the server (not the GENERATED ones, see the first screenshot), we get a visualisation of the issue, as seen in the second screenshot.

After receiving an initial handshake packet from the PQUIC server, the aioquic client replies with an initial packet (pn: 1) and a handshake packet (pn: 2). The PQUIC server receives the first packet, fails to decrypt it, giving a "header or decryption error: 408." message and finally dropping the packet. The second handshake packet isn't mentioned in the console output. In response the PQUIC server replies with another handshake packet (pn: 1), and two 1RTT packets (pn: 0 and 1). The aioquic client receives these packets and can decrypt the handshake packet and the first 1RTT packet, but not the second 1RTT packet, logging a "packet_decrypt_error". The aioquic client thus acknowledges the handshake packet and the first 1RTT packet, but not the second 1RTT packet.
Later on, the PQUIC server identifies this second 1RTT packet as lost, as seen in the third screenshot. However, in this case the connection is able to continue, and the 999999 bytes are transferred, resulting in a clean connection termination.

By inspecting the client's QLOG and the server's LOG from Case 2 I assume that the same problem occurs, because the client QLOG contains a "packet_decrypt_error", and the server log contains a "header or decryption error: 408.". The connection is also able to continue and terminate as in Case 1.

However, in Case 3 this problem also occurs, but when the packet is identified as lost by the pquic server, the connection is unable to continue. The PQUIC server starts sending 1RTT packets that each contain a ping and padding frame, which are all acknowledged by my aioquic client. These last two steps are repeated in a ping-ack loop as seen in the fourth screenshot, eventually leading to a very long timeout and a connection termination.

My questions:
-Is there an issue with the multipath_rr_cond plugin that prevents the server from sending stream data when a 1RTT packet is identified as lost? Resulting in the ping-ack loop?
-What is causing the decryption problem?

(Small edit note: when generating your own traces, PQUIC saves its packets in the QLOG file with {"packet_type": "onertt"}, whereas aioquic saves the packets with {"packet_type": "1rtt"}, search and replace the PQUIC entries to "1rtt" so that both QLOG files have the same notation, otherwise the visualisation will show almost all packets as "lost".)

load balance work with MPQUIC

the network topo as following:
MP-client ---- Load Balance --- MP-Server

I cant find a good solution for LB to route multiple path of a client to a MP-Server.

Build failing

While trying to build pquic, the final make step in pquic. The error is-
45%] Built target picoquic-core
make[2]: *** No rule to make target 'picoquic/michelfralloc/libmichelfralloc.a', needed by 'picoquic_ct'. Stop.
make[1]: *** [CMakeFiles/Makefile2:114: CMakeFiles/picoquic_ct.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

Error in running multipath_rtt_cond.plugin after commit

Could you check if commit e08ae8f is causing errors when running plugin multipath_rtt_cond.plugin / also multipath_rr_cond.plugin

picoquic/transport.c
int prepare_plugin_transport_extensions(picoquic_cnx_t* cnx, uint8_t* bytes, size_t bytes_max)
{
...
fprintf(stderr, "cnx->protoop_inputv[0]: %p\n", &cnx->protoop_inputv[0]);
cnx->protoop_inputv[0] = (protoop_arg_t) &value;
...
}
prints out
cnx->protoop_inputv[0]: (nil)
and throws an error
Address 0x0 is not stack'd, malloc'd or (recently) free'd

Inverse interop with aioquic packet reception issue

I am currently trying to perform a multipath connection between a modified aioquic server and a pquic client, and I have a question: Does the pquic multipath plugin allow for multiple sending uniflows to send data to the same IP address? whilst also sending from a different port?

I'll give a short example:
For testing purposes, I provide an aioquic server with multiple addresses: "::1:4433", "::1:4444" and "::1:4455" (same IP, different port)
I also run your pquic client with the multipath plugin enabled (same as in previous cases)
I then connect the client with IP address "::1:XXXX" to the aioquic server on IP address "::1:4433"
Will the aioquic server then be able to send data from two separate uniflows?
They are bound to the following 4-tuples:
("::1:4433" to "::1:XXXX")
("::1:4455" to "::1:XXXX")
(Notice that the client address is the same)

The reasoning behind this question is that it seems like the PQUIC client stops responding to the aioquic server after receiving stream data.

1. The handshake completes normally, and the client sends mp_new_connection_ids along with add_address frames to the server.

2. The server then creates the context for two additional sending uniflows, and binds them to the same destination IP address of the client, while having a different source IP address from the server (The 2 tuples seen above).

3. The client is able to parse the two path challenges that are received from the server directly after sending those frames, and responds accordingly with a path response twice over its initial uniflow (one for each path challenge, see the "challenge and accept.png" screenshot below).
   One Note: the client does seem to log an event named "peer_address_changed", but i am not sure about its meaning.
   

4. The server then starts to send stream data to the client: 3 1-RTT packets containing Stream 0 data over uniflow 0, and a number of 1-RTT packets, also containing stream 0 data over uniflows 1 and 2.

However, the client only logs the reception of all the packets sent over uniflow 0, and only the first two packets that are sent over uniflow 1, no packets are received from uniflow 2 (see the "stop responding.png" screenshot below).

I can't seem to figure out why the client just stops responding after receiving the two packets from uniflow 1. The client never
sends back an ack regarding packets that carry stream frames.

I have included the QLogs from the aioquic server (AMPs_large.qlog) and the PQUIC client (PMPc_large.qlog), as well as the PQUIC client console log (PMPc_large.txt) in Logs.zip

I have also heavily modified the PMPc_large.qlog file, due to it not generating correctly:
-translated all "onertt"-packet_types to "1rtt".
-Added "configuration": {"time_offset": 0, "time_units": "us"}, "common_fields": {"group_id": "1", "ODCID": "1", "reference_time": XXXXXXXX}, "event_fields": ["relative_time", "category", "event_type", "trigger", "context", "data"] to the end of the qlog, since they were not added upon closing the connection.
-changed "reference_time" from XXXXXXXX to the same reference time as the aioquic qlog-value, which is 1609341639759941.
-changed some packet numbers to be able to identify which uniflow is sending the data (100000+ is uniflow 1, 200000+ is uniflow 2).

It is also important to note that the console log of the PQUIC client doesn't reflect these modifications. At the end of the log the client receives two packets with sequence values 1 and 2, these are respectively the two packets that are received from uniflow 1, and should match the packet number values of 100001 and 100002.

Adding plugins insreases transmission time.

Hi,
I ran picoquicdemo with different plugins. But I observed that the plugins which should have decreased the transmission time of the data actually end up increasing time taken.

picoquicdemo with no plugin took 8755.857 ms for around 100MB of data.

picoquicdemo with datagram plugin took 9469.609 ms for around 100MB of data.

picoquicdemo with datagram, fec and multipath plugins took 14611.648 ms for around 100MB of data.

Is it supposed to happen like this or there is something that I'm doing wrong?
Am I missing out anything important regarding the plugins?

Incorrect way of prunning the retransmission queue.

Hi,
During testing commit 841c822, we discovered that the PQUIC server uses Picoquic code that does not correctly prune the retransmission queue in picoquic/sender.c:picoquic_implicit_handshake_ack().

The pruning direction on the retransmission queue is opposite to how the packets are added in picoquic/sender.c:picoquic_queue_for_retransmit().

Therefore the picoquic_packet_t* p_next = p->next_packet; in picoquic/sender.c:picoquic_implicit_handshake_ack() should be picoquic_packet_t* p_next = p->previous_packet;

Running picoquicvpn

FATAL ERROR: no protocol operation with id get_message_socket and hash 1748423259152196505
I always get this error when I'm trying to get the ./picoquicvpn to work.
I am using Debian 9 with Kernel 4.9.0.7
I followed the guide on pquic.org -> ./picoquicdemo and the fec plugin works without any problem.

How to use PQUIC with multipath plugin to transport in Internet

HI, I use a host with 2 wireless NICs to run "picoquicdemo" as the client.Local address of these NICs are "192.168.8.109" and "192.168.16.108". I run a "picoquicdemo" as the server in a cloud server. After transport, I only see packets from one path whose local address is "192.168.8.109".
The log in server says it can receive the ADD_ADDRESS frame of another local address:

c0ee25bf2974cec0: Receiving packet type: 6 (1rtt protected phi0), S0,
c0ee25bf2974cec0:     <af711b59bc472944>, Seq: 0 (0)
c0ee25bf2974cec0:     Decrypted 1065 bytes
c0ee25bf2974cec0:     MP NEW CONNECTION ID for Uniflow 0x01 CID: 0x972b5096415f01ff, b0708ea45006a624e4aaedd4212b6827
c0ee25bf2974cec0:     MP NEW CONNECTION ID for Uniflow 0x02 CID: 0x872e3f2206499d88, a0de3583edf0446a33d724b27429434d
c0ee25bf2974cec0:     ADD ADDRESS with ID 0x01 Address: 117.132.196.117
c0ee25bf2974cec0:     MP ACK for path 0x00 (nb=0), 0
c0ee25bf2974cec0:     padding, 987 bytes

also send a path challenge frame

c0ee25bf2974cec0: Sending packet type: 6 (1rtt protected phi0), S1,
c0ee25bf2974cec0:     <972b5096415f01ff>, Seq: 0 (0)
c0ee25bf2974cec0:     Prepared 9 bytes
c0ee25bf2974cec0:     path_challenge: 77aea8399346831a

But client can not receive this path challenge.
I also tried to config policy routing and NAT, but it didn't work.
How should I do to make two path established successfully in Internet? Thanks :）

Mac OS build errors

I am getting build errors with what seems to be IDE truncated lines that got checked in with a "$" and the rest of the line missing. I can't yet follow the ifdefs to understand the correct fix.

pquic/picoquic/picosocks.c

Line 514 in 841c822

if (addr_from->sa_family == AF_INET && length > PICOQUIC_INITIAL_MTU_IP$

/usr/bin/ld: cannot find -lprofiler

tried to build and facing below error

[ 52%] Linking C executable picoquicdemo
/usr/bin/ld: cannot find -lprofiler: No such file or directory
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/picoquicdemo.dir/build.make:124: picoquicdemo] Error 1
make[1]: *** [CMakeFiles/Makefile2:147: CMakeFiles/picoquicdemo.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

How to ensure pquic use

Fatal error when using -q <name> argument on picoquicdemo

When running the picoquicdemo along with the -q argument, the endpoint throws a fatal error.
Error message: FATAL ERROR: no protocol operation with id set_qlog_file and hash hash_value

Retention of the unused encryption keys

Hi,

During testing commit 841c822, we discovered that the PQUIC server does not discard the unused encryption keys after moving to a new encryption level.

The unused encryption keys, such as the Initial key, should be discarded once QUIC has moved to the Handshake encryption level (https://www.rfc-editor.org/rfc/rfc9001#name-discarding-unused-keys), same as the Handshake key after moving to the 1-RTT encryption level. An attacker can obtain the Initial key by capturing the Initial packets sent between the client and server during the connection establishment. If the server/client still accepts and processes Initial packets at any encryption level, an attacker can disrupt the connection by sending a CONNECTION_CLOSE frame in an Initial packet to the server/client, causing a connection close that is not initiated by either the client or the server.

Running pquic picoquicdemo with multipath plugin throws error

Running the picoquicdemo with the multipath plugin results in a server exit, and giving the following output:

./picoquicdemo -P plugins/multipath/multipath.plugin
Starting PicoQUIC server on port 4443, server name = ::, just_once = 0, hrr= 0, 1 local plugins and 0 both plugins
local plugin plugins/multipath/multipath.plugin
Unrecognized plugin option: "param"
Impossible to parse first plugin line
Cannot extract plugin id
Error when setting local plugins to inject
Server exit, ret = 1
Server exit with code = 1