emqx / quic

QUIC protocol for Erlang & Elixir

Quicer

QUIC (Next-generation transport protocol) erlang library.

msquic NIF binding.

Project Status: Preview

OS Support

OS	Status
Linux	Supported
macOS	Supported
Windows	Help Needed

Add to your project

rebar.config

{deps, [
    {quicer, {git, "https://github.com/qzhuyan/quic.git", {tag, "0.1.0"}}},
    ...

Examples

Ping Pong server and client

Server

application:ensure_all_started(quicer),
Port = 4567,
LOptions = [ {certfile, "cert.pem"}
           , {keyfile,  "key.pem"}
           , {alpn, ["sample"]}
           , {peer_bidi_stream_count, 1}
             ],
{ok, L} = quicer:listen(Port, LOptions),
{ok, Conn} = quicer:accept(L, [], 120000),
{ok, Conn} = quicer:handshake(Conn),
{ok, Stm} = quicer:accept_stream(Conn, []),
receive {quic, <<"ping">>, Stm, _Props} -> ok end,
{ok, 4} = quicer:send(Stm, <<"pong">>),
quicer:close_listener(L).

Client

application:ensure_all_started(quicer),
Port = 4567,
{ok, Conn} = quicer:connect("localhost", Port, [{alpn, ["sample"]}, {verify, none}], 5000),
{ok, Stm} = quicer:start_stream(Conn, []),
{ok, 4} = quicer:send(Stm, <<"ping">>),
receive {quic, <<"pong">>, Stm, _Props} -> ok end,
ok = quicer:close_connection(Conn).

Try connect to Google with QUIC transport

%% Connect to google and disconnect, 
%% You could also tweak the parameters to see how it goes
{ok, Conn} = quicer:connect("google.com", 443, [{alpn, ["h3"]}, 
                            {verify, verify_peer}, 
                            {peer_unidi_stream_count, 3}], 5000),
quicer:shutdown_connection(Conn).

More examples in test dir

refer to test dir.

Documentation

Get Started

1. Understand the handles and the ownership in Terminology

2. Then check how to receives the data and signals: Messages

3. Read more in msquic doc

Offline hex doc

$ make doc
$ firefox doc/index.html

Dependencies

1. OTP25+
2. rebar3
3. cmake3.16+

Build and test

Dev mode

$ make ci

Troubleshooting

Log to stdout

Debug log could be enabled to print to stdout with the envvar QUIC_LOGGING_TYPE=stdout

QUIC_LOGGING_TYPE=stdout make

%% Debug one testcase
QUIC_LOGGING_TYPE=stdout rebar3 ct --suite test/quicer_connection_SUITE.erl --case tc_conn_basic_verify_peer

Decrypt traffic with Wireshark

Client could specify the connect param sslkeylogfile to record tls secrets for wireshark to decrypt.

    {ok, Conn} = quicer:connect(
        "google.com",
        443,
        [
            {verify, verify_peer},
            {sslkeylogfile, "/tmp/SSLKEYLOGFILE"},
            {peer_unidi_stream_count, 3},
            {alpn, ["h3"]}
        ],
        5000
    )

License

Apache License Version 2.0

quic's Issues

Elixir Error on build.sh

I have a error when try use mix.compile

Error Log:

/build.sh 'v2.2.3'
Current option settings
errexit         on
noglob          off
ignoreeof       off
interactive     off
monitor         off
noexec          off
stdin           off
xtrace          off
verbose         off
vi              off
emacs           off
noclobber       off
allexport       off
notify          off
nounset         on
privileged      off
nolog           off
debug           off
Patching Msquic 2.2.3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   697  100   697    0     0   1882      0 --:--:-- --:--:-- --:--:--  1878
1 out of 1 hunk FAILED
Skip patching patches/73a11d7bdc724432964a2d4bdc4211ed29823380.patch, already applied
-- CMAKE Version: 3.22.1
-- Source Dir: /home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/msquic
-- Host System name: Linux
-- Setting policy 0091
-- System name: Linux
-- System version: 6.2.0-35-generic
-- Platform version: 
-- Build type: RelWithDebInfo
-- QUIC Platform: linux
-- Version Build ID: 0
-- Version Suffix: -private
-- Enabling OpenSsl configuration tests
-- Enabling CA file tests
-- QUIC_ENABLE_LOGGING is false. Disabling logging
-- Found libatomic: /usr/lib/x86_64-linux-gnu/libatomic.so.1
-- Found libnuma: /usr/lib/x86_64-linux-gnu/libnuma.so.1
-- numa.h not found. If build fails, install libnuma-dev
-- Enabling shared ephemeral port work around
-- Configuring for OpenSSL 1.1
-- Setting openssldir to /home/linuxbrew/.linuxbrew/etc/[email protected]
-- Configuring OpenSSL: /home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/msquic/submodules/openssl/config;CC=/usr/bin/cc;CXX=/usr/bin/c++ enable-tls1_3;no-makedepend;no-dgram;no-ssl3;no-psk;no-srp;no-zlib;no-egd;no-idea;no-rc5;no-rc4;no-afalgeng;no-comp;no-cms;no-ct;no-srp;no-srtp;no-ts;no-gost;no-dso;no-ec2m;no-tls1;no-tls1_1;no-tls1_2;no-dtls;no-dtls1;no-dtls1_2;no-ssl;no-ssl3-method;no-tls1-method;no-tls1_1-method;no-tls1_2-method;no-dtls1-method;no-dtls1_2-method;no-siphash;no-whirlpool;no-aria;no-bf;no-blake2;no-sm2;no-sm3;no-sm4;no-camellia;no-cast;no-md4;no-mdc2;no-ocb;no-rc2;no-rmd160;no-scrypt;no-seed;no-weak-ssl-ciphers;no-shared;no-tests;--openssldir="/home/linuxbrew/.linuxbrew/etc/[email protected]";--prefix=/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build/_deps/opensslquic-build/openssl
-- Configuring for OpenSSL
-- Configuring done
-- Generating done
-- Build files have been written to: /home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build
make[1]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[2]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[3]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[3]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[3]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
[  2%] Built target OpenSSL_Target
make[3]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
Consolidate compiler generated dependencies of target core
make[3]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
Consolidate compiler generated dependencies of target platform
make[3]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
[ 50%] Built target core
[ 68%] Built target platform
make[3]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
Consolidate compiler generated dependencies of target msquic
make[3]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
[ 70%] Built target msquic
make[3]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
Consolidate compiler generated dependencies of target quicer_static
make[3]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
[ 85%] Built target quicer_static
make[3]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
Consolidate compiler generated dependencies of target quicer_nif
make[3]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
[100%] Built target quicer_nif
make[2]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[1]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[1]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[2]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[3]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[3]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
[  2%] Built target OpenSSL_Target
make[3]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[3]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
[ 20%] Built target platform
make[3]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[3]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
[ 68%] Built target core
make[3]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[3]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
[ 70%] Built target msquic
make[3]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[3]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
[ 85%] Built target quicer_static
make[3]: Entrando no diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
make[3]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
[100%] Built target quicer_nif
make[2]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
Install the project...
-- Install configuration: "RelWithDebInfo"
-- Up-to-date: /usr/local/lib/libmsquic.so.2.2.3
-- Up-to-date: /usr/local/lib/libmsquic.so.2
-- Up-to-date: /usr/local/lib/libmsquic.so
-- Up-to-date: /usr/local/include/msquic.h
-- Up-to-date: /usr/local/include/msquic_winuser.h
-- Up-to-date: /usr/local/include/msquic_posix.h
-- Up-to-date: /usr/local/include/quic_sal_stub.h
-- Up-to-date: /usr/local/share/msquic/msquic-config.cmake
-- Up-to-date: /usr/local/share/msquic/msquic.cmake
-- Up-to-date: /usr/local/share/msquic/msquic-relwithdebinfo.cmake
-- Up-to-date: /home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/priv/libquicer_nif.so.0.0.300
-- Up-to-date: /home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/priv/libquicer_nif.so.1
-- Up-to-date: /home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/priv/libquicer_nif.so
make[1]: Saindo do diretório '/home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/c_build'
realpath: /home/sanonichan/projetos/matheuscamarques/quic_http3_server/http3_server_ex/deps/quic/_build/prod/lib/quicer/priv: Arquivo ou diretório inexistente
./build.sh: 96: [: !=: unexpected operator
===> Analyzing applications...
===> Compiling quicer

snabbkaffe. 0.17.3: case failed on master branch after we bump snabbkaffe

snabbkaffe may find new issue on master branch

Checks are

                   ?assert(?strict_causality(#{ ?snk_kind := debug
                                              , function := "connection_controlling_process"
                                              , tag := "exit"
                                              , resource_id := CRid
                                              },
                                             #{ ?snk_kind := debug
                                              , function := "resource_conn_down_callback"
                                              , tag := "start"
                                              , resource_id := CRid
                                              },
                                             Trace)),
                   %% check that it triggers an immediate connection shutdown
                   ?assert(?strict_causality(#{ ?snk_kind := debug
                                              , function := "resource_conn_down_callback"
                                              , tag := "end"
                                              , resource_id := CRid
                                              },
                                             #{ ?snk_kind := debug
                                              , context := "callback"
                                              , function := "ClientConnectionCallback"
                                              , tag := "event"
                                              , mark := ?QUIC_CONNECTION_EVENT_SHUTDOWN_COMPLETE
                                              , resource_id := CRid
                                              },
                                             Trace)),
                   %% check that client side immediate shutdown triggers a stream shutdown
                   ?assert(?causality(#{ ?snk_kind := debug
                                       , function := "resource_conn_down_callback"
                                       , tag := "end"
                                       , resource_id := CRid
                                       },
                                      #{ ?snk_kind := debug
                                       , context := "callback"
                                       , function := "ClientStreamCallback"
                                       , mark := ?QUIC_STREAM_EVENT_SEND_SHUTDOWN_COMPLETE
                                       , tag := "event"
                                       , resource_id := SRid
                                       },
                                      Trace)),
                   %% check that client side conn shutdown happens after stream shutdown
                   ?assert(?strict_causality(#{ ?snk_kind := debug
                                              , context := "callback"
                                              , function := "ClientStreamCallback"
                                              , mark := ?QUIC_STREAM_EVENT_SEND_SHUTDOWN_COMPLETE
                                              , tag := "event"
                                              , resource_id := SRid
                                              },
                                             #{ ?snk_kind := debug
                                              , context := "callback"
                                              , function := "ClientConnectionCallback"
                                              , tag := "event"
                                              , mark := ?QUIC_CONNECTION_EVENT_SHUTDOWN_COMPLETE
                                              , resource_id := CRid
                                              },
                                             Trace)),
                   %% check that client side immediate shutdown triggers a close at server side
                   ?assert(?strict_causality( #{ ?snk_kind := quic_shutdown
                                               , module := quicer_conn_acceptor
                                               , '~meta' := #{pid := _PID}},
                                              #{ ?snk_kind := quic_closed
                                               , module := quicer_conn_acceptor
                                               ,'~meta' := #{pid := _PID}},
                                              Trace))

Traces are

Trace is [#{'$kind' => '$trace_begin','~meta' => #{time => -576460699595632}},
          #{'$kind' => debug,context => "callback",
            function => "ServerListenerCallback",mark => 0,
            resource_id => 140242026360512,tag => "fast_conn",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ServerConnectionCallback",mark => 9,
            resource_id => 140242026360512,tag => "event",'~meta' => #{}},
          #{'$kind' => quic_new_conn,
            conn => #Ref<0.1073225519.3176529920.194733>,
            module => quicer_conn_acceptor,
            '~meta' =>
                #{gl => <0.572.0>,node => nonode@nohost,pid => <0.1523.0>,
                  time => -576460699593970}},
          #{'$kind' => debug,context => "nif",function => "async_handshake_1",
            mark => 0,resource_id => 140242026360512,tag => "start",
            '~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ClientConnectionCallback",mark => 7,
            resource_id => 140242026369136,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ClientConnectionCallback",mark => 10,
            resource_id => 140242026369136,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ClientConnectionCallback",mark => 0,
            resource_id => 140242026369136,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ClientConnectionCallback",mark => 9,
            resource_id => 140242026369136,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ClientStreamCallback",mark => 0,
            resource_id => 140242158217008,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ServerConnectionCallback",mark => 0,
            resource_id => 140242026360512,tag => "event",'~meta' => #{}},
          #{'$kind' => quic_connected_slow,
            conn => #Ref<0.1073225519.3176529920.194733>,
            module => quicer_conn_acceptor,
            '~meta' =>
                #{gl => <0.572.0>,node => nonode@nohost,pid => <0.1523.0>,
                  time => -576460699586243}},
          #{'$kind' => debug,context => "callback",
            function => "ClientStreamCallback",mark => 2,
            resource_id => 140242158217008,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ServerConnectionCallback",mark => 9,
            resource_id => 140242026360512,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ServerConnectionCallback",mark => 6,
            resource_id => 140242026360512,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ServerStreamCallback",mark => 1,
            resource_id => 140241889423504,tag => "event",'~meta' => #{}},
          #{'$kind' => new_stream,module => quicer_stream,
            stream => #Ref<0.1073225519.3176529920.194736>,
            '~meta' =>
                #{gl => <0.572.0>,node => nonode@nohost,pid => <0.1556.0>,
                  time => -576460699585966}},
          #{'$kind' => stream_data,module => quicer_stream,
            stream => #Ref<0.1073225519.3176529920.194736>,
            '~meta' =>
                #{gl => <0.572.0>,node => nonode@nohost,pid => <0.1556.0>,
                  time => -576460699585959}},
          #{'$kind' => debug,context => "callback",
            function => "ServerStreamCallback",mark => 2,
            resource_id => 140241889423504,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ClientConnectionCallback",mark => 14,
            resource_id => 140242026369136,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ClientStreamCallback",mark => 1,
            resource_id => 140242158217008,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "nif",
            function => "connection_controlling_process",
            mark => 140242395753536,resource_id => 140242026369136,
            tag => "enter",'~meta' => #{}},
          #{'$kind' => debug,context => "nif",
            function => "connection_controlling_process",
            mark => 140242395753536,resource_id => 140242026369136,
            tag => "exit",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "resource_conn_down_callback",mark => 94726386207240,
            resource_id => 140242026369136,tag => "start",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "resource_conn_down_callback",mark => 94726386207240,
            resource_id => 140242026369136,tag => "end",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ClientStreamCallback",mark => 6,
            resource_id => 140242158217008,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ClientStreamCallback",mark => 7,
            resource_id => 140242158217008,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ServerConnectionCallback",mark => 2,
            resource_id => 140242026360512,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ServerStreamCallback",mark => 6,
            resource_id => 140241889423504,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ServerStreamCallback",mark => 7,
            resource_id => 140241889423504,tag => "event",'~meta' => #{}},
          #{'$kind' => quic_shutdown,module => quicer_conn_acceptor,
            '~meta' =>
                #{gl => <0.572.0>,node => nonode@nohost,pid => <0.1523.0>,
                  time => -576460699585187}},
          #{'$kind' => debug,context => "callback",
            function => "ServerConnectionCallback",mark => 3,
            resource_id => 140242026360512,tag => "event",'~meta' => #{}},
          #{'$kind' => debug,context => "callback",
            function => "ServerConnectionCallback",mark => 0,
            resource_id => 140242026360512,tag => "shutdown_complete",
            '~meta' => #{}},
          #{'$kind' => quic_closed,module => quicer_conn_acceptor,
            '~meta' =>
                #{gl => <0.572.0>,node => nonode@nohost,pid => <0.1523.0>,
                  time => -576460699585119}},
          #{'$kind' => '$trace_end','~meta' => #{time => -576460699585102}}], 

*** System report during quicer_snb_SUITE:tc_conn_owner_down/1 2022-01-12 09:40:24.688 ***
=CRITICAL REPORT==== 12-Jan-2022::09:40:24.687945 ===
"check stage" failed: error
{panic,#{'$kind' => "Causality violation",
         causes_without_effect =>
             [#{'$kind' => debug,context => "callback",
                function => "resource_conn_down_callback",
                mark => 94726386207240,resource_id => 140242026369136,
                tag => "end",'~meta' => #{}}],
         effects_without_cause => []}}
Stacktrace: [{snabbkaffe,causality,5,
                         [{file,"/home/runner/work/quic/quic/_build/default/lib/snabbkaffe/src/snabbkaffe.erl"},
                          {line,469}]},
             {quicer_snb_SUITE,'-tc_conn_owner_down/1-fun-20-',2,
                               [{file,"/home/runner/work/quic/quic/test/quicer_snb_SUITE.erl"},
                                {line,367}]},
             {snabbkaffe,run_trace_spec,3,
                         [{file,"/home/runner/work/quic/quic/_build/default/lib/snabbkaffe/src/snabbkaffe.erl"},
                          {line,640}]},
             {snabbkaffe,'-check_stage/3-lc$^0/1-0-',3,
                         [{file,"/home/runner/work/quic/quic/_build/default/lib/snabbkaffe/src/snabbkaffe.erl"},
                          {line,621}]},
             {snabbkaffe,check_stage,3,
                         [{file,"/home/runner/work/quic/quic/_build/default/lib/snabbkaffe/src/snabbkaffe.erl"},
                          {line,621}]},
             {quicer_snb_SUITE,tc_conn_owner_down,1,
                               [{file,"/home/runner/work/quic/quic/test/quicer_snb_SUITE.erl"},
                                {line,329}]},
             {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
             {test_server,run_test_case_eval1,6,
                          [{file,"test_server.erl"},{line,1292}]}]

macOS: Code Signature Invalid

thread 4 Crashed:: 1_scheduler
0   dyld                          	       0x100a12504 dyld3::MachOFile::compatibleSlice(Diagnostics&, void const*, unsigned long, char const*, dyld3::Platform, bool, dyld3::GradedArchs const&) + 76
1   dyld                          	       0x1009f695c invocation function for block in dyld4::JustInTimeLoader::makeJustInTimeLoaderDisk(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, bool, unsigned int) + 96
2   dyld                          	       0x1009f695c invocation function for block in dyld4::JustInTimeLoader::makeJustInTimeLoaderDisk(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, bool, unsigned int) + 96
3   dyld                          	       0x1009fbd58 dyld4::SyscallDelegate::withReadOnlyMappedFile(Diagnostics&, char const*, bool, void (void const*, unsigned long, bool, dyld4::FileID const&, char const*) block_pointer) const + 132
4   dyld                          	       0x1009f68c8 dyld4::JustInTimeLoader::makeJustInTimeLoaderDisk(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, bool, unsigned int) + 204
5   dyld                          	       0x1009f06d0 invocation function for block in dyld4::Loader::getLoader(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&) + 1384
6   dyld                          	       0x1009efbc0 dyld4::Loader::forEachResolvedAtPathVar(dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, dyld4::ProcessConfig::PathOverrides::Type, bool&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) + 780
7   dyld                          	       0x1009ef7ec invocation function for block in dyld4::Loader::forEachPath(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) + 148
8   dyld                          	       0x1009e5ee4 dyld4::ProcessConfig::PathOverrides::forEachPathVariant(char const*, dyld3::Platform, bool, bool&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) const + 672
9   dyld                          	       0x1009ef740 dyld4::Loader::forEachPath(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&, void (char const*, dyld4::ProcessConfig::PathOverrides::Type, bool&) block_pointer) + 172
10  dyld                          	       0x1009eff60 dyld4::Loader::getLoader(Diagnostics&, dyld4::RuntimeState&, char const*, dyld4::Loader::LoadOptions const&) + 864
11  dyld                          	       0x100a01ca8 invocation function for block in dyld4::APIs::dlopen_from(char const*, int, void*) + 248
12  dyld                          	       0x1009e6a7c dyld4::RuntimeState::withLoadersWriteLock(void () block_pointer) + 100
13  dyld                          	       0x100a019b4 dyld4::APIs::dlopen_from(char const*, int, void*) + 472
14  beam.smp                      	       0x1005b1cac erts_sys_ddll_open_noext + 44
15  beam.smp                      	       0x1005b1c44 erts_sys_ddll_open + 132
16  beam.smp                      	       0x1005431a4 erts_load_nif + 716
17  libsystem_pthread.dylib       	       0x18eaf426c _pthread_start + 148
18  libsystem_pthread.dylib       	       0x18eaef08c thread_start + 8

connect() hangs indefinitely

Hi,
I have a situation in our local system where quicer:connect() with a timeout of 10000 hangs indefinitely.
The stack dump is:

 {messages,
     [connect,
      {'$gen_call',
          {<0.330.0>,#Ref<0.787081997.2662072321.82090>},
          {new_up_stream,<0.330.0>}},
      {'$gen_call',
          {<0.331.0>,#Ref<0.787081997.2662072321.82091>},
          {new_up_stream,<0.331.0>}}]},
 {current_stacktrace,
     [{quicer,flush,2,[{file,"src/quicer.erl"},{line,981}]},
      {quicer,connect,4,[{file,"src/quicer.erl"},{line,227}]},
      {volga_q_connector,conn_list,3,
          [{file,"src/volga_q_connector.erl"},{line,134}]},
      {volga_q_connector,handle_info,2,
          [{file,"src/volga_q_connector.erl"},{line,173}]},
      {gen_server,try_dispatch,4,
          [{file,"src/gen_server.erl"},{line,1123}]},
      {gen_server,handle_msg,6,
          [{file,"src/gen_server.erl"},{line,1200}]},
      {proc_lib,init_p_do_apply,3,
          [{file,"proc_lib.erl"},{line,240}]}]}]

The messages in the queue are of course local to our system. I've tried to reproduce this in the quicer tests to no avail. Regardless, the stackdump is proof of error imho. It's a reconnect situation, where a previous connection to another host has just died.
Opinions please.

Fail to compile on M1 Mac

Attempting to compile emqtt and getting failure:

/Users/XXX/XXX/XXX/_build/default/lib/quicer/msquic/src/platform/datapath_kqueue.c:1984:14: error: variable 'TailBuffer' set but not used [-Werror,-Wunused-but-set-variable]
    uint8_t* TailBuffer = SendData->Buffers[SendData->BufferCount - 1].Buffer;
             ^
1 error generated.
make[3]: *** [msquic/src/platform/CMakeFiles/platform.dir/datapath_kqueue.c.o] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [msquic/src/platform/CMakeFiles/platform.dir/all] Error 2
make[1]: *** [all] Error 2

I am using emqtt on tag 1.4.7. I also did try to clone emqx/quic with same result.

Calling quicer from iex

I'm currently trying to run the example and open a connection from my iex shell.

:quicer.listen(22, %{:cert => "./domain.pem", :key => "./domain.key", :alpn => ["sample"], :verify => :none})
{:error, :quic_tls}

I'm generating the keys with:

openssl genrsa -out keypair.pem 2048
openssl x509 -in domain.crt -out domain.pem -outform PEM

I think either I'm using broken keys or I'm incorrectly calling Erlang from Elixir.

A quick example of how to do it correctly would be appreciated.

Build fails: too many filenames given

Build fails:

#12 182.0 [ 82%] Building C object CMakeFiles/quicer_static.dir/c_src/quicer_ctx.c.o
#12 182.0 cc1: error: too many filenames given; type 'cc1 --help' for usage
#12 182.0 cc1: fatal error: CMakeFiles/quicer_static.dir/c_src/quicer_nif.c.d: No such file or directory
#12 182.0 compilation terminated.

GCC-11 but even GCC-10 fails
CMake 3.20

Parallelism issues

Hello, I wanted to ask if you have done some (recent) testing with many connections in parallel. In Cowboy if I run the tests sequentially they run mostly fine, but if I run them in parallel I always get a few that seem to get stuck. I am not yet sure where exactly, but it happens soon after the connection has been opened and both client and server have started their unidi streams (3 each). So it likely happens either when the client starts the first bidi stream, sends data on it, or waits for the response; or when the server receives this bidi stream, the data and/or sends the response.

Another issue I had that I've seen resolved is msquic refusing connections when overloaded; here increasing the number of acceptors and having the client retry seems to help.

race cond: quicer 0.0.303 still MsQuic->StreamSend after stream is closed

44	./nptl/pthread_kill.c: No such file or directory.
Core was generated by `/home/runner/work/_temp/.setup-beam/rebar3/bin/rebar3 -B -sbtu -A1 -- -root /op'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (no_tid=0, signo=6, 
    threadid=139627331290688) at ./nptl/pthread_kill.c:44
[Current thread is 1 (Thread 0x7efd857b8640 (LWP 10709))]
#0  __pthread_kill_implementation (no_tid=0, signo=6, 
    threadid=139627331290688) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=139627331290688)
    at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=139627331290688, signo=signo@entry=6)
    at ./nptl/pthread_kill.c:89
#3  0x00007efdc9442476 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/posix/raise.c:26
#4  0x00007efdc94287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007efd2d76b00d in quic_bugcheck (
    File=File@entry=0x7efd2d9309a8 "/home/runner/work/quic/quic/msquic/src/core/api.c", Line=Line@entry=1027, 
    Expr=Expr@entry=0x7efd2d92e932 "!Stream->Flags.HandleClosed")
    at /home/runner/work/quic/quic/msquic/src/platform/platform_posix.c:93
#6  0x00007efd2d7193fe in MsQuicStreamSend (Handle=<optimized out>, 
    Buffers=0x7efd7000e580, BufferCount=<optimized out>, 
    Flags=QUIC_SEND_FLAG_NONE, ClientSendContext=0x7efd7000e560)
    at /home/runner/work/quic/quic/msquic/src/core/api.c:1027
#7  0x00007efd2d6f0fe0 in send3 (env=0x7efd857b3bf0, argc=3, 
    argv=0x7efd857b3d00)
    at /home/runner/work/quic/quic/c_src/quicer_stream.c:715
#8  0x000055a0f7636508 in beam_jit_call_nif (c_p=0x55a0f8d79320, 
    I=<optimized out>, reg=0x7efd857b3d00, fp=0x7efd2d6f0c93 <send3>, 
    NifMod=<optimized out>) at beam/jit/beam_jit_common.cpp:563

macOS: test failure

lots of tests are failed due to

 {error,
         {{{badmatch,{error,config_error,tls_error}},
....

This is because generating cert files are failed with LibreSSL

unknown option -addext
req [options] outfile
where options  are
 -inform arg    input format - DER or PEM
 -outform arg   output format - DER or PEM
 -in arg        input file
 -out arg       output file
 -text          text form of request
 -pubkey        output public key
 -noout         do not output REQ
 -verify        verify signature on REQ
 -modulus       RSA modulus
 -nodes         don't encrypt the output key
 -subject       output the request's subject
 -passin        private key password source
 -key file      use the private key contained in file
 -keyform arg   key file format
 -keyout arg    file to send the key to
 -newkey rsa:bits generate a new RSA key of 'bits' in size
 -newkey dsa:file generate a new DSA key, parameters taken from CA in 'file'
 -newkey ec:file generate a new EC key, parameters taken from CA in 'file'
 -[digest]      Digest to sign with (md5, sha1, md4)
 -config file   request template file.
 -subj arg      set or modify request subject
 -multivalue-rdn enable support for multivalued RDNs
 -new           new request.
 -batch         do not ask anything during request generation
 -x509          output a x509 structure instead of a cert. req.
 -days          number of days a certificate generated by -x509 is valid for.
 -set_serial    serial number to use for a certificate generated by -x509.
 -newhdr        output "NEW" in the header lines
 -asn1-kludge   Output the 'request' in a format that is wrong but some CA's
                have been reported as requiring
 -extensions .. specify certificate extension section (override value in config file)
 -reqexts ..    specify request extension section (override value in config file)
 -utf8          input characters are UTF8 (default ASCII)
 -nameopt arg    - various certificate name options
 -reqopt arg    - various request text options



*** User 2023-03-17 14:58:37.718 ***[🔗](file:///Users/ezhuwya/repo/quic/_build/test/logs/[email protected]_14.58.37/lib.quicer.logs/run.2023-03-17_14.58.37/quicer_suite.init_per_suite.html#e-2)
/Users/ezhuwya/repo/quic/_build/test/lib/quicer/test/quicer_SUITE_data/server.csr: No such file or directory


*** User 2023-03-17 14:58:37.724 ***[🔗](file:///Users/ezhuwya/repo/quic/_build/test/logs/[email protected]_14.58.37/lib.quicer.logs/run.2023-03-17_14.58.37/quicer_suite.init_per_suite.html#e-3)
unknown option -addext
req [options] outfile
where options  are
 -inform arg    input format - DER or PEM
 -outform arg   output format - DER or PEM
 -in arg        input file
 -out arg       output file
 -text          text form of request
 -pubkey        output public key
 -noout         do not output REQ
 -verify        verify signature on REQ
 -modulus       RSA modulus
 -nodes         don't encrypt the output key
 -subject       output the request's subject
 -passin        private key password source
 -key file      use the private key contained in file
 -keyform arg   key file format
 -keyout arg    file to send the key to
 -newkey rsa:bits generate a new RSA key of 'bits' in size
 -newkey dsa:file generate a new DSA key, parameters taken from CA in 'file'
 -newkey ec:file generate a new EC key, parameters taken from CA in 'file'
 -[digest]      Digest to sign with (md5, sha1, md4)
 -config file   request template file.
 -subj arg      set or modify request subject
 -multivalue-rdn enable support for multivalued RDNs
 -new           new request.
 -batch         do not ask anything during request generation
 -x509          output a x509 structure instead of a cert. req.
 -days          number of days a certificate generated by -x509 is valid for.
 -set_serial    serial number to use for a certificate generated by -x509.
 -newhdr        output "NEW" in the header lines
 -asn1-kludge   Output the 'request' in a format that is wrong but some CA's
                have been reported as requiring
 -extensions .. specify certificate extension section (override value in config file)
 -reqexts ..    specify request extension section (override value in config file)
 -utf8          input characters are UTF8 (default ASCII)
 -nameopt arg    - various certificate name options
 -reqopt arg    - various request text options

Listener failed to start while IPv6 is disabled.

Some user reports that listener fails to start while IPv6 is disabled in Kernel.

{unknown_quic_status,97}

As in
emqx/emqx#6683 (reply in thread)

when he disable ipv6 in kernel args:

    GRUB_CMDLINE_LINUX="ipv6.disable=1 ro crashkernel=auto ..."

But I could not reproduce it if I only disable ipv6 via sysconfig

sysctl -w net.ipv6.conf.all.disable_ipv6=1

Getting the negotiated ALPN protocol

Hello! Another thing I noticed was missing is the equivalent of ssl:negotiated_protocol to retrieve the ALPN protocol that was selected. For HTTP/3 for example the selected protocol should be "h3".

why controlling_process need Locks a mutex

The controlling_process nif implementation is Locks a mutex，According to my understanding, controlling_process can only call this function if it originally owns the process, there will be no concurrent operation, Should there be no lock operation?

caught a ethr_mutex_lock(): Invalid argument (22)

%%% quicer_SUITE: .............................................................
../include/internal/ethr_mutex.h:656: Fatal error in ethr_mutex_lock(): Invalid argument (22)
make: *** [ct] Abort trap: 6
%%% quicer_snb_SUITE:

msquic 2.2 ca certificate file setting in SSL_CTX

microsoft/msquic@a1762b1

Build fails in NixOS

Enviroment

Version: e850f4feaa907b1e6ea653ff20bde46f4ca33815 (latest commit)

Dockerfile with reproduction steps:

FROM nixos/nix
COPY . .
RUN nix-shell -p gcc gnumake cmake rebar3 erlangR23 perl openssl --run make

Actual behavior

Build fails with

/nix/store/5ddb4j8z84p6sjphr0kh6cbq5jd12ncs-binutils-2.35.1/bin/ld: cannot find -lATOMIC-NOTFOUND
collect2: error: ld returned 1 exit status
make[3]: *** [msquic/src/bin/CMakeFiles/msquic.dir/build.make:107: msquic/bin/Release/libmsquic.so] Error 1

Desired behavior

Build passes

Overview

This is caused by buggy CMakeLists.txt in msquic. This bug was fixed in latest versions of msquic

Ref:
https://github.com/microsoft/msquic/blob/v1.8.0/src/inc/CMakeLists.txt#L34
https://github.com/microsoft/msquic/blob/main/src/inc/CMakeLists.txt#L40

EXIT message not handled in `quicer_conn_acceptor` ?

** {function_clause,
       [{quicer_conn_acceptor,handle_info,
            [{'EXIT',<0.10630.0>,killed},
             {state,#Ref<0.3483442753.2112225282.217011>,<0.10513.0>,
                 #Ref<0.3483442753.2112225280.108660>,
                 {#{alpn => ["mqtt"],
                    cert =>
                        "/__w/emqx/emqx/apps/emqx/_build/test/lib/emqx/etc/certs/cert.pem",
                    conn_acceptors => 16,idle_timeout_ms => 15000000,
                    key =>
                        "/__w/emqx/emqx/apps/emqx/_build/test/lib/emqx/etc/certs/key.pem"},
                  #{conn_callback => emqx_quic_connection,limiter => #{},
                    listener => {quic,default},
                    peer_bidi_stream_count => 10,peer_unidi_stream_count => 1,
                    zone => default},
                  [{stream_callback,emqx_quic_stream}]},
                 emqx_quic_connection,
                 #{conn_callback => emqx_quic_connection,limiter => #{},
                   listener => {quic,default},
                   peer_bidi_stream_count => 10,peer_unidi_stream_count => 1,
                   slow_start => true,
                   stream_opts => [{stream_callback,emqx_quic_stream}],
                   zone => default},
                 true}],

can't compile

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
ATOMIC
linked by target "quicer_static" in directory /usr/src/emqx5/_build/default/lib/quicer

-- Generating done
CMake Generate step failed. Build files cannot be regenerated correctly.
make[1]: *** [Makefile:11: build-nif] Error 1
make[1]: Leaving directory '/usr/src/emqx5/_build/default/lib/quicer'
===> Hook for compile failed!

accept_stream hangs when connection dies

If client end dies, accept_stream hangs with a {quic, closed, Conn} message in the message queue.
A possible fix is:

accept_stream(Conn, Opts, Timeout) when is_list(Opts) ->
  accept_stream(Conn, maps:from_list(Opts), Timeout);
accept_stream(Conn, Opts, Timeout) when is_map(Opts) ->
  % @todo make_ref
  % @todo error handling
  NewOpts = maps:merge(default_stream_opts(), Opts),
  case quicer_nif:async_accept_stream(Conn, NewOpts) of
    {ok, Conn} ->
      receive
        {quic, new_stream, Stream} ->
          {ok, Stream};
        {quic, closed, Conn} ->
          {error, connection_closed}

      after Timeout ->
          {error, timeout}
      end;
    {error, _} = E ->
      E
  end.

Not using TLS

Does allow_insecure option permits the use of unencrypted communication?
I was trying to call quicer:listen/2 without cert and key, but that doesn't work:

2> quicer:listen(12346, #{alpn => ["sample"], verify => none, allow_insecure => true}).
{error,badarg}

Is this supposed to be analogous to setting the QUIC_PARAM_CONN_DISABLE_1RTT_ENCRYPTION option?

multiple connects fail

I'm seeing an error I cannot understand. A client that issues a series of connect() calls, it always fails on connection number 495.

quicer.tar.gz

The little test program attached can be used to reproduce. Run qt.erl and issue {connect, 500}. in the client

quicer:listen/2 error

How can I find out the reason for the failure to start a listener? Trying to run in the repository root:

$ rebar3 shell

1> application:ensure_all_started(quicer).
{ok,[]}
2> quicer:listen(12345, #{certfile => "test/example/server.pem", keyfile => "test/example/server.key", alpn => ["sample"], verify => none, allow_insecure => true}).
{error,config_error,tls_error}

example test fail

according to:
#129 (comment)

`quicer:open_lib()` crashes with `noproc` if called early in node boot

Taking a look at using quic in an "Alternative Carrier for the Erlang Distribution", and it appears that file:read_file_info/1 cannot be called that early in node startup. The crash I see originates from

quic/src/quicer_nif.erl

Line 217 in 0742e6d

case file:read_file_info(PrivDir) of

and results in the following error:

The on_load function for module quicer_nif returned:
{noproc,{gen_server,call,[file_server_2,{read_file_info,[...]},infinity]}}

Can't get stream ID after receiving stream_closed message

This no longer works on main:

receive
         {quic, stream_closed, StreamRef, Flags} →
             {ok, StreamID} = quicer:get_stream_id(StreamRef),
             ...
end

Before I could get the StreamID, now I can't. It's fine if we can't use the StreamRef anymore, but I would expect to be able to retrieve metadata at least.

I guess it's not a big deal since I can get it from my state but it's surprising.

does not appear to contain CMakeLists.txt

1. 环境:
   centos: 7.9
   cmake version: 3.6.2
   rebar3 version: rebar 3.18.0 on Erlang/OTP 24 Erts 12.3.1
   erlang version: Erlang/OTP 24 Erts 12.3.1
   gcc version:
   gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
   Copyright (C) 2015 Free Software Foundation, Inc.
   This is free software; see the source for copying conditions. There is NO
   warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

2. 日志输出:

❯ /usr/local/erlang_24_3_3/bin/escript ./rebar3 compile
===> Verifying dependencies...
===> Fetching quicer (from {git,"https://github.com/emqx/quic",
                  {ref,"19c9a678b67325895c6b28795bc58e191e277494"}})
Submodule 'submodules/msquic' (https://github.com/microsoft/msquic.git) registered for path 'submodules/msquic'
Cloning into '/root/ws_dhg/srv_normal_x7/_build/default/lib/quicer/submodules/msquic'...
Submodule path 'submodules/msquic': checked out '3be5c66bd488767b77ddfa2aa439c93797ce4437'
Submodule 'submodules/googletest' (https://github.com/google/googletest) registered for path 'submodules/msquic/submodules/googletest'
Submodule 'submodules/openssl' (https://github.com/quictls/openssl.git) registered for path 'submodules/msquic/submodules/openssl'
Submodule 'submodules/wil' (https://github.com/microsoft/wil) registered for path 'submodules/msquic/submodules/wil'
Cloning into '/root/ws_dhg/srv_normal_x7/_build/default/lib/quicer/submodules/msquic/submodules/googletest'...
Cloning into '/root/ws_dhg/srv_normal_x7/_build/default/lib/quicer/submodules/msquic/submodules/openssl'...
Cloning into '/root/ws_dhg/srv_normal_x7/_build/default/lib/quicer/submodules/msquic/submodules/wil'...
Submodule path 'submodules/msquic/submodules/googletest': checked out 'e8512bc38c4c0060858c3306b0660a3f126aee30'
Submodule path 'submodules/msquic/submodules/openssl': checked out 'a6e9d76db343605dae9b59d71d2811b195ae7434'
Submodule 'boringssl' (https://boringssl.googlesource.com/boringssl) registered for path 'submodules/msquic/submodules/openssl/boringssl'
Submodule 'krb5' (https://github.com/krb5/krb5) registered for path 'submodules/msquic/submodules/openssl/krb5'
Submodule 'pyca.cryptography' (https://github.com/pyca/cryptography.git) registered for path 'submodules/msquic/submodules/openssl/pyca-cryptography'
Cloning into '/root/ws_dhg/srv_normal_x7/_build/default/lib/quicer/submodules/msquic/submodules/openssl/boringssl'...
Cloning into '/root/ws_dhg/srv_normal_x7/_build/default/lib/quicer/submodules/msquic/submodules/openssl/krb5'...
Cloning into '/root/ws_dhg/srv_normal_x7/_build/default/lib/quicer/submodules/msquic/submodules/openssl/pyca-cryptography'...
Submodule path 'submodules/msquic/submodules/openssl/boringssl': checked out '2070f8ad9151dc8f3a73bffaa146b5e6937a583f'
Submodule path 'submodules/msquic/submodules/openssl/krb5': checked out '890ca2f401924cdcb88f493950b04700bbe52db3'
Submodule path 'submodules/msquic/submodules/openssl/pyca-cryptography': checked out '09403100de2f6f1cdd0d484dcb8e620f1c335c8f'
Submodule path 'submodules/msquic/submodules/wil': checked out '86353933d5fba0caf61503c8b09e10e182dccdfb'
CMake Error: The source directory "/root/ws_dhg/srv_normal_x7/_build/default/lib/quicer/c_build" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
===> Hook for compile failed!

shutdown_stream hangs

This is very similar to #118 , if one end call close_stream, and the other end close the connection, quicer:shutdown_stream/4 hangs with a closed message on the stream

3> rp(i(<0.84.0>)).
[{current_function,{quicer,shutdown_stream,4}},
 {initial_call,{erlang,apply,2}},
 {status,waiting},
 {message_queue_len,3},
 {links,[<0.83.0>]},
 {dictionary,[]},
 {trap_exit,false},
 {error_handler,error_handler},
 {priority,normal},
 {group_leader,<0.66.0>},
 {total_heap_size,1469},
 {heap_size,987},
 {stack_size,36},
 {reductions,1761},
 {garbage_collection,[{max_heap_size,#{error_logger => true,kill => true,
                                       size => 0}},
                      {min_bin_vheap_size,46422},
                      {min_heap_size,233},
                      {fullsweep_after,65535},
                      {minor_gcs,7}]},
 {suspending,[]},
 {monitors,[]},
 {monitored_by,[#Ref<0.1102007550.629538817.255092>]},
 {messages,[{quic,peer_send_shutdown,
                  #Ref<0.1102007550.629538817.255102>},
            {quic,shutdown,#Ref<0.1102007550.629538817.255092>},
            {quic,closed,#Ref<0.1102007550.629538817.255092>}]},
 {current_stacktrace,[{quicer,shutdown_stream,4,
                              [{file,"/home/klacke/git/quic/src/quicer.erl"},
                               {line,608}]},
                      {qt,recv_ping,4,[{file,"qt.erl"},{line,159}]},
                      {qt,client,4,[{file,"qt.erl"},{line,377}]},
                      {erl_eval,do_apply,7,[{file,"erl_eval.erl"},{line,748}]},
                      {shell,exprs,7,[{file,"shell.erl"},{line,691}]},
                      {shell,eval_exprs,7,[{file,"shell.erl"},{line,647}]},
                      {shell,eval_loop,3,[{file,"shell.erl"},{line,632}]}]}]

The Ref that ends with 92 is the Connection

unload nif module casue undefined behaviors.

When unload quicer_nif erl module beam will unload the dynamic library and releases all the opened resource types. leaves undefined behavior.

We should reopen resources types in upgrade callback for reloading

int (*upgrade)(ErlNifEnv* caller_env, void** priv_data, void** old_priv_data, ERL_NIF_TERM load_info)

Providing our own stream reference for quicer:start_stream

Hello, I was thinking it would be great if the reference that is returned by quicer:start_stream could be provided by the user instead. For example in Gun I already have a reference when creating a request, before QUIC is involved, therefore having a second reference for what is essentially the same thing is a bit of a waste (a bidi stream = 1 request/response in HTTP/3 = 1 Gun stream reference).

does not compile

`quicer/c_src/quicer_config.c:324:37: error: passing argument 3 of ‘enif_get_uint64’ from incompatible pointer type [-Werror=incompatible-pointer-types]
if (!enif_get_uint64(env, evalue, &value0))
/home/pi/erl/24.0.2/usr/include/erl_nif_api_funcs.h:135:89: note: expected ‘ErlNifUInt64 *’ {aka ‘long long unsigned int ’} but argument is of type ‘long unsigned int ’
ERL_NIF_API_FUNC_DECL(int,enif_get_uint64,(ErlNifEnv, ERL_NIF_TERM term, ErlNifUInt64 ip));
~~~~~~~~~~~~~~^~
/home/pi/erl/24.0.2/usr/include/erl_nif.h:345:76: note: in definition of macro ‘ERL_NIF_API_FUNC_DECL’

define ERL_NIF_API_FUNC_DECL(RET_TYPE, NAME, ARGS) extern RET_TYPE NAME ARGS

                                                                        ^~~~

cc1: all warnings being treated as errors
make[3]: *** [CMakeFiles/quicer_static.dir/build.make:76: CMakeFiles/quicer_static.dir/c_src/quicer_config.c.o] Error 1
`
erlang 24.0.2
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian

Building OpenSSL fails on linux x86_64

First, I had to install libatomic which isn't in the readme, I can make a PR for that. But then it is building OpenSSL (is that really required?) and fails:

Operating system: x86_64-whatever-linux2
Can't locate FindBin.pm in @INC (you may need to install the FindBin module) (@INC contains: /usr/local/lib64/perl5/5.34 /usr/local/share/perl5/5.34 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at /home/tristan/Devel/quic/msquic/submodules/openssl/Configure line 15.
BEGIN failed--compilation aborted at /home/tristan/Devel/quic/msquic/submodules/openssl/Configure line 15.
Can't locate FindBin.pm in @INC (you may need to install the FindBin module) (@INC contains: /usr/local/lib64/perl5/5.34 /usr/local/share/perl5/5.34 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at /home/tristan/Devel/quic/msquic/submodules/openssl/Configure line 15.
BEGIN failed--compilation aborted at /home/tristan/Devel/quic/msquic/submodules/openssl/Configure line 15.
This system (linux-x86_64) is not supported. See file INSTALL for details.
make[4]: *** [_deps/opensslquic-build/CMakeFiles/OpenSSL_Target.dir/build.make:81: _deps/opensslquic-build/submodules/openssl/Makefile] Error 1
make[4]: Leaving directory '/home/tristan/Devel/quic/c_build'
make[3]: *** [CMakeFiles/Makefile2:265: _deps/opensslquic-build/CMakeFiles/OpenSSL_Target.dir/all] Error 2

This is Fedora 36. I can't believe linux-x86_64 is actually "not supported" as the error message says in the 4th to last line :), so no idea what is really going on.

quicer:listen() twice

I cannot execute listen() twice in the same VM instance.
Try

-module(abc).


-compile(export_all).
-compile(nowarn_export_all).


start() ->
    application:ensure_all_started(quicer),
    Port = 4567,
    LOptions = [ {cert, "./server.pem"}
               , {key,  "./server.key"}
               , {verify, none}
               , {active, false}
               , {alpn, ["sample"]}
               , {peer_bidi_stream_count, 64000}
               ],
    L = proc_lib:spawn_link(fun() ->
                          {ok, L} = quicer:listen(Port, LOptions),
                          listener(L)
                      end),
    L.

listener(_L) ->
    receive
        die ->
            ok
    end.

from the shell, in my book if the process dies, it should be possible to execute abc:start() again, bit it isn't
listen() then returns

{error,listener_start_error,{unknown_quic_status,91}}

To run: compile and then do:

Eshell V13.1.2  (abort with ^G)
1> abc:start().
<0.94.0>
2> a=p.
** exception error: no match of right hand side value p
3> abc:start().
<0.98.0>
4> =CRASH REPORT==== 8-Dec-2022::22:53:54.029762 ===

We earlier had a discussion on errno, regarding EMFILE, but 91 seems far fetched. It's "Protocol wrong type for socket"

run lux test in CI

currently, lux tests are not in use in CI.
A cheap way to do it is compile/install lux in each run.

apt install lux is not a good option it is too heavy and mostly failed.

Getting the peer TLS certificate

Hello, I may have missed something. Is there a way to get the peer TLS certificate, and if so how? Thank you.

`quicer:peername` no longer accepts stream handle

Hi,
after b983be5 it's no longer possible to pass Stream to quicer:peername and I am wondering whether this was intentional or not?

Weird unidi streams behavior

For HTTP/3 I am expected to receive from at least 3 unidi streams.

I wrote a CT test that looks like this:

unidi_create_critical_first(Config) →
    {ok, Conn} = quicer:connect("localhost", config(port, Config),
        #{alpn ⇒ ["h3"], verify ⇒ none, peer_unidi_stream_count ⇒ 3}, 5000),
    {ok, Conn} = quicer:async_accept_stream(Conn, []),
    timer:sleep(1000),
    ct:pal("~p", [process_info(self(), messages)]),
    ct:pal("~p", [process_info(self(), messages)]),
    ok.

This prints the following:

----------------------------------------------------
2023-08-30 17:15:57.986
{messages,[{quic,streams_available,#Ref<0.3047038216.2871918595.261224>,
                 #{bidi_streams => 100,unidi_streams => 3}}]}


----------------------------------------------------
2023-08-30 17:15:57.986
{messages,[{quic,streams_available,#Ref<0.3047038216.2871918595.261224>,
                 #{bidi_streams => 100,unidi_streams => 3}},
           {quic,new_stream,#Ref<0.3047038216.2871918592.249713>,
                 #{flags => 1,is_orphan => false}},
           {quic,new_stream,#Ref<0.3047038216.2871918592.249714>,
                 #{flags => 1,is_orphan => true}},
           {quic,new_stream,#Ref<0.3047038216.2871918592.249715>,
                 #{flags => 1,is_orphan => true}},
           {quic,<<0,4,0>>,
                 #Ref<0.3047038216.2871918592.249713>,
                 #{absolute_offset => 0,flags => 0,len => 3}}]}

First thing I notice is that the first ct:pal somehow doesn't print all the messages, I'm not sure why that is. But not important for this issue. If I do a receive I receive all these messages in the order they are printed with no issue.

The first problem is that the single async_accept_stream call resulted in 3 unidi streams getting accepted. I did not expect that at all.

The second problem is that I am not receiving the data from the two other unidi streams, even if I put more async_accept_stream calls.

core dump when listener recv buffer size set to 1024

After open listener with invalid args, the config resource detor releases a QUIC_CONFIG which is uninitialized in use causing coredump.

Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--
Core was generated by `./rebar3 -B -sbtu -A1 -- -root /home/ubuntu/OTP/24.3.4.6 -progname erl -- -home'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000ffff182427ac in MsQuicConfigurationClose (Handle=0x969)
    at /home/ubuntu/repo/quic/msquic/src/core/configuration.c:281
281	    if (Handle != NULL && Handle->Type == QUIC_HANDLE_TYPE_CONFIGURATION) {
[Current thread is 1 (Thread 0xffff728e01e0 (LWP 822391))]
/home/ezhuwya/.kerl/builds/24.3.4/otp_src_24.3.4/erts/etc/unix/etp-commands: No such file or directory.
#0  0x0000ffff182427ac in MsQuicConfigurationClose (Handle=0x969)
    at /home/ubuntu/repo/quic/msquic/src/core/configuration.c:281
#1  0x0000aaaac3ae4550 in run_resource_dtor (vbin=0xaaaafbde88c8)
    at beam/erl_nif.c:2677
#2  0x0000aaaac395a1d8 in handle_misc_aux_work (awdp=awdp@entry=0xffff73899378, 
    aux_work=aux_work@entry=2049, waiting=0) at beam/erl_process.c:1865
#3  0x0000aaaac3961854 in handle_aux_work (awdp=0xffff73899378, 
    orig_aux_work=<optimized out>, waiting=0) at beam/erl_process.c:2701
#4  0x0000aaaac396ff38 in erts_schedule (esdp=<optimized out>, p=<optimized out>, 
    calls=<optimized out>) at beam/erl_process.c:9677
#5  0x0000aaaac3978308 in process_main (esdp=0x969, esdp@entry=0xffff73899280)
    at beam/emu/beam_emu.c:356
#6  0x0000aaaac3966940 in sched_thread_func (vesdp=0xffff73899280)
    at beam/erl_process.c:8656
#7  0x0000aaaac3b8c714 in thr_wrapper (vtwd=0xffffcd7824e0)
    at pthread/ethread.c:122
#8  0x0000ffffb4a6f624 in start_thread (arg=0xaaaac3b8c668 <thr_wrapper>)
    at pthread_create.c:477
#9  0x0000ffffb49c649c in thread_start ()
    at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

quic:connect()

If quic:connect() fails nicely with for example

{error,transport_down,
                            #{error => 1,status => unreachable}}

an unnecessary message

{quic,closed,#Ref<0.1073868949.2575171596.191148>,
            #{is_app_closing => false,is_handshake_completed => false,
              is_peer_acked => true}}

is subsequently sent to the caller.

I find this weird, the call has already failed. Also the flag is_peer_acked is confusing.

active false on accept_stream

If a server does accept_stream() with {active, false} and the client immediately sends data after the call to start_stream(), sometimes the data from the client ends up in the message queue as a {quic, Bin, Stream ... } message and a subsequent call to recv() fail with einval.
This happens rarely, but it happens.
A fix is

diff --git i/c_src/quicer_stream.c w/c_src/quicer_stream.c
index 19529b1..5b32c8a 100644
--- i/c_src/quicer_stream.c
+++ w/c_src/quicer_stream.c
@@ -370,7 +370,7 @@ async_start_stream2(ErlNifEnv *env,
   //
   if (QUIC_FAILED(Status = MsQuic->StreamStart(s_ctx->Stream,
                                                // @todo flag in options
-                                               QUIC_STREAM_START_FLAG_NONE)))
+                                               QUIC_STREAM_START_FLAG_IMMEDIATE)))
     {
       // note, stream call back would close the stream
       // destroy_s_ctx should not be called here

This behavior is btw good I think. I.e that the call to accept_stream() returns after the client does start_stream() and not until the client sends some data, which is the behavior today.

passive recv get dup bytes

We caught an issue that there are dup bytes received in tc: tc_stream_passive_receive_buffer

quicer_SUITE:tc_stream_passive_receive_buffer failed on line 973
Reason: {badmatch,{ok,<<"ong">>}}

The reason is that the MsQuic API call: STREAM_RECEIVE_COMPLETE is an async call and gets rejected (errno 95) because one call must correspond to one QUIC_STREAM_EVENT_RECEIVE event. Multiple calls for one QUIC_STREAM_EVENT_RECEIVE is ignored.

Because it is an async API and the application doesn't know if it is success or not, Quicer need extra sync/check while using this API.

GLOG:

[0][1370.2a87][09:14:31.423484][strm][0x7f7e8806e670] Indicating QUIC_STREAM_EVENT_RECEIVE [4 bytes, 1 buffers, 0x0 flags]
[0][1370.2a87][09:14:31.423489][conn][0x7f7e905e68e0] Execute: 4
[0][1370.2a87][09:14:31.423490][conn][0x7f7e905e68e0] Flushing Send. Allowance=12000 bytes
[0][1370.2a87][09:14:31.423491][pack][1099511629798] Created in batch 1099511628130
[0][1370.2a87][09:14:31.423492][C][TX][12] SH DestCid:00537dcabf65063777 KP:0 SB:1 (Payload 9 bytes)
[0][1370.2a87][09:14:31.423492][C][TX][12]   ACK Largest:10 Delay:0
[0][1370.2a87][09:14:31.423493][C][TX][12]     9 - 10
[0][1370.2a87][09:14:31.423493][pack][1099511629798] Encrypting
[0][1370.2a87][09:14:31.423494][pack][1099511629798] Finalizing
[0][1370.2a87][09:14:31.423495][conn][0x7f7e905e68e0][TX][12] 5 (35 bytes)
[0][1370.2a87][09:14:31.423496][conn][0x7f7e905e68e0] Sending batch. 1 datagrams
[0][1370.2a87][09:14:31.423496][data][0x7f7e6400de50] Send 35 bytes in 1 buffers (segment=1452) Dst=[::1]:37928, Src=[::1]:60712
[0][1370.2a87][09:14:31.423509][pack][1099511628130] Batch sent
[0][1370.2a87][09:14:31.423509][conn][0x7f7e905e68e0] Flush complete flags=0x0
[0][1370.2a87][09:14:31.423510][conn][0x7f7e905e68e0] Scheduling: 0
[0][1370.2a87][09:14:31.423511][wrkr][0x7f7e9011df40] IsActive = 0, Arg = 4294967295
[0][1370.2a87][09:14:31.423511][wrkr][0x7f7e9011df40] QueueDelay = 0
[0][1370.2a80][09:14:31.423519][data][0x7f7e88094bc0] Recv 35 bytes (segment=35) Src=[::1]:37928 Dst=[::1]:60712
[0][1370.2a80][09:14:31.423519][pack][1099511629450] Received
[0][1370.2a80][09:14:31.423521][conn][0x7f7e8c004d80] Queuing 1 UDP datagrams
[0][1370.2a80][09:14:31.423522][conn][0x7f7e8c004d80] Scheduling: 1
[1][1370.2a86][09:14:31.423535][wrkr][0x7f7e9011d810] IsActive = 1, Arg = 1
[1][1370.2a86][09:14:31.423536][conn][0x7f7e8c004d80] Scheduling: 2
[1][1370.2a86][09:14:31.423536][wrkr][0x7f7e9011d810] QueueDelay = 1
[1][1370.2a86][09:14:31.423537][conn][0x7f7e8c004d80] Execute: 1
[1][1370.2a86][09:14:31.423537][conn][0x7f7e8c004d80] Recv 1 UDP datagrams
[1][1370.2a86][09:14:31.423538][conn][0x7f7e8c004d80] IN: BytesRecv=6946
[1][1370.2a86][09:14:31.423539][conn][0x7f7e8c004d80] Batch Recv 1 UDP datagrams
[1][1370.2a86][09:14:31.423539][pack][1099511629450] Decrypting
[1][1370.2a86][09:14:31.423541][S][RX][12] SH DestCid:00537dcabf65063777 KP:0 SB:1 (Payload 9 bytes)
[1][1370.2a86][09:14:31.423541][S][RX][12]   ACK Largest:10 Delay:0
[1][1370.2a86][09:14:31.423542][S][RX][12]     9 - 10
[1][1370.2a86][09:14:31.423542][conn][0x7f7e8c004d80][RX][12] � (19 bytes)
[1][1370.2a86][09:14:31.423543][S][TX][9] ACKed (0.177 ms)
[1][1370.2a86][09:14:31.423544][conn][0x7f7e8c004d80][TX][9] 5 ACKed
[1][1370.2a86][09:14:31.423544][conn][0x7f7e8c004d80] Path[0] MTU updated to 1440 bytes
[1][1370.2a86][09:14:31.423545][conn][0x7f7e8c004d80] Path[0] Mtu Discovery Search Packet Sending with MTU 1500
[1][1370.2a86][09:14:31.423545][conn][0x7f7e8c004d80] Scheduling flags 0x2147483648 to 0x0
[1][1370.2a86][09:14:31.423546][conn][0x7f7e8c004d80] Queueing send flush, reason=0
[1][1370.2a86][09:14:31.423546][S][TX][10] ACKed (0.105 ms)
[1][1370.2a86][09:14:31.423547][conn][0x7f7e8c004d80][TX][10] 5 ACKed
[1][1370.2a86][09:14:31.423547][strm][0x7f7e6007b0d0] Received ack for 4 bytes, offset=4, FF=0x0
[1][1370.2a86][09:14:31.423548][strm][0x7f7e6007b0d0] SF:0 FC:65536 QS:8 MAX:8 UNA:8 NXT:8 RECOV:0-0
[1][1370.2a86][09:14:31.423549][conn][0x7f7e8c004d80] Updated Rtt=0.337 ms, Var=0.261
[1][1370.2a86][09:14:31.423549][conn][0x7f7e8c004d80] OUT: BytesSent=7961 InFlight=0 InFlightMax=6000 CWnd=12000 SSThresh=4294967295 ConnFC=16777208 ISB=131072 PostedBytes=0 SRtt=337
[1][1370.2a86][09:14:31.423550][conn][0x7f7e8c004d80] OUT: StreamFC=65528 StreamSendWindow=65536
[1][1370.2a86][09:14:31.423551][conn][0x7f7e8c004d80] Canceling 2
[1][1370.2a86][09:14:31.423551][time][0x7f7e9011d8a8] Updating Connection 0x7f7e8c004d80.
[1][1370.2a86][09:14:31.423552][time][0x7f7e9011d8a8] Next Expiration = {214781165, 0x7f7e8c004d80}.
[1][1370.2a86][09:14:31.423552][S][RX][12] Marked for ACK (ECN=0)
[1][1370.2a86][09:14:31.423553][conn][0x7f7e8c004d80] Setting 4, delay=5000000 us
[1][1370.2a86][09:14:31.423553][time][0x7f7e9011d8a8] Updating Connection 0x7f7e8c004d80.
[1][1370.2a86][09:14:31.423553][time][0x7f7e9011d8a8] Next Expiration = {214781278, 0x7f7e8c004d80}.
[1][1370.2a86][09:14:31.423554][conn][0x7f7e8c004d80] Execute: 4
[1][1370.2a86][09:14:31.423554][conn][0x7f7e8c004d80] Flushing Send. Allowance=12000 bytes
[1][1370.2a86][09:14:31.423555][pack][2199023256861] Created in batch 2199023255755
[0][1370.1378][09:14:31.423555][ api] Enter 23 (0x7f7e8806e670).
[1][1370.2a86][09:14:31.423556][S][TX][11] SH DestCid: KP:0 SB:1 (Payload 1435 bytes)
[1][1370.2a86][09:14:31.423556][S][TX][11]   PING
[0][1370.1378][09:14:31.423556][strm][0x7f7e8806e670] Receive complete call [1 bytes]
[1][1370.2a86][09:14:31.423558][S][TX][11]   PADDING Len:1430
[0][1370.1378][09:14:31.423558][conn][0x7f7e905e68e0] Scheduling: 1
[1][1370.2a86][09:14:31.423558][pack][2199023256861] Encrypting
[1][1370.2a86][09:14:31.423560][pack][2199023256861] Finalizing
[1][1370.2a86][09:14:31.423560][conn][0x7f7e8c004d80][TX][11] 5 (1452 bytes)
[1][1370.2a86][09:14:31.423561][conn][0x7f7e8c004d80] Sending batch. 1 datagrams
[1][1370.2a86][09:14:31.423561][data][0x7f7e88094bc0] Send 1452 bytes in 1 buffers (segment=0) Dst=[::1]:60712, Src=[::1]:37928
[0][1370.2a87][09:14:31.423567][wrkr][0x7f7e9011df40] IsActive = 1, Arg = 1
[0][1370.2a87][09:14:31.423567][conn][0x7f7e905e68e0] Scheduling: 2
[0][1370.2a87][09:14:31.423568][wrkr][0x7f7e9011df40] QueueDelay = 1
[0][1370.2a87][09:14:31.423568][conn][0x7f7e905e68e0] Execute: 9
[0][1370.2a87][09:14:31.423569][strm][0x7f7e8806e670] Receive complete [1 bytes]
[0][1370.2a87][09:14:31.423570][strm][0x7f7e8806e670] Recv State: 2
[0][1370.2a87][09:14:31.423571][conn][0x7f7e905e68e0] Scheduling: 0
[0][1370.2a87][09:14:31.423571][wrkr][0x7f7e9011df40] IsActive = 0, Arg = 4294967295
[0][1370.2a87][09:14:31.423571][wrkr][0x7f7e9011df40] QueueDelay = 0
[1][1370.2a81][09:14:31.423574][data][0x7f7e6400de50] Recv 1452 bytes (segment=1452) Src=[::1]:60712 Dst=[::1]:37928
[0][1370.1378][09:14:31.423574][ api] Exit
[1][1370.2a81][09:14:31.423575][pack][2199023257188] Received
[0][1370.1378][09:14:31.423575][ api] Enter 24 (0x7f7e8806e670).
[0][1370.1378][09:14:31.423575][conn][0x7f7e905e68e0] Scheduling: 1
[1][1370.2a81][09:14:31.423575][conn][0x7f7e905e68e0] Queuing 1 UDP datagrams
[1][1370.2a86][09:14:31.423581][pack][2199023255755] Batch sent
[1][1370.2a86][09:14:31.423582][conn][0x7f7e8c004d80] Flush complete flags=0x0
[1][1370.2a86][09:14:31.423582][conn][0x7f7e8c004d80] Scheduling: 0
[1][1370.2a86][09:14:31.423583][wrkr][0x7f7e9011d810] IsActive = 0, Arg = 4294967295
[0][1370.2a87][09:14:31.423583][wrkr][0x7f7e9011df40] IsActive = 1, Arg = 1
[1][1370.2a86][09:14:31.423583][wrkr][0x7f7e9011d810] QueueDelay = 0
[0][1370.2a87][09:14:31.423583][conn][0x7f7e905e68e0] Scheduling: 2
[0][1370.2a87][09:14:31.423584][wrkr][0x7f7e9011df40] QueueDelay = 1
[0][1370.2a87][09:14:31.423584][conn][0x7f7e905e68e0] Execute: 10
[0][1370.2a87][09:14:31.423585][strm][0x7f7e8806e670] Recv State: 1
[0][1370.2a87][09:14:31.423585][strm][0x7f7e8806e670] Indicating QUIC_STREAM_EVENT_RECEIVE [3 bytes, 1 buffers, 0x0 flags]
[0][1370.1378][09:14:31.423588][ api] Exit 4294967294
[0][1370.1378][09:14:31.423595][ api] Enter 23 (0x7f7e8806e670).
[0][1370.1378][09:14:31.423596][strm][0x7f7e8806e670] Receive complete call [0 bytes]
[0][1370.1378][09:14:31.423597][ api] Exit
[0][1370.1378][09:14:31.423597][ api] Enter 24 (0x7f7e8806e670).
[0][1370.1378][09:14:31.423598][ api] Exit 4294967294
[1][1370.2a87][09:14:31.423616][conn][0x7f7e905e68e0] Execute: 1
[1][1370.2a87][09:14:31.423616][conn][0x7f7e905e68e0] Recv 1 UDP datagrams
[1][1370.2a87][09:14:31.423617][conn][0x7f7e905e68e0] IN: BytesRecv=9413
[1][1370.2a87][09:14:31.423617][conn][0x7f7e905e68e0] Batch Recv 1 UDP datagrams
[0][1370.1378][09:14:31.423618][ api] Enter 23 (0x7f7e8806e670).
[1][1370.2a87][09:14:31.423618][pack][2199023257188] Decrypting
[0][1370.1378][09:14:31.423618][strm][0x7f7e8806e670] Receive complete call [2 bytes]
[0][1370.1378][09:14:31.423619][ api] Error 95
[0][1370.1378][09:14:31.423620][ api] Exit
[0][1370.1378][09:14:31.423620][ api] Enter 24 (0x7f7e8806e670).
[1][1370.2a87][09:14:31.423620][C][RX][11] SH DestCid: KP:0 SB:1 (Payload 1435 bytes)
[0][1370.1378][09:14:31.423621][ api] Exit 4294967294
[1][1370.2a87][09:14:31.423621][C][RX][11]   PING
[0][1370.1378][09:14:31.423622][ api] Enter 23 (0x7f7e8806e670).
[0][1370.1378][09:14:31.423622][strm][0x7f7e8806e670] Receive complete call [0 bytes]
[1][1370.2a87][09:14:31.423622][C][RX][11]   PADDING Len:1430
[1][1370.2a87][09:14:31.423623][conn][0x7f7e905e68e0][RX][11] � (1436 bytes)
[0][1370.1378][09:14:31.423623][ api] Error 95
[0][1370.1378][09:14:31.423623][ api] Exit
[0][1370.1378][09:14:31.423624][ api] Enter 24 (0x7f7e8806e670).
[0][1370.1378][09:14:31.423624][ api] Exit 4294967294
[1][1370.2a87][09:14:31.423625][C][RX][11] Marked for ACK (ECN=0)
[1][1370.2a87][09:14:31.423625][conn][0x7f7e905e68e0] Starting ACK_DELAY timer for 25 ms
[1][1370.2a87][09:14:31.423626][conn][0x7f7e905e68e0] Setting 1, delay=25000 us
[1][1370.2a87][09:14:31.423626][time][0x7f7e9011dfd8] Updating Connection 0x7f7e905e68e0.
[1][1370.2a87][09:14:31.423626][time][0x7f7e9011dfd8] Next Expiration = {209806351, 0x7f7e905e68e0}.
[1][1370.2a87][09:14:31.423627][conn][0x7f7e905e68e0] Setting 4, delay=5000000 us
[1][1370.2a87][09:14:31.423628][conn][0x7f7e905e68e0] Execute: 9
[1][1370.2a87][09:14:31.423628][strm][0x7f7e8806e670] Receive complete [0 bytes]
[1][1370.2a87][09:14:31.423629][strm][0x7f7e8806e670] Recv State: 2
[1][1370.2a87][09:14:31.423630][conn][0x7f7e905e68e0] Execute: 10
[1][1370.2a87][09:14:31.423630][strm][0x7f7e8806e670] Recv State: 1
[1][1370.2a87][09:14:31.423631][strm][0x7f7e8806e670] Indicating QUIC_STREAM_EVENT_RECEIVE [3 bytes, 1 buffers, 0x0 flags]
[1][1370.2a87][09:14:31.423634][conn][0x7f7e905e68e0] Execute: 10
[1][1370.2a87][09:14:31.423634][strm][0x7f7e8806e670] Recv State: 1
[1][1370.2a87][09:14:31.423634][conn][0x7f7e905e68e0] Execute: 10
[0][1370.1378][09:14:31.423635][ api] Enter 23 (0x7f7e8806e670).
[1][1370.2a87][09:14:31.423635][conn][0x7f7e905e68e0] Scheduling: 0
[0][1370.1378][09:14:31.423635][strm][0x7f7e8806e670] Receive complete call [3 bytes]

error setttings field

https://github.com/emqx/quic/blob/main/c_src/quicer_config.c line:732 and line:739 set same parm line:732 should set
MtuDiscoverySearchCompleteTimeoutUs

Auto reconnects

In lossy/bad network test (20% pkts drop and 10% reorderd packets), we found once the init handshake is lost, we are not able to reconnect.

One solution is do it in the application but maybe it is better to add support in quicer just follow what TCP stack will do with retries.

# sysctl var in linux
net.ipv4.tcp_syn_retries = 6

Getting cert_untrusted_root

I'm trying to create a simple program to test the library. I already created the cert and key files but when I try to connect the client with the server the following error is shown:
** (MatchError) no match of right hand side value: {:error, :transport_down, %{error: 304, status: :cert_untrusted_root}} (quic 0.1.0) lib/quic.ex:16: Quic.client_start/0
I'm quite new to Quic, so maybe this is a silly issue, but It would be quite helpful if someone could give me some guidence.

Note: I'm using the library with Elixir, don't know if this could be the problem.

Shutdown stream error code

Hello, I may be misunderstanding something. But I expect that when I do this:

quicer:shutdown_stream(StreamRefA, ?QUIC_STREAM_SHUTDOWN_FLAG_GRACEFUL, Code, infinity),

On the other side I then receive this:

{quic, stream_closed, StreamRefB, #{error := Code}}

With the Code value the same before and after. But in the message the error Code is always 0 for me.

I tried to look in the tests but I do not see where the value is checked. I tried with ?QUIC_STREAM_SHUTDOWN_FLAG_ABORT as well but I get 0 too.

quality: OSS-Fuzz check

stream deallocated when it is not closed

caught in a PR:
https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334

test case:

tc_stream_send_shutdown_complete

This is assertion error, but the test was executed without lttng/snk log.

Program terminated with signal SIGABRT, Aborted.
[546](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:547)
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[547](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:548)
[Current thread is 1 (Thread 0x7fab74e0f700 (LWP 1921))]
[548](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:549)
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
[549](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:550)
#1  0x00007fabb8a45859 in __GI_abort () at abort.c:79
[550](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:551)
#2  0x00007fabb8a45729 in __assert_fail_base (
[551](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:552)
    fmt=0x7fabb8bdb588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
[552](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:553)
    assertion=0x7fab4e95a06e "s_ctx->is_closed == TRUE", 
[553](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:554)
    file=0x7fab4e95a020 "/home/runner/work/quic/quic/c_src/quicer_nif.c", 
[554](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:555)
    line=791, function=<optimized out>) at assert.c:92
[555](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:556)
#3  0x00007fabb8a56fd6 in __GI___assert_fail (
[556](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:557)
    assertion=0x7fab4e95a06e "s_ctx->is_closed == TRUE", 
[557](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:558)
    file=0x7fab4e95a020 "/home/runner/work/quic/quic/c_src/quicer_nif.c", 
[558](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:559)
    line=791, 
[559](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:560)
    function=0x7fab4e95b5e0 <__PRETTY_FUNCTION__.23883> "resource_stream_dealloc_callback") at assert.c:101
[560](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:561)
#4  0x00007fab4e74f581 in resource_stream_dealloc_callback (
[561](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:562)
    env=0x7fab74e0a7e0, obj=0x55969b6471e8)
[562](https://github.com/emqx/quic/actions/runs/3248143092/jobs/5329167334#step:5:563)
    at /home/runner/work/quic/quic/c_src/quicer_nif.c:791

non-portable shebang in get-msquic.sh

I trying to use emqx/emqtt 1.4.5 in an Elixir project. While compiling, an error is reported about emqx/quic:

[nix-shell:~/myproject]$ mix compile
sh: /home/myuser/myproject/deps/quicer/get-msquic.sh: /bin/bash: bad interpreter: No such file or directory
sh: line 1: /home/myuser/myproject/deps/quicer/get-msquic.sh: Success
===> Hook for compile failed!

** (Mix) Could not compile dependency :quicer, "/home/myuser/.mix/rebar3 bare compile --paths /home/myuser/myproject/_build/dev/lib/*/ebin" command failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile quicer", update it with "mix deps.update quicer" or clean it with "mix deps.clean quicer"

I am using NixOS so /bin/bash does not exists.

The shebang #!/usr/bin/env bash would make the get-msquic.sh script more portable.

make: cmake: Permission denied

Hi, when I am trying to install quicer using Mix.install I am getting

[17:38] michal:quicer ((868d5af2…)) | make build-nif
./get-msquic.sh v1.8.0
cmake -B c_build
make: cmake: Permission denied
make: *** [Makefile:22: build-nif] Error 127

I also tried to clone and run make build-nif manually but I got the same error. When i run cmake -B c_build everything works okay 🤔

Listing connections

Hello! Me again.

In order to write the Cowboy HTTP/3 test suite I need for some cases a way to retrieve the pid of the peer of the connection. For TCP this looks like the get_remote_pid_tcp function here: https://git.ninenines.eu/ct_helper.git/tree/src/ct_helper.erl#n124

I need to do something equivalent for QUIC: get the list of connections, go through them until I find the connection with the right peer name, and then get the pid of the controlling process for that connection.

When I have that pid I can then send messages or signals such as sys:terminate(whatever) to test graceful connection closure.

Is there perhaps an undocumented way to do this, that I can use, and if not is that something that could be added?

Cheers.

Can you implement some cases like cowboy?

Reference cowboy can achieve some case like this?

能否参考 cowboy 这样实现一些案例吗？