quicr / libquicr Goto Github PK

Transport based on QuicR API

License: BSD 2-Clause "Simplified" License

CMake 1.96% Makefile 0.29% C++ 97.45% Dockerfile 0.22% Shell 0.08%

libquicr's Introduction

libquicr

An API library that implements publish/subscribe protocol draft-ietf-moq-transport-04. The API supports both client and server. Server is intended to be implemented as a relay.

Build

Use make to build libquicr.

Use make test to run tests.

Use make cclean to clean build files.

Build without InfluxDB

To build without InfluxDB, run cmake with -DLIBQUICR_WITHOUT_INFLUXDB=ON

Self-signed Certificate

Server requires a TLS certificate and key file. For development and testing, use a self-signed certificate. Below are the steps to create a self-signed certificate and private ey.

OpenSSL/BorningSSL

cd build/cmd/moq-example
openssl req -nodes -x509 -newkey rsa:2048 -days 365 \
    -subj "/C=US/ST=CA/L=San Jose/O=Cisco/CN=test.m10x.org" \
    -keyout server-key.pem -out server-cert.pem

MbedTLS

openssl req -nodes -x509 -newkey ec:<(openssl ecparam -name prime256v1) -days 365 \
    -subj "/C=US/ST=CA/L=San Jose/O=Cisco/CN=test.m10x.org" \
    -keyout server-key.pem -out server-cert.pem

    openssl ecparam -name prime256v1 -genkey -noout -out server-key-ec.pem
    openssl req -nodes -x509 -key server-key-ec.pem -days 365 \
        -subj "/C=US/ST=CA/L=San Jose/O=Cisco/CN=test.m10x.org" \
        -keyout server-key.pem -out server-cert.pem

MOQ Implementation Documentation

See MOQ Implementation

libquicr's People

Contributors

Stargazers

Watchers

Forkers

zidate palerikm quicr clarkzjw markcanfly

libquicr's Issues

Handles incoming messages as streams rather than full objects

Deprecate use of boringSSL

Depends on issue #162. When mbedTLS is working well, deprecate boringSSL completely.

Assign new quic stream per track ( don't reuse control stream)

Deadlock in QuicRClientRawSession

The session_mutex is locked when unsubscribe() is called. That then calls removeSubscribeState()

void
QuicRClientRawSession::unsubscribe(const quicr::Namespace& quicr_namespace,
                                   const std::string& /* origin_url */,
                                   const std::string& /* auth_token */)
{
  // The removal of the delegate is done on receive of subscription ended
  std::lock_guard<std::mutex> lock(session_mutex);

  messages::MessageBuffer msg{};
  messages::Unsubscribe unsub{ 0x1, quicr_namespace };
  msg << unsub;

  removeSubscribeState(quicr_namespace,
                       SubscribeResult::SubscribeStatus::ConnectionClosed);
  transport->enqueue(transport_context_id, transport_stream_id, msg.take());
}

And removeSubscribeState() is this:

void
QuicRClientRawSession::removeSubscribeState(
  const quicr::Namespace& quicr_namespace,
  const SubscribeResult::SubscribeStatus& reason)
{
  std::lock_guard<std::mutex> _(session_mutex);

  if (!!subscribe_state.count(quicr_namespace)) {
    subscribe_state.erase(quicr_namespace);
  }

  if (!!sub_delegates.count(quicr_namespace)) {
    if (auto sub_delegate = sub_delegates[quicr_namespace].lock())
      sub_delegate->onSubscriptionEnded(quicr_namespace, reason);

    sub_delegates.erase(quicr_namespace);
  }
}

The same mutex is locked again.

One of the two needs to be removed. I'd suggest removing it from unsubscribe(), as it doesn't appear to be necessary there.

MoQT API Cullen Comments

Here's a PDF version of Cullen's comments including response/questions/etc.

cullens_questions_responses.pdf

A list of agreed changes from Cullen's comments. The trailing (#) are the corresponding comment number.

QuicR Server: Handle MoQ Messages in the statemachine

Most of the changes on the client is needed on the server + more to handle multiple connections.

In stream mode, getting read errors

Getting read errors as follows with stream mode.

2023-09-06T15:57:23.748817 [CRITICAL] [CMGR] [QSES] Received read exception error while reading from message buffer: length of decoded data must match separately decoded length: 78 != 65614

QuicrClient: Support Message Handling for MoQ Mesages

This work requires few changes
1: Mapping between MOQ identifiers ( track-alias, subscribe-id, full trackname) to QUICR identifiers ( Namespace, Name)
2. Support handlers for MoQ Messages
3. Handle subscriptions ( for publishers) forwarded
4. URI convertor integration

epic: Integrate mbedTLS using picoquic support via picoTLS

Update libquicr and submodules to support mbedTLS over OpenSSL.

Picoquic supports mbedTLS via picotls. Update transport to configure picoquic/picotls to use mbedTLS instead of boringSSL.
Configure other submodules to support/use mbedtls instead of openssl. Make sure openssl is not required or used by submodules. If they are, use mbedtls via the transport in the same way that boringssl is used today.

Crash in `on_recv_notify`

I don't have much detail here, dropping this so I don't forget. I saw a crash in the following on teardown.

#0	0x0000000104ea129c in quicr::QuicRTransportDelegate::on_recv_notify(unsigned long long const&, unsigned long long const&) ()

This seems to happen in conjunction when there's another weird issue, where I'm seemingly getting loss or the relay is running very slowly even locally. When that's happening, I always get the above on teardown.

qmedia`quicr::QuicRTransportDelegate::on_recv_notify:
    0x105585248 <+0>:   sub    sp, sp, #0x140
    0x10558524c <+4>:   stp    x28, x27, [sp, #0x120]
    0x105585250 <+8>:   stp    x29, x30, [sp, #0x130]
    0x105585254 <+12>:  add    x29, sp, #0x130
    0x105585258 <+16>:  stur   x0, [x29, #-0x18]
    0x10558525c <+20>:  stur   x1, [x29, #-0x20]
    0x105585260 <+24>:  stur   x2, [x29, #-0x28]
    0x105585264 <+28>:  ldur   x8, [x29, #-0x18]
    0x105585268 <+32>:  str    x8, [sp, #0x38]
    0x10558526c <+36>:  stur   wzr, [x29, #-0x2c]
    0x105585270 <+40>:  b      0x105585274               ; <+44>
    0x105585274 <+44>:  ldur   w8, [x29, #-0x2c]
    0x105585278 <+48>:  subs   w8, w8, #0x96
    0x10558527c <+52>:  b.ge   0x1055855cc               ; <+900>
    0x105585280 <+56>:  b      0x105585284               ; <+60>
    0x105585284 <+60>:  ldr    x8, [sp, #0x38]
    0x105585288 <+64>:  ldr    x8, [x8, #0x8]
    0x10558528c <+68>:  add    x0, x8, #0x8
    0x105585290 <+72>:  bl     0x10557d290               ; std::__1::shared_ptr<qtransport::ITransport>::operator->() const
    0x105585294 <+76>:  ldur   x1, [x29, #-0x20]
    0x105585298 <+80>:  ldur   x2, [x29, #-0x28]
->  0x10558529c <+84>:  ldr    x8, [x0] EXC_BAD_ACCESS (code=1, address=0x0)

Relation between libquicr and quicrq

Hi,

I am interested in WebTransport, and I found this Moq working group and these repositories. I am currently exploring the https://github.com/Quicr/quicrq repository but it seems to me that the repository is dead. Or it is finished? Also, I don't see the relation between this repository and the quicrq. You reimplemented the quicrq project in the libquicr?

Thanks in advance!

Use control data descriptor for all pub/sub messages

Create control data descriptor (e.g., stream) using updated transport API as detailed in Quicr/transport#88. The control data descriptor should be unidirectional for now. All pub/sub (control) messages should use this descriptor. No published objects are sent over this. Priority should be set to the highest at 0.

Server: Using namespace only for state map key results in stale entries till on_connection_status() is called

When a client abruptly terminates, such as sigterm/fault/etc. the server side will not know the connection has dropped until idle timeout. The current idle timeout is 30 seconds. During this time, when the client reconnects it uses the same namespaces, such as when publishing. Since the old connection has not timed out yet, state is still there with the old connection for the same publish intent. This results in the new connection not working correctly because of the old state.

Fix is to replace the current state with the latest publish intent that updates it.

2023-12-06T08:43:46.224086 [INFO] [CMGR] [QSES] [QUIC] [PQIC] icoquic/loss_recovery.c:520 [picoquic_retransmit_needed_packet]: Too many retransmits of packet number 46, disconnect
2023-12-06T08:43:46.778406 [INFO] [CMGR] [QSES] [QUIC] remote: 127.0.0.1 port: 54767 conn_id: 5192594944 retransmits increased, delta: 2 total: 38
2023-12-06T08:43:47.219077 [INFO] [CMGR] [QSES] [QUIC] Closing connection stream_id: 0 Idle timeout 127.0.0.1Delete data context 0 in conn_id: 5192594944
2023-12-06T08:43:47.219175 [INFO] [CMGR] [QSES] [QUIC] Delete default context for conn_id: 5192594944, closing connection
2023-12-06T08:43:47.219233 [DEBUG] [CMGR] [QSES] connection_status: conn_id: 5192594944 status: 3
2023-12-06T08:43:47.219284 [INFO] [CMGR] [QSES] Removing state for context_id: 5192594944

Server should create stream using priority of first sent object to subscription

Currently picoquic only sets priority for the entire stream and not per SFRAME. When the first object is sent to a subscriber, a stream will be created using the publisher priority for that object.

Considering the publisher can change priorities at any time per object, we will NOT create new stream when the publisher changes. We will only match and use the publisher priority based on the first object being sent and on new_stream.

Publish Intent from new connection is being removed by previous connection

When a client abruptly terminates (crash, sigterm, ...) there will be no graceful close. Even with graceful close, there is a delay in removing the intent. If the client connects and sends publish intent, it will not be added to the state since it already exists and later it could be removed when the old connection is cleaned up.

Correct handling to support fast reconnects and lingering old connections.

Add MoQ message encode/decode.

MoQ message needs encoder and decoder support.

There is start of this on this dev branch: https://github.com/Quicr/libquicr/blob/moq-suhas-feb2024/src/moq_message.cpp

We should see if it makes sense to use it as-is or rewrite as needed

Investigate and remove conn_id and data_ctx_id from server delegate

Transport connection ID and data context flow ID might not be needed for delegate callbacks and server methods. See about removing those.

MOQT: Implement moqt protocol state machine

Change the current libquicr implementation to be compliant with MOQT protocol state machine.

Most of libquicr works in a similar fashion to MOQT with the following changes:

Add additional connection parameters
Track alias needs to be created and conveyed via subscribe, server will accept (always) or deny with subscribe error
Publisher does not start sending after announce (aka publish intent). It waits till it receives a subscribe from the relay.
Consider sending announce cancel long with unannounce
Publisher matching will only be on namespace, not on track name. When the relay receives data from publisher, it will publish with a track alias that was created based on a matching received subscribe that includes both track namespace and name. The first and subsequent subscribes will be associated to this publisher track alias, enabling received objects with that track alias to be forwarded to all subscribers of the same track namespace/track name.
Publisher will not be able to send other tracks till it gets a subscribe from the relay for a specific track. Relay has no idea what to subscribe to (e.g., the relay knows nothing about namespaces or track names), therefore the relay requires some other client to first subscribe for it.
When subscribers are zero (similar to a reference count for a given object), the relay will unsubscribe

Support different publisher stream mapping modes

MoQ needs streaming modes defined by publisher to be supported end to end. There are 3 different modes

Stream per track
Stream per object
Stream per group.

We need to update the code to match the messaging format.

Control Stream and Object Stream Mapping

This is a meta issue to understand the overall scope and various sub-issues will capture the individual tasks needed.

This needs following tasks as per discussions with, Tim, Scott and Suhas

(1) Data Context (Transport )

replaces stream context
maps to the namespace
current app_data stream Id value
metrics
priority_queue for TX
Buffers for Write (current object in progress)
congestion_state
pacing info ( not for demo )
Report on transport send/recv rates

(2) Control Stream for Clients or Peering Support

Unidirectional QUIC Stream per Peer
There is one control stream for the QUICR Session.
Allow resetting when the control streams gets corrupted ( QUIC Stream state goes totally
messed up) or stuck in retransmissions.

(3) AppData Stream

Undirectional QUIC Stream
Being able to reset or close a given unidirectional stream

(4) Libquicr

Publish Named Object carries the Delivery Preference
Add 3 bits header to carry the delivery preference
API to transport indicates ( bool create_stream, bool flush_buffers )


c_s , f_b --> expected behavior
 
0   0  --> current stream and buffers, time_queue will expire based on TTL (is default for audio, existing objects within a group other than the object 0)

 0   1  -->  used per group change, don't replace current stream,  reset priority queue TX buffer,  flush Write buffer ( for video )

 1   0 --> used per group change, replace current stream, flush Write buffer, don't reset priority queue 

 1   1 --> user per group change, replace current stream, flush write and pq for tx ( start all fresh for a new group) ( for video )

RX Context

Buffers for RX
timed_queue
Stream Id

(4) Qmedia/Decimus

need a way to indicate the right boolean values for create_stream and flush_buffers

(5) Stream Per Group

Use one AppData Stream per group change
Reset an AppData Stream for a given group when a new group arrives and existing group is still not finished.

Transport not obeying shutdown while connecting

When told to disconnect, if the transport is in the middle of trying to connect, everything else obeys he command, but transport keeps on connecting away, until it times out finally, and realizes it was told to shutdown.

Steps to reproduce (using Decimus):

Set relay address to somewhere random where it could never possibly connect.
Attempt to connect to a meeting
Leave the meeting while it is connecting
Watch logs, transport will still be logging it is trying to connect (at least with QUIC)

UDP actually just seems to keep trying forever and ever, never lets go.

First discussed: #56 (comment)

Update libquicr to use transport API changes for data descriptor id

The transport API uses data descriptor ID instead of context_id and stream_id. Update libquicr client, server, and test/cmd code to use this updated API.

See Quicr/transport#88 for more details.

Support for QUIC Varint

Move our implementation of encoding varint to RFC9000

this can be a decent start : https://github.com/Quicr/libquicr/blob/moq-suhas-feb2024/src/encode.cpp#L97

Support Subscribe Locations

We don't have a way to answer things from a given starting location and end at a provided end location.
This needs 2 major changes

Add suppport for locations in subscriptions
Add caching support to look for the objects matching the location filters.

MOQT: Implement stream buffer to support parsing uintvar message headers

MOQT does not have a message header length field, only a series of uintvar params. This requires libquicr to parse received messages uintvar by uintvar to get the size of the header and data, even with control messages.

Implement stream buffer APi changes and refactor existing messages to support stream buffer parsing instead of received objects (HDR + data).

Rename transport context_id to conn_id

context_id is the connection Id. Rename context_id to conn_id.

See Quicr/transport#85 for more details.

Add priority to QuicR header on publish object

Currently priority is locally significant only. The remote side, such as the relay, is unaware of the priority on received objects.

Add priority to publish object header.

Export libquicr and transport metrics to influxDb

Implement transport API to collect and export metrics to influxDB. Libquicr will add namespace and other tags/fields to the metrics from transport to make it possible to correlate metrics with Decimus.

Implement subscriber resume/pause message and relay support

Decimus will use a new Qmedia API to instruct stopping or starting of subscriptions.

Transport mode Pause and Resume will be added to instruct the relay to stop/start sending data for a subscription.

Pause/Resume is only valid for existing subscription that has a valid subscribe() with the desired transport mode as reliable/unreliable/... Pause/Resume is just to start/stop data flows for the subscription and will not be propagated via peers/etc. Qmedia will send subscribe() with transport mode to Pause to stop flows. Qmedia will start data flow using subscribe() with Resume transport mode. Qmedia will send a subscribe based on start/stop to libquicr.

clang-tidy stripping headers

clang-tidy is stripping headers causing build failures for macos builds.

Refactor subscriber ID to be used both for publishers and subscribers

Transport connection ID and data context ID are passed to server app/relay. It's used only for connection correlation, such as to remove duplicates from the same connection and to prevent same messages going back to the source. This could be replaced using subscriber ID but it should be renamed to client Id instead. The context can be updated to track subscriber or publisher or both.

Implement per group streams

Update the publish API to toggle per-group streams for a given namespace. Use the new transport API replaceStream() to change streams on group change. See Quicr/transport#88 for more details.

Implement draft-ietf-moq-transport-04

Implement all control messages and all object data stream modes.

Refactor libquicr quicr client/server into moq instance that can be used by both server and client. Refactor API to be track based. Abstract in the API much of the work and state that MOQT requires. The client should have simple subscribe and publish track methods to establish data flows.

Ensure metrics continue to work with MOQ implementation.

Create new mod-example client/server programs to show how to use the client APIs for both client and server. Include a date publishing/subscribe implementation in the example.

Update the README for MOQ.