deveel / deveel.webhooks Goto Github PK

Describe the Improvement

Align the multi-tenancy scenarios to use the Finbuckle Multi-Tenant library, removing the in-house development

Expected Benefits

The Finbuckle Multi-Tenant library is widely used to implement tenant context resolution scenarios, having a good community behind and a rich set of configurations and extensions.

The adoption of the library to replace the implementation of the tenant selection done from within the Webhooks library should release some of the supporting charges from the project, while at the same time enabling further tenant configurations (databases) resolution scenarios.

Possible Side Effects

• The design of the library is currently very bound to the concept of "Tenant" in domain layer, requiring the tenant identifier in every call: adopting a Tenant Context from Finbuckle.MultiTenant's design would disrupt this approach.
• The database implementation of the MongoDb storage designs the tenant handling itself, that might be disrupted by the adoption of the Finbuckle.MultiTenant design

Currently, the Deveel.Webhooks library does not support receiving webhooks from Twilio, a cloud communications platform.
This issue is to add support for Twilio webhooks, which are used to receive SMS and voice call notifications. Adding support for Twilio webhooks will make it easier to build integrations with Twilio and enable real-time updates for SMS and voice call applications and services.

Expected Deliverables - The contributor will need to implement a new Deveel.Webhooks.Receive.Twilio library to add support for receiving webhooks from Twilio. This will involve adding a new implementation of the receiver to handle Twilio webhook payloads, as well as modifying the existing webhook receiving code to detect and handle Twilio webhook requests.
The contributor will also need to update the library documentation to reflect the new capability.

Difficulty Level - Moderate - The contributor will need to have some familiarity with the Twilio API and the process for setting up and verifying Twilio webhooks. However, there are plenty of resources available to help them understand the Twilio API and how to use it.

Resources:

• Twilio Webhooks documentation
• Twilio API documentation
• Deveel.Webhooks documentation

Contact Information: If you have any questions or need guidance, please reach out to me at [antonello at deveel dot com].

The current set of ASP.NET Core receivers allows two different types of webhook handlers:

1. Handlers registered as services and resolved by a factory/based middleware, that does not accept parameters
2. Handlers defined directly in the convention-based middleware as callbacks

We should extend the design to have a convention-based middleware that receive handlers from dependency injection and accepts arguments to configure its execution

Example:

public class Program {
    public void ConfigureServices(IServiceCollection services) {
        services.AddWebhookReceiver<MyWebhook>()
            .AddHandler<MyHandler>();
    }

    public void Configure(IApplicationBuilder app) {
        app.UseWebhookReceiver<MyWebhook>("/webhooks" new WebhookReceiverOptions {
        });
    }
}

While implementing the Facebook Receiver, major changes were introduced in the design of the Deveel.Webhooks.Receiver.AspNetCore, by making the parsers and the signers not being injected in the container anymore, to better control the behavior of the receivers.

A similar design would be beneficial for the senders, making a cleaner architecture of the senders, reducing the insecurity of the serialization, and signature, opening to a new method to configure senders.

When sending webhooks requests to receivers, the sender service should introduce a set of headers to correlate the delivery session and the delivery attempts.

This addition should help both the sender and the receiver to better trace issues that might occurr during the delivery process of a webhook.

The current framework is implemented on top of the .NET 6.0 framework, which has been put on Long Term Support (LTS) by Microsoft, the first phase in the process of retirement.

We should aim to add support for the .NET 7.0 and potentially the .NET 8.0 frameworks, to make the Deveel Webhooks future-proof.

Expected Benefits

The newer versions of the .NET framework improve the performances of several of the low-level components, resulting in improved performances of the applications.

Even not rewriting the code of the Deveel Webhooks framework, to use the latest constructs of the .NET, we might benefit by providing assemblies that are built against the newer versions of the .NET framework.

Possible Side Effects

Describe the Improvement

Adopt an alternative implementation of the Repository Pattern used by the webhook subscription management to interface Mongo databases.

Expected Benefits

Removing the custom implementation of the Mongo persistency layer from the code of the framework, adopting an external library (such as MongoFramework) would greatly reduce the load on the project, helping to set the focus on the core value of the framework.

Possible Side Effects

The multi-tenant context resolution might cause some issues, since the logic of this framework is to operate in a background with an identifier of the tenant being passed as parameter of functions: the Mongo data layer implemented right now has a design that allows the construction of the stores using an explicit parameter, while implementations like MongoFramework rely on Finbuckle to resolve this information.

The current implementation of the Webhook Subscription Management feature uses a storage model that is based on an internet contract, based on the Repository pattern (IWebhookSubscriptionStore).

Deveel has recently released the Deveel.Repository project, implementing all the functions for the management of the WebhookSubscription and WebhookDeliveryResult entities.

Expected Benefits

Replacing the existing storage model with the one provided by Deveel.Repository would introduce the best practices and methods provided externally, unifying the development and maintenance efforts for both projects.

Adopting the Deveel.Repository as the model for the management of stores will also align the multi-tenant database management.

Possible Side Effects

The Deveel.Repository project has not been used (at the moment of this issue, 10/13/2023) by external projects that would extend its base CRUD functions, and at its possible that the migration will cause some incompatibilities.

Description
The evaluation of the webhooks for filtering the subscriptions builds a representation of the webhooks into an anonymous object, to make the Dynamic LINQ filter to be executed.
When doing so, a Dynamic Assembly and a Dynamic Module are created to define the objects resulting from the serialization of the webhook in its DTO format, to run the subscribers filters against the object that ultimately they will receive (or not), instead of doing against the .NET object (JSON/XML serialization might change drastically the naming or format of the data).

When the structure of the webhook is complex and composed of more than one complex object, the process fails to run the filter, since the two complex objects (dyanmically defined as anonymous types) are defined in two different assembly/modules.

To Reproduce
Steps to reproduce the behavior:

1. Define a webhook that is composed by more than one property with a non-primitive data type (eg. PropertyA and PropertyB)
2. Create one or more filters for that type of the webhook, that uses one or more of the fields defined by the second property (eg. PropertyB)
3. Run a notification process

Expected behavior

The filter should not fail when defined against the fields of a complex property of a webhook.

The current implementation of the MongoDb and the Entity Framework storage layer libraries use a custom-made implementation to support multi-tenancy scenarios, although it is based on the MongoFramework provisioning of the support.

We should consider adopting the streamlined approach of using Finbuckle.MultiTenant framework to support the multi-tenant scenarios to generalize even more capabilities.

Although the MongoDB version is using the mentioned Finbuckle.MultiTenant framework, its usage is not really clean, since the need to resolve the tenant in the background.

Expected Benefits

Relying on a mature and widely used library to support the multi-tenant scenarios should relieve the project from the load of a few important tasks, and it would allow users of the libraries to configure the infrastructure just once.
The native and constant support of Finbuckle.MultiTenant for the EntityFramework will also result as a benefit to the project, that will not have to deal with a custom implementation for that storage layer.

Possible Side Effects

Major changes to the MongoDb storage layer will be necessary, and a minor version increase is recommended.

Further Reading

• Finbuckle.MultiTenant - https://www.finbuckle.com/MultiTenant
• EntityFramework - https://learn.microsoft.com/en-us/aspnet/entity-framework

Create a new webhook receiver implementation in Deveel.Webhooks for receiving SMS delivery reports from the InfoBip SMS provider. The implementation should conform to the Deveel.Webhooks receiver API and should be able to receive, parse and store delivery reports from the InfoBip service.

The receiver should implement the following features:

• Parse and validate the incoming webhook requests from the InfoBip service.
• Map the incoming webhook data to a strongly-typed C# object that can be used to represent the delivery report.
• Store the delivery report data to a database or another type of persistence mechanism.
• The implementation should also include relevant tests for ensuring the correctness of the implementation.

Expected Skills

• C# programming language
• ASP.NET Core web development
• Experience working with webhooks and API integrations
• Familiarity with InfoBip SMS API and documentation is a plus

Additional Information

• InfoBip SMS Delivery Reports Webhook API Documentation: https://dev.infobip.com/docs/sms-delivery-reports-webhook
• Deveel.Webhooks GitHub Repository: https://github.com/deveel/deveel.webhooks

Please feel free to ask any questions if you need more information.

The notification service of the Deveel.Webhooks project is currently supporting only multi-tenant scenarios for the notifications, which requires additional configurations of the applications, to circumvent the limitation (eg. adding the multi-tenant support and use only one static tenant).

Although this is a feature that has sense in large-scale scenarios, in most of the situations this might be excessive and hard to handle.

The proposed solution might provide one or more of the two options:

• Multi-Tenant Notifier - A notifier service that is requiring the identifier of the tenant to resolve webhook subscriptions, provided as alternative to another service that will not require this input and use the regular storage
• Implicit Resolution of the Tenants - The notifier service will not require the unique identifier of the tenant to notify, scanning for all tenants registered in the current context, notifying to all subscriptions matching the event
• Tenant Identifier in Event - The EventInfo structure might be extended with an optional the identifier of the tenant of the subscriptions to the event, to let the notifier identify and resolve subscriptions specific for that context

The current implementation of the WebhookSubscriptionBuilder also allows the specification of the types and storage of the Delivery Results, while they are conceptually separated entities.

While some storage layers support both Webhook Subscriptions and Delivery Result logs (eg. MongoDB), this is not a given, and some database systems are not really designed for storing a large number of logs (eg. SQL relational databases).

Expected Benefits

We expect that the design will be cleaner and the separation will be more evident to the consumers of the framework.

Possible Side Effects

Currently, the Deveel.Webhooks library supports webhook notifications in a variety of formats, including JSON and XML.

This feature request is to add support for CloudEvents, a standard format for describing event data in a common way. CloudEvents make it easier to build event-driven systems that are interoperable across different platforms and services.

Background: Webhooks are a mechanism for notifying an application or service about events that occur in another application or service. They are commonly used for integrating third-party services and applications with each other. However, there is no standard format for webhook payloads, which can make it difficult to integrate different services and applications. In response to this problem, the Cloud Native Computing Foundation (CNCF) created the CloudEvent standard in 2018. CloudEvents provide a way to describe event data in a common format, making it easier to build event-driven systems that can work with different services and platforms.

Expected Deliverables - The contributor will need to modify the Deveel.Webhooks library to support CloudEvents as a notification format. This will involve adding a new class to handle CloudEvent payloads, as well as modifying the existing webhook notification code to use the CloudEvent format when requested. The contributor will also need to update the library documentation to reflect the new capability.

Difficulty Level - Moderate - The contributor will need to have some familiarity with the CloudEvent standard and how it is used. However, there are plenty of resources available to help them understand the standard and how to use it.

Resources - The following resources may be helpful for the contributor:

• CloudEvents documentation
• Introduction to CloudEvents
• Deveel.WebHooks documentation

Contact Information: If you have any questions or need guidance, please reach out to me at [antonello at deveel dot com].

Describe the Improvement

Implement a design that binds the processings of webhooks (and CloudEvents) throw ASP.NET Core middleware

Expected Benefits

This approach should leverage the scalability of the ASP.NET Core middelware design, allowing scalability to multiple structure of webhooks (or CloudEvents), handlers of the received webhooks, and eventually other providers (such as Twilio, SendGrid, SendinBlue, MailJet, etc.).

Possible Side Effects

Microsoft has stopped supporting the individual ASP.NET Core libraries, now requiring web projects to depend on the full ASP.NET Core SDK.

The current implementation of the Webhook Receivers (e.g.. SendGrid, Twilio, Facebook) libraries that are reading from HTTP requests depend on some of the discontinued ASP.NET Core libraries: although this is not an immediate issue, it might become a demand to transition to the new pattern.

Expected Benefits

The Webhook Receiver libraries will be future-proof (at least for a while: until Microsoft won't change the dependency strategy again), and will not cause incompatibility issues with the new versions of the SDKs.

Possible Side Effects

Applications using the older versions of the SDKs might be forbidden from using the new Deveel Webhooks Receiver libraries

Further Reading

• Use ASP.NET Core APIs in a class library

Describe the Improvement

Upgrade the the libraries to the .NET 6.0 framework

Expected Benefits

The libraries in the project are currently build on top of the .NET 5.0, that is going to be retired soon and that at the moment is not receiving any further updates: the update should avoid having situations of not usability of the library in the short-to-mid-term horizon

Possible Side Effects

Currently the notifier is notifying the receivers for each single event, potentially creating situations that might overwhelm receivers with an eccessive amount of webhooks.

The framework should provide a mechanism for accummulating batches of webhooks for each receiver, and notify them aggregated in one single request.

The main challenges posed by this idea are:

• The establishment of a temporary storage for the aggregation of the webhooks to be batched
• Fire up the delivery of the aggregated webhooks even when the maximum size of the batch is not reached, but a configured maximum time to wait has expired (to prevent an event being notified too late in time) - this can be implemented by a background process, that is not suitable for every scenario
• Design of the webhook batch element - while at the moment the Webhook Factory constructs only one webhook at a time, we will need a new design to obtain a meta object that carries an array of webhooks

** Current Context **
The logic of building instance of the HttpClient class to delivery Webhooks to the URLs of the subscribers has been implemented ad-hoc in the default implementation of the IWebhookSender service, using the configurations of the subscriber to instantiate them with base URLs, headers, retries and other elements.

Ideally, we should have used the IHttpClientFactory pattern from the .NET framework, but the complexity of the case doesn't provide this option:

• The IHttpClientFactory is a singleton within the application - although it's possible to segregate the configurations (using named clients or typed clients), it doesn't provide (at least transparently) dyanmic configurations at runtime
• Each destination of the delivery provides specific configurations, that are relevant for the context of the delivery (eg. retries, fallbacks, etc.), that cannot be shared with other receivers
• Such configurations are resolved only at runtime, once the receiver of a webhook is resolved from the event subscription

** Proposed Solution **

We should implement an abstraction that can be used to create HttpClient instances based on the parameters from the receivers.

• The abstraction should be generic and reusable
• The default implementation of the factory must hold a reference to the constructed clients, to dispose them once the scope has expired
• The default implementation of the factory should be able to recycle the same instance of an HttpClient for a given subscription (not just by destination), to avoid the proliferation of connections (pooling)

Expected Benefits

The implementation of such model would simplify the code of the WebhookSender, making the HttpClient factory a separate provisioning of the framework, reusable from other implementations.

Possible Side Effects

No negative side effects are identified at this point in time.

We should receive an alternative for the storage of webhook subscriptions and webhook logs to SQL database, handled by the Entity Framework (eg. Azure SQL Server, Postgres, etc.).

Expected Deliverables - The contributor will need to provide a new library Deveel.Webhooks.EntityFramework to add a data persistence layer using EntityFramework Core. This will involve adding new classes and modifying existing ones. The contributor will also need to create a database schema to store the subscription and delivery result data, and write the necessary EntityFramework Core code to interact with the database.

Difficulty Level - Moderate - The contributor will need to have some experience with EntityFramework Core and database design. However, there are plenty of resources available to help them understand the concept of EntityFramework Core and how to use it.

Resources - The following resources may be helpful for the contributor:

1. EntityFramework Core documentation
2. Getting started with EntityFramework Core
3. Introduction to database design
4. Deveel.Webhooks documentation

If you have any questions or need guidance, please reach out to me at [antonello at deveel dot com].

Create a new webhook receiver implementation in Deveel.Webhooks for receiving SMS delivery reports and status updates from the Link Mobility SMS provider. The implementation should conform to the Deveel.Webhooks receiver API and should be able to receive, parse and store delivery reports and status updates from the Link Mobility service.

The receiver should implement the following features:

• Parse and validate the incoming webhook requests from the Link Mobility service.
• Map the incoming webhook data to a strongly-typed C# object that can be used to represent the delivery report or status update.
• The implementation should also include relevant tests for ensuring the correctness of the implementation.

Expected Skills

• C# programming language
• ASP.NET Core web development
• Experience working with webhooks and API integrations
• Familiarity with Link Mobility SMS API and documentation is a plus

Additional Information

• Link Mobility SMS Webhooks Documentation: https://linkmobility.atlassian.net/wiki/spaces/LIF/pages/814101518/SMS+Webhooks
• Deveel.Webhooks GitHub Repository: https://github.com/deveel/deveel.webhooks

Please feel free to ask any questions if you need more information.

The Deveel.Webhooks project currently does not support MailJet webhooks and emails. This issue is to add support for MailJet so that users of the project can receive and process incoming webhooks and emails from that provider in ASP.NET Core applications.

Tasks

• Implement a new package Deveel.Webhooks.Receiver.MailJet with middleware components to handle incoming webhooks and emails from MailJet.
• Write a parser for MailJet webhook events and a handler to process them. See the MailJet documentation on webhooks for more information on the event format.
• Write a parser for MailJet email events and a handler to process them. See the MailJet documentation on emails for more information on the email event format.
• Update the documentation to include examples and instructions for how to use the new MailJet package.

Expected Outcome
The Deveel.Webhooks project will have support for MailJet webhooks and emails, allowing users to receive and process incoming webhooks and emails from MailJet using the new Deveel.Webhooks.Receiver.MailJet package.

** Difficulty** - Easy

SQL Server database table creation issue for the following foreign keys :-

Table - [webhook_delivery_results]

Cannot create the foreign key "FK_webhook_delivery_results_event_info_event_id" with the SET NULL referential action, because one or more referencing columns are not nullable.

Cannot create the foreign key "FK_webhook_delivery_results_webhook_receivers_receiver_id" with the SET NULL referential action, because one or more referencing columns are not nullable.

Cannot create the foreign key "FK_webhook_delivery_results_webhooks_WebhookId" with the SET NULL referential action, because one or more referencing columns are not nullable.

Code example:

builder.Services.AddWebhookSubscriptions<DbWebhookSubscription>()
    .UseEntityFramework(ef => 
        ef.UseContext(options => 
            options.UseSqlServer(builder.Configuration.GetConnectionString(nameof(WebhookDbContext)))));

using (var scope = app.Services.CreateScope())
{
    var dataContext = scope.ServiceProvider.GetRequiredService<WebhookDbContext>();
    dataContext.Database.EnsureCreated();   
}

The foreign key constraints possible requires to be delete on cascade?

To optimize the performances in the notification of webhooks to subscribers, the service should provide a cache that optimizes the I/O operations and queries towards the database system.

The first level of resolution should be the Tenant of the subscriptions, and then the Event Type within the scope.

Additional filters in a webhook subscription cannot be used as keys to cached items, since it's unpredictable if they would meet the conditions given, not being assured to be constant.

Describe the Improvement

Use a new design for constructing a Webhook Sender service, using the builder pattern.

Expected Benefits

A new and controlled builder will make the configuration process cleaner and more scalable

Possible Side Effects

• The redeisng of the building patterns of the webhook sender might likely cause a cascading effect in the design of the dependencies needed by the sender service.
• It is possible that the redesign will separate the core functions of the sender from the servicing of the webhook management (eg. subscription management, notifications, etc.)

Currently, the Deveel.Webhooks library does not support receiving webhooks from Facebook.
This issue is to add support for Facebook webhooks, which are used to receive real-time updates from the Facebook platform. Adding support for Facebook webhooks will make it easier to build integrations with Facebook and enable real-time updates for applications and services.

Expected Deliverables - The contributor will need to provide a new Deveel.Webhooks.Receiver.Facebook library to add support for receiving webhooks from Facebook. This will involve adding a new class to handle Facebook webhook payloads, as well as implementing a new webhook receiver detect and handle Facebook webhook requests, and provide some utility methods to hook this feature into ASP.NET Core applications.
The contributor will also need to update the library documentation to reflect the new capability.

Difficulty Level - Moderate - The contributor will need to have some familiarity with the Facebook Graph API and the process for setting up and verifying Facebook webhooks. However, there are plenty of resources available to help them understand the Facebook API and how to use it.

Resources - The following resources may be helpful for the contributor:

• Facebook Webhooks documentation
• Facebook Graph API documentation
• Deveel.Webhooks documentation

Contact Information: If you have any questions or need guidance, please reach out to me at [antonello at deveel dot com].

The webhook handling system is executing callbacks to the handlers only in sequence, that might cause one or more of them not being executed if a previous one failed: this behavior might have sense in some scenarios, in which the consumer wants to be assured that a given function is not invoked if a previous one wasn't executed successfully.

Anyway, in some other scenarios the sequence order is less meaningful and the implementation should execute parallel handling of the webhook, isolating the single status of each process.

We should provide a mechanism for configuring the execution strategy when handling received webhooks.

Currently, the Deveel.Webhooks library does not support receiving webhooks from SendGrid, an email delivery platform.
This issue is to add support for SendGrid webhooks, which are used to receive emails and email delivery notifications. Adding support for SendGrid webhooks will make it easier to build integrations with SendGrid and enable real-time updates for email delivery applications and services.

Expected Deliverables - The contributor will need to create the Deveel.Webhooks.Receiver.SendGrid library to add support for receiving webhooks from SendGrid. This will involve adding a new class to handle SendGrid webhook payloads, as well as extending the ecisting webhook receiving code to detect and handle SendGrid webhook requests, and hooking up this feature in ASP.NET Core applications.
The contributor will also need to update the library documentation to reflect the new capability.

Difficulty Level - Moderate - The contributor will need to have some familiarity with the SendGrid API and the process for setting up and verifying SendGrid webhooks. However, there are plenty of resources available to help them understand the SendGrid API and how to use it.

Resources:

• SendGrid Webhooks documentation
• SendGrid API documentation
• Deveel.Webhooks documentation

Contact Information: If you have any questions or need guidance, please reach out to me at [antonello at deveel dot com].

The current implementation of the service tries to reach a single destination URL for the delivery of a webhook, trying a number of configured times before failing.

In some challenging and mission-critical scenarios this might not be a good approach, since the receivers might be distributed between multiple services, for load balancing and tolerance.

It should be possible to configure an array of possible destinations for the delivery, and allowing to specify one of the following strategies:

• First Available Receiver - The service attempts to deliver to the webhook to the destinations in sequence, stopping when the first of them responded successfully; optionally, we might want to implement a ping function to test the receiver before reaching it
• Random - Select a random destinations between the ones configured and then attempt the number of times, until success or failure

The current implementation of the service is tightly bound to the JSON format and this would be limiting in some scenarios in which the receivers would want to accept contents of other types (eg. XML, www-form-urleconded).

This would also mean that subscribers might be provided the possibility to choose the format of the webhook, between those available from the provider.

In a later development of the design, we added support for the XML format.

Expected Deliverables - The contributor will need to modify the Deveel.Webhooks library to support for the www-form-urlencoded . This will involve adding new classes and modifying existing ones. The contributor will also need to update the documentation to explain how to use the new feature.

Difficulty Level - Easy
The contributor should have knowledge of the URL encoding for the serialization of complex objects that represent the webhooks.

Resources - The following resources might be helpful for the contributor

1. MSDN Documentation: The official documentation for the .NET Framework provides an overview of serialization, including how to serialize objects to www-form-urlencoded format using the HttpUtility class. You can find the documentation here: https://docs.microsoft.com/en-us/dotnet/standard/serialization/how-to-serialize-an-object-to-query-string-data.

2. Stack Overflow: Stack Overflow is a popular Q&A site for programming questions, and there are many questions and answers related to serializing .NET objects to www-form-urlencoded format. Here's a link to a question that may be helpful: https://stackoverflow.com/questions/2142535/how-to-serialize-object-to-query-string-using-reflection.

3. CodeProject: CodeProject is a community of developers who share their knowledge and code samples. Here's an article that provides an example of how to serialize a .NET object to www-form-urlencoded format: https://www.codeproject.com/Articles/1103822/Serialize-NET-object-to-www-form-urlencoded-string.

4. NuGet Packages: There are several NuGet packages available that provide serialization functionality, including QueryStringConverter and WebHelpers. You can search for these packages on the NuGet website or in the NuGet Package Manager in Visual Studio.

If you have any request of information, please reach me at [antonello at deveel dot com].

Provide a set of sample projects to describe how to use the framework, for sending webhooks directly, notifying subscribers and receive notifications

This should help the potential users to better understand the logic behind the framework, how to instrument their applications, how to enable them to send and receive webhooks / events.