ikyriak / idempotentapi

A .NET library that handles the HTTP write operations (POST and PATCH) that can affect only once for the given request data and idempotency-key by using an ASP.NET Core attribute (filter).

Idempotent API (v2.4.0)

Understanding Idempotency

A distributed system consists of multiple components located on different networked computers, which communicate and coordinate their actions by passing messages to one another from any system. For example, I am sure that you have heard of the microservices architecture, which is a kind of distributed system.

Creating Web APIs for distributed systems is challenging because of distribution pitfalls such as process failures, communication failures, asynchrony, and concurrency. One common requirement and challenge when building fault-tolerant distributed applications is the need to be idempotent.

• In mathematics and computer science, an operation is idempotent when applied multiple times without changing the result beyond the initial application.
• Fault-tolerant applications can continue operating despite the system, hardware, and network faults of one or more components, ensuring high availability and business continuity for critical applications or systems.

Idempotence in Web APIs ensures that the API works correctly (as designed) even when consumers (clients) send the same request multiple times. For example, this case can happen when the API failed to generate the response (due to process failures, temporary downtime, etc.) or because the response was generated but could not be transferred (network issues).

Imagine a scenario in which the user clicks a “Pay” button to make a purchase. For unknown reasons, the user receives an error, but the payment was completed. If the user clicks the “Pay” button again or the request is re-sent by a retry library, we would result in two payments! Using idempotency, the user will get a successful message (e.g., on the second try), but only one charge would be performed.

Creating Idempotent Web APIs is the first step before using a resilient and transient-fault-handling library, such as Polly. The Polly .NET library allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner.

The IdempotentAPI library provides an easy way to develop idempotent Web APIs. In the following sections, we will see the idempotency in the different HTTP methods, how the IdempotentAPI library works, its code, and finally, how to use the IdempotentAPI NuGet packages.

Idempotency in HTTP (Web)

HTTP defines a set of request methods (HTTP verbs: GET, POST, PUT, PATCH, etc.) to indicate the desired action to be performed for a given resource. An idempotent HTTP method can be called many times without resulting in different outcomes. Safe methods are HTTP methods that do not modify the resources. In Table 1, we can see details about which HTTP methods are idempotent or/and safe.

Table 1. - Idempotent or/and Safe HTTP methods (verbs).

HTTP Method	Idempotent	Safe	Description
GET	Yes	Yes	Safe HTTP methods do not modify resources. Thus, multiple calls with this method will always return the same response.
OPTIONS	Yes	Yes	Same as the previous HTTP method.
HEAD	Yes	Yes	Same as the previous HTTP method.
PUT	Yes	No	The PUT HTTP method is idempotent because calling this HTTP method multiple times (with the same request data) will update the same resource and not change the outcome.
DELETE	Yes	No	The DELETE HTTP method is idempotent because calling this HTTP method multiple times will only delete the resource once. Thus, numerous calls of the DELETE HTTP method will not change the outcome.
POST	No	No	Calling the POST method multiple times can have different results and will create multiple resources. For that reason, the POST method is not idempotent.
PATCH	No	No	The PATCH method can be idempotent depending on the implementation, but it isn’t required to be. For that reason, the PATCH method is not idempotent.

Idempotent Web Consumer

The creation of an idempotent consumer is an essential factor in HTTP idempotency. The API server would need a way to recognize subsequent retries of the same request. Commonly, the consumer generates a unique value, called idempotency-key, which the API server uses for that purpose. In addition, when building an idempotent consumer, it is recommended to:

• Use "V4 UUIDs" for the creation of the idempotency unique keys (e.g. “07cd2d27-e0dc-466f-8193-28453e9c3023”).
• Use techniques like the exponential backoff and random jitter, i.e., including an exponential and random delay between continuous requests.

The IdempotentAPI Library

The IdempotentAPI is an open-source NuGet library, which implements an ASP.NET Core attribute (filter) to handle the HTTP write operations (POST and PATCH) that can affect only once for the given request data and idempotency-key.

How IdempotentAPI Works

The API consumer (e.g., a Front-End website) sends a request including an Idempotency-Key header unique identifier (default name: IdempotencyKey). The API server checks if that unique identifier has been used previously for that request and either returns the cached response (without further execution) or save-cache the response along with the unique identifier. The cached response includes the HTTP status code, the response body, and headers.

Storing data is necessary for idempotency, but if the data are not expired after a certain period, it will include unneeded complexity in data storage, security, and scaling. Therefore, the data should have a retention period that makes sense for your problem domain.

The IdempotentAPI library performs additional validation of the request’s hash-key to ensure that the cached response is returned for the same combination of Idempotency-Key and Request to prevent accidental misuse.

The following figure shows a simplified example of the IdempotentAPI library flow for two exact POST requests. As shown, the IdempotentAPI library includes two additional steps, one before the controller’s execution and one after constructing the controller’s response.

Figure 1. - A simplified example of the IdempotentAPI flow for two exact POST requests.

✔ Features

• ⭐ Simple: Support idempotency in your APIs easily with simple steps 1️⃣2️⃣3️⃣.
• 🔍 Validations: Performs validation of the request’s hash-key to ensure that the cached response is returned for the same combination of Idempotency-Key and Request to prevent accidental misuse.
• 🌍 Use it anywhere!: IdempotentAPI targets .NET Standard 2.0. So, we can use it in any compatible .NET implementation (.NET Framework, .NET Core, etc.). Click here to see the minimum .NET implementation versions that support each .NET Standard version.
• ⚙ Configurable: Customize the idempotency in your needs.
  • Configuration Options (see below for more details)
  • Logging Level configuration
• 🔧 Caching Implementation based on your needs.
  • 🏠 DistributedCache: A build-in caching that is based on the standard IDistributedCache interface.
  • 🦥 FusionCache: A high performance and robust cache with an optional distributed 2nd layer and some advanced features.
  • ... or you could use your own implementation 😉
• 🔀 Support idempotency in a Cluster Environment (i.e., a group of multiple server instances) using Distributed Locks.
  • samcook/RedLock.net: Supports the Redis Redlock algorithm.
  • madelson/DistributedLock: Supports multiple technologies such as Redis, SqlServer, Postgres and many more.
• 💪Powerful: Can be used in high-load scenarios.
• ✳ NEW ✳ - ✅ Supports Minimal APIs.

📦 Main NuGet Packages

Package Name	Description	Release
IdempotentAPI	The implementation of the IdempotentAPI library.
IdempotentAPI.AccessCache	The access cache implementation of the IdempotentAPI project.
IdempotentAPI.MinimalAPI	The implementation to support IdempotentAPI in Minimal APIs.

📦 Caching NuGet Packages

Package Name	Description	Release
IdempotentAPI.Cache.Abstractions	The cache definition of the IdempotentAPI project.
IdempotentAPI.Cache.DistributedCache	The default caching implementation, based on the standard IDistributedCache interface.
IdempotentAPI.Cache.FusionCache	Supports caching via the FusionCache third-party library.

📦 Distributed Locking NuGet Packages

Package Name	Description	Release
IdempotentAPI.DistributedAccessLock.Abstractions	The distributed access lock definition of the IdempotentAPI project.
IdempotentAPI.DistributedAccessLock.MadelsonDistributedLock	The Madelson DistributedLock implementation for the definition of the IdempotentAPI.DistributedAccessLock.
IdempotentAPI.DistributedAccessLock.RedLockNet	The RedLockNet implementation for the definition of the IdempotentAPI.DistributedAccessLock.

🌟 Quick Start

Let's see how we could use the NuGet packages in a Web API project. For more examples and code, you can check the sample projects. The IdempotentAPI can be installed via the NuGet UI or the NuGet package manager console:

PM> Install-Package IdempotentAPI -Version 2.4.0

and, register the IdempotentAPI Core services for either controller-based APIs or minimal APIs.

// For Controller-Based APIs:
services.AddIdempotentAPI();

// OR

// For Controller-Based APIs: Register the `IIdempotencyOptions` that will enable the use of the `[Idempotent(UseIdempotencyOption = true)]` option.
services.AddIdempotentAPI(idempotencyOptions);

OR

// For Minimal APIs:
builder.Services.AddIdempotentMinimalAPI(new IdempotencyOptions());

Step 1️⃣.🅰: Register the Caching Storage

As we have seen, storing-caching data is necessary for idempotency. Therefore, the IdempotentAPI library needs an implementation of the IIdempotencyCache to be registered in the Program.cs or Startup.cs file depending on the used style (.NET 6.0 or older). The IIdempotencyCache defines the caching storage service for the idempotency needs.

Currently, we support the following two implementations (see the following table). However, you can use your implementation 😉. Both implementations support the IDistributedCache either as primary caching storage (require its registration) or as secondary (optional registration).

Thus, we can define our caching storage service in the IDistributedCache, such as in Memory, SQL Server, Redis, NCache, etc. See the Distributed caching in the ASP.NET Core article for more details about the available framework-provided implementations.

	Support Concurrent Requests	Primary Cache	2nd-Level Cache	Advanced Features
IdempotentAPI.Cache.DistributedCache (Default)	✔️	IDistributedCache	❌	❌
IdempotentAPI.Cache.FusionCache	✔️	Memory Cache	✔️ (IDistributedCache)	✔️

Choice 1 (Default): IdempotentAPI.Cache.DistributedCache

Install the IdempotentAPI.Cache.DistributedCache via the NuGet UI or the NuGet package manager console.

// Register an implementation of the IDistributedCache.
// For this example, we are using a Memory Cache.
services.AddDistributedMemoryCache();

// Register the IdempotentAPI.Cache.DistributedCache.
services.AddIdempotentAPIUsingDistributedCache();

Choice 2: Registering: IdempotentAPI.Cache.FusionCache

Install the IdempotentAPI.Cache.FusionCache via the NuGet UI or the NuGet package manager console. To use the advanced FusionCache features (2nd-level cache, Fail-Safe, Soft/Hard timeouts, etc.), configure the FusionCacheEntryOptions based on your needs (for more details, visit the FusionCache repository).

// Register the IdempotentAPI.Cache.FusionCache.
// Optionally: Configure the FusionCacheEntryOptions.
services.AddIdempotentAPIUsingFusionCache();

💡 TIP: To use the 2nd-level cache, we should register an implementation for the IDistributedCache and register the FusionCache Serialization (NewtonsoftJson or SystemTextJson). For example, check the following code example:

// Register an implementation of the IDistributedCache.
// For this example, we are using Redis.
services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = "YOUR CONNECTION STRING HERE, FOR EXAMPLE:localhost:6379";
});

// Register the FusionCache Serialization (e.g. NewtonsoftJson).
// This is needed for the a 2nd-level cache.
services.AddFusionCacheNewtonsoftJsonSerializer();

// Register the IdempotentAPI.Cache.FusionCache.
// Optionally: Configure the FusionCacheEntryOptions.
services.AddIdempotentAPIUsingFusionCache();

Step 1️⃣.🅱: Register the Distributed Locks for Cluster Environment (Optional)

Currently, we support the following two implementations to support idempotency in a Cluster Environment (i.e., a group of multiple server instances) using Distributed Locks.

The DistributedLockTimeoutMilli attribute option should be used to set the time the distributed lock will wait for the lock to be acquired (in milliseconds).

	Supported Technologies
samcook/RedLock.net	Redis Redlock
madelson/DistributedLock	Redis, SqlServer, Postgres and many more.

Choice 1: None

If you do not need to support idempotency in a Cluster Environment, you do not have to register anything. So, skip this step 😉.

Choice 2: samcook/RedLock.net (Redis)

Install the IdempotentAPI.DistributedAccessLock.RedLockNet via the NuGet UI or the NuGet package manager console. The samcook/RedLock.net supports the Redis Redlock algorithm.

// Define the Redis endpoints:
List<DnsEndPoint> redisEndpoints = new List<DnsEndPoint>()
{
	new DnsEndPoint("localhost", 6379)
};

// Register the IdempotentAPI.DistributedAccessLock.RedLockNet:
services.AddRedLockNetDistributedAccessLock(redisEndpoints);

Choice 3: madelson/DistributedLock (Multiple Techonologies)

Install the IdempotentAPI.DistributedAccessLock.MadelsonDistributedLock via the NuGet UI or the NuGet package manager console. The madelson/DistributedLock supports multiple technologies such as Redis, SqlServer, Postgres and many more.

// Register the distributed lock technology.
// For this example, we are using Redis.
var redicConnection = ConnectionMultiplexer.Connect("localhost:6379");
services.AddSingleton<IDistributedLockProvider>(_ => new RedisDistributedSynchronizationProvider(redicConnection.GetDatabase()));

// Register the IdempotentAPI.DistributedAccessLock.MadelsonDistributedLock
services.AddMadelsonDistributedAccessLock();

Step 2️⃣: Decorate Response Classes as Serializable

The response Data Transfer Objects (DTOs) need to be serialized before caching. For that reason, we will have to decorate the relative DTOs as [Serializable]. For example, see the code below.

using System;

namespace WebApi_3_1.DTOs
{
    [Serializable]
    public class SimpleResponse
    {
        public int Id { get; set; }
        public string Message { get; set; }
        public DateTime CreatedOn { get; set; }
    }
}

Step 3️⃣: Set Controller Operations as Idempotent

In your Controller class, add the following using statement. Then choose which operations should be Idempotent by setting the [Idempotent()] attribute, either on the controller’s class or on each action separately. The following two sections describe these two cases. However, we should define the Consumes and Produces attributes on the controller in both cases.

using IdempotentAPI.Filters;

Using the Idempotent Attribute on a Controller’s Class

By using the Idempotent attribute on the API Controller’s Class, all POST and PATCH actions will work as idempotent operations (requiring the IdempotencyKey header).

[ApiController]
[Route("[controller]")]
[Consumes("application/json")] // We should define this.
[Produces("application/json")] // We should define this.
[Idempotent(Enabled = true)]
public class SimpleController : ControllerBase
{
    // ...
}

Using the Idempotent Attribute on a Controller’s Action

By using the Idempotent attribute on each action (HTTP POST or PATCH), we can choose which of them should be Idempotent. In addition, we could use the Idempotent attribute to set different options per action.

[HttpPost]
[Idempotent(ExpireHours = 48)]
public IActionResult Post([FromBody] SimpleRequest simpleRequest)
{
    // ...
}

Using the IdempotentAPI Endpoint Filter on Minimal APIs

The latest IdempotentAPI.MinimalAPI package should be installed and then add the IdempotentAPIEndpointFilter in your endpoints.

app.MapPost("/example",
    ([FromQuery] string yourParam) =>
    {
        return Results.Ok(new ResponseDTOs());
    })
    .AddEndpointFilter<IdempotentAPIEndpointFilter>();

⚙ Idempotent Attribute Options

The Idempotent attribute provides a list of options, as shown in the following table.

Table 2. - Idempotent attribute options

Name	Type	Default Value	Description
Enabled	bool	true	Enable or Disable the Idempotent operation on an API Controller’s class or method.
ExpireHours (Obsolete)	int	24 hours	The retention period (in hours) of the idempotent cached data. This option will be deprecated.
ExpiresInMilliseconds	double	24 hours	The retention period (in milliseconds) of the idempotent cached data.
HeaderKeyName	string	IdempotencyKey	The name of the Idempotency-Key header.
DistributedCacheKeysPrefix	string	IdempAPI_	A prefix for the DistributedCache key names.
CacheOnlySuccessResponses	bool	True	When true, only the responses with 2xx HTTP status codes will be cached.
DistributedLockTimeoutMilli	double	NULL	The time the distributed lock will wait for the lock to be acquired (in milliseconds). This is Required when a IDistributedAccessLockProvider is provided.
IsIdempotencyOptional	bool	False	Set the idempotency as optional to be introduced to existing endpoints easily (which should be backward compatible).

📚 The Source Code (IdempotentAPI)

Let’s have a quick look at the IdempotentAPI project and its main code files.

• /Core/Idempotency.cs: The core implementation containing the idempotency logic applied before and after the request’s execution.
• /Filters/IdempotencyAttributeFilter.cs: The filter implementation (of IActionFilter and IResultFilter) uses the core implementation on specific steps of the filter pipeline execution flow.
• /Filters/IdempotencyAttribute.cs: The idempotency attribute implementation for its input options and to initialize the idempotency filter.
• /Helpers/Utils.cs: Several helper static functions for serialization, hashing, and compression.

📑 Summary

A distributed system consists of multiple components located on different networked computers, which communicate and coordinate their actions by passing messages to one another from any system. As a result, fault-tolerant applications can continue operating despite the system, hardware, and network faults of one or more components.

Idempotency in Web APIs ensures that the API works correctly (as designed) even when consumers (clients) send the same request multiple times. Applying idempotency in our APIs is the first step before using a resilient and transient-fault-handling library, such as Polly.

The IdempotentAPI is an open-source NuGet library, which implements an ASP.NET Core attribute (filter) to handle the HTTP write operations (POST and PATCH) that can affect only once for the given request data and idempotency-key.

To ensure high availability and business continuity of your critical Web APIs, the IdempotentAPI library is your first step 😉. Any help in coding, suggestions, giving a GitHub Star ⭐, etc., are welcome.

📜 License

The IdempotentAPI is MIT licensed.

idempotentapi's Issues

Make all code async

Thanks @dimmy-timmy

Current code is mostly sync which could lead to thread pool starvation for highload scenarios
We have this issue currently with redis . We are observing a lot of errors like Timeout performing EVAL

PR: #47

Default name of idempotency header key

According to the rfc draft the proposed name would be Idempotency-Key so we should rename the default.

https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/

Comments?

I made a demo that can't run

I created a web api project and run the console to report an error

        [HttpPost("Create")]
        [Idempotent()]
        public async Task<IActionResult> Create(WeatherForecastModel weatherForecastDto)
        {
            return new JsonResult(weatherForecastDto.Summary);
        }
       [Serializable]
       public class WeatherForecastModel
      {
        public string Summary { get; set; }
      }

I use postman to simulate the request

Can the author make a demo that uses IdempotentAPI to simulate multiple repeated submissions that cannot be successful?

Performance Issue when using Distributed Cache in MSSQL DB

I am facing a performance issue when applying the IdempotantAPi using Distributed Cache through MSSql DB
The ExpireHours is 24

But I Found the DB Query is taking much time in some cases (Around 35 Seconds)

This is the query I have found :

DECLARE @ExpiresAtTime DATETIMEOFFSET; SET @ExpiresAtTime = (CASE WHEN (@? IS NUll) THEN @? ELSE DATEADD(SECOND, Convert(bigint, @?), @?) END);UPDATE [dbo].[?empotentCache] SET ? = @?, ExpiresAtTime = @ExpiresAtTime,? = @?, ? = @? WHERE ? = @? IF (@@rowcount = ?) BEGIN INSERT INTO [dbo].[?empotentCache] (?, ?, ExpiresAtTime, ?, ?) VALUES (@?, @?, @ExpiresAtTime, @?, @?); END

The question here is how to solve this performance issue, Also want to know why there is an update statement in DB before inserting the Record?

Possibility to configure controllers via IdempotencyOptions

There are projects that contain a large number of controllers. Because of this, important idempotency settings such as ExpiresInMilliseconds, DistributedLockTimeoutMilli are dispersed throughout the project. And this increases the likelihood of a configuration error.

It seems to me that it would be convenient to add the ability to configure controllers via DI registration of IdempotencyOptions and optionally use these settings in the IdempotentAttribute.

Feature: Configurable JSON serializers

We use NodaTime in our DTOs for things like Instant and LocalDate, which requires a custom JSON serializer NodaTime.Serialization.SystemTextJson.

Problem: JsonSerializationException on unsupported property types in DTO

IdempotentAPI does serialize the first successful request, but fails to deserialize the second request with Newtonsoft.Json.JsonSerializationException:

Newtonsoft.Json.JsonSerializationException: Error converting value 5/7/2024 5:28:54 AM to type 'NodaTime.Instant'.

Path '['Context.Result'].ResultValue.Jobs.$values[0].StatusUpdated', line 1, position 3312.\n ---> System.ArgumentException: Could not cast or convert from System.DateTime to NodaTime.Instant.

called from

IdempotentAPI.Helpers.Utils.DeSerialize<T>(byte[] compressedBytes)
IdempotentAPI.Core.Idempotency.ApplyPreIdempotency(ActionExecutingContext context)
IdempotentAPI.Filters.IdempotencyAttributeFilter.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)

It seems Newtonsoft JSON is used internally and without any support for configurable JSON serializers.

Proposal

1. Allow configuring JSON serializer externally, for example see how FusionCache adds its own serializer interface to support both Newtonsoft and System.Text.Json and external configuration
   a. FusionCacheNewtonsoftJsonSerializerExtensions.AddFusionCacheNewtonsoftJsonSerializer()
   b. FusionCacheSystemTextJsonSerializerExtensions.AddFusionCacheSystemTextJsonSerializer().
2. Without yet understanding the internal details, could IdempotentAPI not just reuse FusionCache to deserialize the response? It is already configured to support our DTOs.

Our setup

services.AddStackExchangeRedisCache(/*...*/);          
services.AddFusionCacheStackExchangeRedisBackplane(/*...*/);        
services.AddFusionCache().TryWithAutoSetup();        

// Configure FusionCache with System.Text.Json, NodaTime serializer and more.
services.AddFusionCacheSystemTextJsonSerializer(new JsonSerializerOptions
{
    DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
    Converters =
    {
        new JsonStringEnumConverter(),
    }
}.ConfigureForNodaTime(DateTimeZoneProviders.Tzdb));

// Reuse FusionCache in IdempotentAPI
services.AddIdempotentAPI();
services.AddIdempotentAPIUsingRegisteredFusionCache();

Request data hash always returns empty byte array

The following code in the GetRequestsDataHash method doesnt seem to work as expected. In our scenarios we keep getting the result always being a byte[0]. Which thus results in the unwanted behaviour of requests with different payloads and same idempotency key are treated as the same.

httpRequest.EnableBuffering();

if (httpRequest.Body.CanRead
    && httpRequest.Body.CanSeek)
{
    using MemoryStream memoryStream = new();
    httpRequest.Body.Position = 0;

    await httpRequest.Body.CopyToAsync(memoryStream).ConfigureAwait(false);
    requestsData.Add(memoryStream.ToArray());

Integers \ decimals are converted to string internally

So there is an issue where if an API returns an integer or a decimal internally it gets converted into a Dictionary<string, string> there by converting the result from (see below) integer \ decimal to string.

First response
{ "testInteger": 1613197826, "testDecimal": 16131978.26 }

Second response
{ "TestInteger": "1613197826", "TestDecimal": "16131978.26" }

Is there a configuration i'm missing somewhere or is it just the implementation internally?

Thanks!

Support MinimalAPI projects

Thanks @hartmark

We have a minimal API and we wanted to use this nugget, but it seems MVC action filters isn't supported so we have used EndpointFilters that was added in .NET 7....

For Minimal APIs, how to configure the idempotent options for a specific endpoint, not generally? (cache deadline, etc)

For Minimal APIs, how to configure the idempotent options for a specific endpoint, not generally? (cache deadline, etc)

In controllers, we can use attribute to configure, but how about Minimal APIs?

IdempotentAPI.DistributedAccessLock.MadelsonDistributedLock not found in NuGet repository

The same IdempotencyKey cannot return a custom error message

The first request can normally receive the returned custom format message

The second request cannot receive the custom format message

The second request with postman return 500
The first

The second

Http Status code of the response is not returned correctly from the cache

Hey again, I found what I believe to be a small bug :) here is the case, in my project, I have the need to send a NotAccepted response and since I couldn't find a method in the ControllerBase that returns a NotAccepted status code, I decided to make my own ObjectResult (let me know if .net has a NotAccepted response that I missed somehow), so I went ahead and made the following function that returns the response that I want

    public static ObjectResult NotAcceptedObjectResult(string message)
    {
        return new ObjectResult(new ErrorModel
        {
            Title = HttpStatusCode.NotAcceptable,
            StatusCode = StatusCodes.Status406NotAcceptable,
            Errors = new[]{
                    message
                },
        })
        {
            StatusCode = StatusCodes.Status406NotAcceptable,
        };
    }

the ErrorModel is just a record that has the properties that you can see, either way, the response that I get from the first call (not from the cache) is correct, but if I repeat the call with the same idempotency key (to get the result from the cache ) the response that I get back from cache is not correct, the response body is the same but the HTTP Status code of the call is 200 instead of 406.

I ended up making these changes

[DefaultStatusCode(DefaultStatusCode)]
public class NotAcceptableObjectResult : ObjectResult
{
    private const int DefaultStatusCode = StatusCodes.Status406NotAcceptable;
    public NotAcceptableResult([ActionResultObjectValue] object? value) : base(value)
    {
        StatusCode = DefaultStatusCode;
    }
}

which is the same as BadRequestObjectResult from .det, and now I return the following from my function

public ObjectResult NotAcceptableResult(string message)
    {
        return new NotAcceptableObjectResult(new ErrorModel
        {
            Title = HttpStatusCode.Conflict,
            StatusCode = StatusCodes.Status409Conflict,
            Errors = new[]{
                    message
                },
        });
    }

after these changes, the first response and the response I get from the cache, are both correct. I am not sure what is causing the bug but here is the bug and the workaround for it.

I hope you can fix it :) If you need more information about the bug let me know.

Get 409 status code when retry

I have 2 services, A and B and A calls B to create objects, and I use Polly to retry requests sent by A. The problem is when B crashed why processing request from A then Polly retries with same IdempotencyKey and it get 409 response instead re-run action method of B. Should we capture request/response only when request completed successfully? What should I do in this case?

Potential idempotency issue

I'm not sure, but if I successfully process the request (e.g., write the transaction to the database) and it fails before saving it in the cache, if the client makes the same request a second time, my app would execute the same transaction twice.

Specify the ExpireHour in Timestamp instead of int

Hi!

In our usecase scenario, the int for the ExpireHours is not flexible enough.

We would like to be able to specify 10 minutes. Or 30. So forcing this parameter to be minimum 1 hour is not ideal.

Would it be possible to change the ExpireHours to ExpireDelay and from int to TimeSpan?

If I do a PR for it, would it be accepted?

Not working in ASP .NET Core 5.0 - BinaryFormatter methods are obsolete

BinaryFormatter serialization methods are obsolete in ASP .NET Core 5.0. The BinaryFormatter is used in the Utils.cs class for Serializing and Deserializing the MemoryStream.

The recommended action is to stop using BinaryFormatter and consider using JsonSerializer or XmlSerializer.

What do you do with Content-Type caching?

By default .NET 6.0 sets Content-Type to "application/json; charset=utf-8" on response.

But, if we replay this Content-Type header into the request, then no outputformatter matches, because SystemTextJsonOutputFormatter only matches "application/json" without a charset parameter.

This requires either removing the charset when caching, or adding the charset to the outputformatter so it matches.

What do you usually do?

On a separate note, I've been modernizing a bit the code, are you receiving PRs?

Latest nuget breaks on minimal API having HttpResponse in its action

Reproduction case:

1. Add HttpResponse in minimal API action.
2. Run test Post_DifferentEndpoints_SameIdempotentKey_ShouldReturnFailure

Expected response2.StatusCode to be HttpStatusCode.BadRequest {value: 400} because Newtonsoft.Json.JsonSerializationException: Self referencing loop d...

Xunit.Sdk.XunitException
Expected response2.StatusCode to be HttpStatusCode.BadRequest {value: 400} because Newtonsoft.Json.JsonSerializationException: Self referencing loop detected for property 'ServiceProvider' with type 'Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope'. Path '[0].HttpContext.ServiceScopeFactory'.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CheckForCircularReference(JsonWriter writer, Object value, JsonProperty property, JsonContract contract, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues(JsonWriter writer, Object value, JsonContainerContract contract, JsonProperty member, JsonProperty property, JsonContract& memberContract, Object& memberValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value)
   at IdempotentAPI.Core.Idempotency.GenerateRequestsDataHashMinimalApi(IList`1 arguments, HttpRequest httpRequest) in C:\CCShare\IdempotentAPI\src\IdempotentAPI\Core\Idempotency.cs:line 477
   at IdempotentAPI.Core.Idempotency.PrepareMinimalApiIdempotency(HttpContext httpContext, IList`1 arguments) in C:\CCShare\IdempotentAPI\src\IdempotentAPI\Core\Idempotency.cs:line 175
   at IdempotentAPI.MinimalAPI.IdempotentAPIEndpointFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next) in C:\CCShare\IdempotentAPI\src\IdempotentAPI.MinimalAPI\IdempotentAPIEndpointFilter.cs:line 67
   at IdempotentAPI.MinimalAPI.IdempotentAPIEndpointFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next) in C:\CCShare\IdempotentAPI\src\IdempotentAPI.MinimalAPI\IdempotentAPIEndpointFilter.cs:line 140
   at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|109_0(ValueTask`1 valueTask, HttpContext httpContext)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

HEADERS
=======
IdempotencyKey: 7119af3f-7eb0-4ed7-979f-d55219ee0084
Host: localhost
, but found HttpStatusCode.InternalServerError {value: 500}.
   at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message)
   at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
   at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
   at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
   at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
   at FluentAssertions.Execution.AssertionScope.FailWith(String message, Object[] args)
   at FluentAssertions.Primitives.EnumAssertions`2.Be(TEnum expected, String because, Object[] becauseArgs)
   at IdempotentAPI.IntegrationTests.SingleApiTests.Post_DifferentEndpoints_SameIdempotentKey_ShouldReturnFailure(Int32 httpClientIndex) in C:\CCShare\IdempotentAPI\tests\IdempotentAPI.IntegrationTests\SingleApiTests.cs:line 232
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_0.<<InvokeTestMethodAsync>b__1>d.MoveNext() in /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 264
--- End of stack trace from previous location ---
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in /_/src/xunit.core/Sdk/ExceptionAggregator.cs:line 90



content1: 
{"idempotency":"00e4970a-a60a-49f5-be13-b45fae92f764","createdOn":"2024-01-22T18:23:14.3735711+01:00"}
content2: 
Newtonsoft.Json.JsonSerializationException: Self referencing loop detected for property 'ServiceProvider' with type 'Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope'. Path '[0].HttpContext.ServiceScopeFactory'.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CheckForCircularReference(JsonWriter writer, Object value, JsonProperty property, JsonContract contract, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues(JsonWriter writer, Object value, JsonContainerContract contract, JsonProperty member, JsonProperty property, JsonContract& memberContract, Object& memberValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value)
   at IdempotentAPI.Core.Idempotency.GenerateRequestsDataHashMinimalApi(IList`1 arguments, HttpRequest httpRequest) in C:\CCShare\IdempotentAPI\src\IdempotentAPI\Core\Idempotency.cs:line 477
   at IdempotentAPI.Core.Idempotency.PrepareMinimalApiIdempotency(HttpContext httpContext, IList`1 arguments) in C:\CCShare\IdempotentAPI\src\IdempotentAPI\Core\Idempotency.cs:line 175
   at IdempotentAPI.MinimalAPI.IdempotentAPIEndpointFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next) in C:\CCShare\IdempotentAPI\src\IdempotentAPI.MinimalAPI\IdempotentAPIEndpointFilter.cs:line 67
   at IdempotentAPI.MinimalAPI.IdempotentAPIEndpointFilter.InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next) in C:\CCShare\IdempotentAPI\src\IdempotentAPI.MinimalAPI\IdempotentAPIEndpointFilter.cs:line 140
   at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteValueTaskOfObject>g__ExecuteAwaited|109_0(ValueTask`1 valueTask, HttpContext httpContext)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

HEADERS
=======
IdempotencyKey: 7119af3f-7eb0-4ed7-979f-d55219ee0084
Host: localhost

IdempotentAPI with dotnet 6 and FusionCache not sending back cached data

Hello, so I am using IdempotentAPI with .net 6 and FusionCache with Redis, I went through your documentation and the fusionCache documentation I got everything up and running.

I can see that the has access and is storing my data on my Redis database, I set up Redis-commander and I can clearly see the data in Redis.

when I call my endpoint I get back data new data every time altho I am providing it with the same IdempotencyKey

info: IdempotentAPI.Core.Idempotency[0] IdempotencyFilterAttribute [Before Controller execution]: Request for POST: [/v1/roles/test]() received (50 bytes) IdempotentAPI.Core.Idempotency: Information: IdempotencyFilterAttribute [Before Controller execution]: Request for POST: [/v1/roles/test]() received (50 bytes) info: IdempotentAPI.Core.Idempotency[0] IdempotencyFilterAttribute [Before Controller]: Return result from idempotency cache (of type Microsoft.AspNetCore.Mvc.ObjectResult) IdempotentAPI.Core.Idempotency: Information: IdempotencyFilterAttribute [Before Controller]: Return result from idempotency cache (of type Microsoft.AspNetCore.Mvc.ObjectResult) info: IdempotentAPI.Core.Idempotency[0] IdempotencyFilterAttribute [Before Controller]: End IdempotentAPI.Core.Idempotency: Information: IdempotencyFilterAttribute [Before Controller]: End info: IdempotentAPI.Core.Idempotency[0] IdempotencyFilterAttribute [Before Controller execution]: Request for POST: [/v1/roles/test]() received (50 bytes) IdempotentAPI.Core.Idempotency: Information: IdempotencyFilterAttribute [Before Controller execution]: Request for POST: [/v1/roles/test]() received (50 bytes) info: IdempotentAPI.Core.Idempotency[0] IdempotencyFilterAttribute [Before Controller]: Return result from idempotency cache (of type Microsoft.AspNetCore.Mvc.ObjectResult) IdempotentAPI.Core.Idempotency: Information: IdempotencyFilterAttribute [Before Controller]: Return result from idempotency cache (of type Microsoft.AspNetCore.Mvc.ObjectResult) info: IdempotentAPI.Core.Idempotency[0] IdempotencyFilterAttribute [Before Controller]: End

let me know if I'm doing anything wrong, or if you require more logs/information.

High concurrency Error

High concurrency Request same key, Cause duplication

A possible issue on systems that are under heavy load with timeout errors in the distributed cache

In some cases, where the system is under a heavy load, and there are timeout errors in the distributed cache (e.g., Redis), the idempotency doesn't seem to work as expected. As a result, the controller method can be executed more than once for the same idempotency key, depending on the number of requests, server's load, etc.

How to disable common info logs?

How to disable common info logs?

My logs are being long and difficult to reach relevant messages.

Is there a way to disable just those info logs? keeping warning and error logs just when they happen

Failed to get correct hash value with same request body

I came across a bug in which for a request body big enough (30kb+), the cache wasn't able to get fetched properly because of a different hash string.

In Idempotency.cs -> GetRequestsDataHash(HttpRequest httpRequest) we just only have to add

httpRequest.EnableBuffering();

and then, make the copy asynchronously to the memoryStream.

The complete code:

if (httpRequest.ContentLength.HasValue
    && httpRequest.Body != null)
{
    // 2022-08-18: Enable buffering for bodies greater than 30k in size.
    //             Then, read the buffer asynchronously.
    httpRequest.EnableBuffering();

    if (httpRequest.Body.CanRead
        && httpRequest.Body.CanSeek)
    {
        using MemoryStream memoryStream = new();
        httpRequest.Body.Position = 0;
        var copy = httpRequest.Body.CopyToAsync(memoryStream);
        copy.Wait();
        requestsData.Add(memoryStream.ToArray());
    }
}

Make idempotency optional

Is there a way to make specific endpoints optionally idempontent, meaning, if the request contains an idempotency header we apply the mechanism, if not, we simple ignore it? From a brief lookup into the code, I think it is not possible right now:

IdempotentAPI/src/IdempotentAPI/Core/Idempotency.cs

Line 157 in b42a502

if (!TryGetIdempotencyKey(context.HttpContext.Request, out _idempotencyKey))

IdempotentAPI/src/IdempotentAPI/Core/Idempotency.cs

Lines 514 to 517 in b42a502

	if (!httpRequest.Headers.ContainsKey(_headerKeyName))
	{
	throw new ArgumentNullException(_headerKeyName, "The Idempotency header key is not found.");
	}

FastEndpoints support?

Have anyone tried to use this with FastEndpoints?: https://fast-endpoints.com/

I followed the setup/examples for the minimal api, and it is almost working. It is caching the response, but when i call the endpoint again, using the same key, i get no data back.

I tested it with the MinimalApi endpoints, and it works as it should.

Response Content-Length mismatch: too few bytes written (0 of 103)

In my project, I used the same IdempotencyKey of the Ocelot gateway to request the interface for the second time and the following exception was thrown.

2021-09-22 15:33:34.3495||INFO|IdempotentAPI.Filters.IdempotencyAttributeFilter|IdempotencyFilterAttribute [After Controller execution]: Response for 406 sent (103 bytes)
2021-09-22 15:33:34.3495||INFO|IdempotentAPI.Filters.IdempotencyAttributeFilter|IdempotencyFilterAttribute [After Controller execution]: SKIPPED (isPreIdempotencyApplied:True, isPreIdempotencyCacheReturned:True)
2021-09-22 15:33:34.3495|2|INFO|Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker|Executed action ST.ODP.Api.Controllers.TestController.InsertOrUpdate (ST.ODP.Api) in 27.9657ms
2021-09-22 15:33:34.3630|13|ERROR|Microsoft.AspNetCore.Server.Kestrel|Connection id "0HMBTJBFLB002", Request id "0HMBTJBFLB002:00000002": An unhandled exception was thrown by the application. System.InvalidOperationException: Response Content-Length mismatch: too few bytes written (0 of 103).

Support idempotency in a cluster environment

To support idempotency in a cluster environment (i.e., a group of multiple server instances), we could use Distributed Locks with Redis via the proposed Redlock algorithm.