roma-glushko / hyx

🧘‍♀️ Lightweight fault tolerant primitives for your modern asyncio Python microservices

🧘‍♂️️Lightweight fault tolerance primitives for your resilient and modern Python microservices

---

Hyx (/ˈhʌɪx/) is a set of well-known stability patterns that are commonly needed when you build microservice-based applications. Hyx is meant to be Hystrix (Java), resilience4j (Java) or Polly (C#) but for the Python world.

Key Features

• Implements five commonly used resiliency patterns with various configurations based on advice and experience of industry leaders (e.g. AWS, Google, Netflix)
• Idiomatic Pythonic implementation based on decorators and context managers
• AsyncIO Native Implementation
• Lightweight. Readable Codebase. High Test Coverage

Requirements

• Python 3.9+
• AsyncIO-powered applications (no sync support?)

Installation

Hyx can be installed from PyPi:

pip install hyx

# or via poetry
poetry add hyx

Component Map

Component	Problem	Solution	Implemented?
🔁 Retry	The failures happen sometimes, but they self-recover after a short time	Automatically retry operation on temporary failures	✅
💾 Cache
⚡️ Circuit Breaker	When downstream microservices have got overloaded, sending even more load can make the situation only worse.	Stop doing requests to your failing microservice temporarily if amount of errors exceeded expected thresholds. Then see if the given time helped the microservice to recover	✅
⏱ Timeout	Sometimes operations may take too much time. We cannot wait that long or after that time the success is unlikely	Bound waiting to a reasonable amount of time	✅
🚰 Bulkhead	If executed without control, some code can take too much resources and put down the whole application (and upstream services) or cause slowness of other places of the application	Fix the amount of calls to the code, queue other calls and fail calls that goes beyond your capacity	✅
🏃‍♂️ Rate Limiter	The microservice can be requested with any rate even one that can put it down if happens by accident	Limit the rate your system can be accessed by	✅
🤝 Fallback	Nothing can guarantee you that your dependencies will work. What would you do when it's failing?	Degrade gracefully by defining some default values or placeholders if your dependencies are down	✅

Inspired by Polly's Resiliency Policies

Acknowledgements

• resilience4j/resilience4j
• Netflix/Hystrix
• slok/goresilience
• App-vNext/Polly
• Diplomatiq/resily

hyx's Issues

📝 Document the rate limiting components

✨ Implement error-rate-based circuit breaker

Hyx is providing a circuit breaker based on the consecutive error number right now.

We need to expand that and provide a circuit breaker based on rolling window stats (e.g. percentage of failed requests).

Let's think about providing the rolling window stats as a separate low-level component, so other project could use it to implement very custom rate limiters or similar things.

References:

• https://github.com/App-vNext/Polly/wiki/Advanced-Circuit-Breaker

✨Implement fallback component

Document Hyx API design principles

✨ Implement a basic rate limiter

✨ Init Pixi

Pixi is an example of a microservice system that used asyncio modern Python to power its services. Microservices are running in Kubernetes cluster with chaos testing tools enabled.

It uses Hyx to improve resiliency of the system and test Hyx under the near-real-world scenarios.

To make this task simpler, I will get my inspiration for Pixi's functionality from Google's https://github.com/GoogleCloudPlatform/microservices-demo

🔨 Introduce a common error class that will indicate that the service is overload

✨ Revisit jitter usage and types of jitters in retry component

Timeout Component

📝 Integrate with analytics and feedback system

📝 Document bulkhead component

🔊 Implement listeners API for rate limiters

Introduce circuit breaker manager

As brought up in #85, I would be great to have a single component that manages seamlessly all breakers in the service.

The manager could be defined somewhere as a singleton and passed around the places (via dependency injection, for example):

from typing import Any

import httpx


class CurrencyRate:
    ...


class CurrencyRates:
    def __init__(self, breaker_manager: CircuitBreakerManager) -> None:
        self._breaker_manager = breaker_manager

    async def get_rates(self) -> list[CurrencyRate]:
        with self.breaker_manager("currency.rates", exceptions=(httpx.NetworkError,)):  # manager creates/retrieves and activates breaker
            async with httpx.AsyncClient() as client:
                response = await client.get("https://api.github.com/events")

                return self._map_rates(response.json())

    def _map_rates(self, data: list[dict[str, Any]]) -> list[CurrencyRate]:
        ...

📝 Add introduction to resilience components

✨ Integrate with OpenTelemetry

Collects metrics from component's lifecycle through OpenTelemetry

✨ Implement hook/event system

Hyx needs to provide various hooks to let user code react to changes in the Hyx's components.

The basic usecase will be metric reporting around component lifecycle.

🚨 Fix MyPy errors in the current codebase

How do I use this?

Coming from resilience4j, I have no idea how to use this nice library. Am I missing the docs?

📝 Document the timeout component

✨ Implement service-level retry budgets

The current retry component is based on the counting retries that happens on the scope of one request.

It's useful to watch for a general amount of requests/retries we have done so far trying to communicate with given dependant mservice.

Retry Pattern

Implement the retry pattern:

• as a decorator that extends existing functions
• accept list of exceptions to retry on
• accept wait strategies
• accept jitter strategies

✨ Implement LRU cache with TTL

📝 Document the fallback component

📝 Document circuit breakers

🔨 Integrate with Google Search Console

👷‍♀️ Configure CI based on Github Actions

✨ Implement adaptive concurrency limiting

Good resources to follow during the implementation:

• https://netflixtechblog.medium.com/performance-under-load-3e6fa9a60581
• https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/adaptive_concurrency_filter
• https://vector.dev/blog/adaptive-request-concurrency/
• https://itnext.io/5-patterns-to-make-your-microservice-fault-tolerant-f3a1c73547b3

Introduce hedging

Sometimes it's reasonable to dispatch a few requests at the same time to organize a race between them for the sake of improving latency (while having some resiliency added). This is call hedging.

Usecases:

• You have several LLM providers and you want to get the next agent "move" as soon as possible to meet real-time latency requirements

Reference:

• https://github.com/resilience4j/resilience4j/tree/master/resilience4j-hedge

🔊 Implement listeners API for the fallback

🐛 Docs: No module named poetry.core.toml

Documentation builds are failing with the following error:

📝 Configure Hyx docs

🔊 Implement listeners API for circuit breakers

📝 Document Retry Component

🔊 Implement listeners API for bulkheads

🔧 add __slots__ to Hyx's components

Implement Circuit Breaker

✨ Implement a CoDel-based bulkhead

Implement CoDel-based bulkhead

✨ Implement efficient LFU Cache with TTL

Implement Bulkhead based on semaphores

Cover retry jitters by tests

Create a logo for the project

👷‍♀️ Implement package build and push on tag/release

Expose ToketBucket as a standalone component

The hyx.ratelimit package provides TokenBucketLimiter component:

hyx/hyx/ratelimit/managers.py

Line 12 in 365a280

class TokenBucketLimiter(RateLimiter):

Its API is rather simplistic and hard to reuse if you want to pull that bucket logic to reuse somewhere (for example, to implement a custom bucket retry).

Let's refactor the TokenBucket logic out of the ratelimiter and expose it outside of Hyx, so other projects could import it.

Document @bucket_retry() decorator

✨ Add LIFO-based bulkheads

In some situations, requests that come some time ago is less relevant to ones that has just arrived. Hyx needs to provide a way to select serving strategy

✨ Implement sliding window based rate limiting

Resources:

• https://dev.to/satrobit/rate-limiting-using-the-sliding-window-algorithm-5fjn

🔊 Complete retry listener API

👷 Run linting on each PR

✨ Implement leaky bucket rate limiter

Hyx is providing a token bucket-based rate limiting at this moment.

Let's implement a leaky bucket-based rate limiting, too.

We should provide LeakyBucket as a standalone component, so other projects could import it to build their very custom logic as well as use it in the leaky bucket limiter implementation.

✨ Design how to bucket rate limiting by users/requesters/etc

The current Hyx's rate limiter APIs were inspired by resilience4j/Polly. So each instance of rate limiter handles one rate limit "shard" (e.g. basically, the level on which you apply rate limiting. It could per user, per user/request route, etc). As you could imagine, this is SUPER low level implementation which is almost certainly insufficient for the regular usecases 😌

What I have found people really needed is an API that would be integrated easily into their API framework of their choice (e.g. Flask, Starlette, FastAPI) and allowed to have many shards right away. With such implementation in place, we will close the most common and basic need for rate limiting.

Now, to be able to store a big number of shards, we will need to explore a use of probabilistic data structures, so we could "compress" the exact shard's limit quotas for the sake of a low memory footprint.

Definition of Done

• Come up with a new rate limiter API that would incorporate a need for supporting many shards. Feel free to review the existing framework-specific middleware (like https://pypi.org/project/fastapi-limiter/) to account for the step 2 of this initiative which is providing a middleware for such frameworks based on framework-agnostic rate limiters that Hyx provides
• Try to implement that new API for the tokenbucket rate limiter using probabilistic data structures to hold its shard state

References

• https://github.com/hajimes/mmh3
• https://dev.to/satrobit/rate-limiting-in-ipv6-era-using-probabilistic-data-structures-15on
• https://florian.github.io/count-min-sketch/
• https://github.com/barrust/pyprobables/blob/master/probables/hashes.py