failsafe-lib / failsafe

Fault tolerance and resilience patterns for the JVM

Failsafe

Failsafe is a lightweight, zero-dependency library for handling failures in Java 8+, with a concise API for handling everyday use cases and the flexibility to handle everything else. It works by wrapping executable logic with one or more resilience policies, which can be combined and composed as needed.

Policies include Retry, CircuitBreaker, RateLimiter, Timeout, Bulkhead, and Fallback. Additional modules include OkHttp and Retrofit.

Usage

Visit failsafe.dev for usage info, docs, and additional resources.

Contributing

Check out the contributing guidelines.

License

Copyright Jonathan Halterman and friends. Released under the Apache 2.0 license.

failsafe's Issues

Add fallback execution support

Add support for fallback actions which are executed when a failure occurs and cannot be retried:

Failsafe...recoverWith(executor).get(executor);

Or possibly:

Failsafe...tryRun(foo).elseRun(bar);
Failsafe...tryRun(foo, executor).elseRun(bar, executor);
Failsafe...tryGet(this::connect).elseGet(this:reconnect);

Consider CircuitBreaker.withMaxConcurrency

This could place an easy cap on concurrent executions/requests through the circuit.

May need to use something like ExecutionRejectedException with max concurrency is reached.

Failsafe vs Hystrix

I'd like to have a small comparison (pros/cons) between your library and others such as Hystrix.
It might help people decide which library best suits their needs.

Listeners not working?

First of all, great framework! Lightweight, super simple to use, yet feature-rich. Enjoying it so far!

Sadly I can't get Listeners to work. Trying the onRetry and onFailedAttempt Listeners with more than one argument, with the retryIf condition failing. Some code:

    private static final RetryPolicy RETRY_POLICY = new RetryPolicy()
            .retryIf((CachedResponse response) -> response == null || response.getStatusFamily() != Family.SUCCESSFUL)
            .retryOn(Exception.class).withDelay(3, TimeUnit.SECONDS).withMaxRetries(3);
// ...
CachedResponse response = Failsafe.<Response> with(RETRY_POLICY)
                .onFailedAttempt((failedResponse, exception, context) -> {
                    System.err.println("response: " + failedResponse);
                    System.err.println("exception: " + exception);
                    }).get(() -> new CachedResponse(target.request().get()));

Some scenarios that I expect to work vs. the ones that actually work:

• Response fails (i.e. retryIf condition fails) with 3 arguments onFailedAttempt / onRetry
• Response fails (i.e. retryIf condition fails) with 2 arguments onFailedAttempt / onRetry
• Response fails (i.e. retryIf condition fails) with 1 argument onFailedAttempt / onRetry (but argument failure is always null, which is correct but not helpful)
• Failure (i.e. retryOn condition fails) with 3 arguments onFailedAttempt / onRetry
• Failure (i.e. retryOn condition fails) with 2 arguments onFailedAttempt / onRetry
• Failure (i.e. retryOn condition fails) with 1 argument onFailedAttempt / onRetry

Let me know if you can reproduce it or if I'm doing something wrong on my side. Thanks!

Offer Spring endpoint for Circuit Breakers

Hi @jhalterman,

as discussed yesterday already, I'd like to create an registry for the circuit breakers I created in order to gather information in which state they're.

These registry should then be exposed by an spring endpoint for monitoring purposes.

I hope you got that idea?

I am now thinking of to opensourcing it. Would you like to add it as a feature in this project, or should I setup a sepperate repo for it?

All the best

Reactor example

Add a reactor example

https://github.com/reactor/reactor

Fallback: Exceptions are wrapped in FailsafeException

Hi, my usecase is that I want to throw my own custom exception as a fallback:

Failsafe.with(retryPolicy)
   .withFallback(new CheckedConsumer<Throwable>() {
       @Override
       public void accept(Throwable throwable) throws Exception {
           throw new MyCustomException(throwable);
       }
   })
   .get(...);

The issue is that whenever I use any of the checked functional interfaces, my custom exception gets wrapped in a FailsafeException. Only when I use a non-checked functional interface like Callable my custom exception is thrown unwrapped. The problem with using Callable is that I no longer get access to the failure (Throwable), and I want to wrap the failure in my custom exception, like in my example code above.

Thanks in advance, and thanks for a great library.

Call new method on retry

I am sending bulk e-mails via Spring's JavaMailSenderImpl class. When there is a failure a MailSendException is thrown containing the failed messages. I would like to retry only those messages that are found within the exception and in a sense recursively keep doing so until finally no exception is thrown or the retry policy has failed. Do you think this is possible with this framework?

New Release

hello:

im not sure of your release cycle, but a lot of your great Listener goodness is not in the latest release (0.4.0?). is the a timeline to publish?

thanks!

chris

Make RetryPolicy non final

I'd like to inherit from RetryPolicy to define default behaviours for certain usecases.

For example I'd like to create an ApacheHttpRetryPolicy which covers alreday the retry on the certain Apache exceptions.

This is not possible when the class is final and actually I don't really see a reason why it's done like that?

Update RecurrentFuture to add retry related eventing

Similar to Lyra's eventing

FailsafeFuture#complete is always calling fallback method

FailsafeFuture#complete:127 this.result = fallback.apply(result, failure);

I think it's not right to always call fallback, even when no error occurred during processing. In that case failure == null, so workaround with checking if failure == null and then doing nothing in fallback is possible. But if feels weird.

What is your opinion on this?

Improve standalone CircuitBreaker usage APIs

Expose before method for standalone usage. Currently execution for standalone usage is somewhat broken, where increments never happen but decrements do.

withMaxDurationPerTry

Hello!

Any chances of adding 'withMaxDurationPerTry' functionality? Currently, 'withMaxDuration' limits the duration of all attempts combined and not of the separate attempt.

Thanks.

Add random retry delay

Add a random retry delay (within some bounded range). This could avoid stampedes such as when multiple clients are disconnected at the same time.

https://news.ycombinator.com/item?id=12152722

RetryPolicy thread safety

Is it safe to use RetryPolicy from multiple threads?

I can see it doesn't set it's own members but it does add members to (array)lists of predicates. This is potentially not thread-safe and can break when using the same RetryPolicy object from multiple threads.

Is there a standard to do this? I can copy the object for now when I modify it on specific threads, but perhaps making it thread safe is possible on your end.

Standalone CircuitBreaker doesn't respect withDelay configuration

There is a note about standalone usage of CircuitBreaker in docs, but in this usage withDelay(long delay, TimeUnit timeUnit) configuration is not respected, which is not obvious. Would be great to support delay inside CircuitBreaker, but at least add a javadoc to withDelay that this method doesn't make any sense for standalone usage.

Consider time based thresholds for CircuitBreaker

For example, where .5 executions fail over 1 minute period. See Polly's advanced circuit breaker as a reference:

https://github.com/App-vNext/Polly

Either requires windowing or inexact stats, neither of which are ideal.

FailsafeFuture.cancel should call completion handlers.

Similar to what CompletableFuture.cancel does. FailsafeFuture.get should also throw CancellationException if the future was cancelled.

Add support for time threshold in retrypolicy

Hi there,

Let's assume I have a RetryPolicy with the a maxduration of 10 seconds and max retries of 3.

Furthermore the requst is in the third retry and already spent 9 seconds with retrying.
It would be great if the third retry wouldn't happen anymore since it will probably take more time then maxDuration and therefore will be interrupted.

In this scenario the application will end up in an undefined state, since the request to the other party might be succesfull but the app will never receive the response since maxDuration will take place.

I hope you got what I mean.

Question: CircularBitSet.java#L33

Why occupiedBits - size and not the other way around?

Not ABle to add Gradle dependency

I am not able to add gradle dependecy.
Added the following line to the build.gradle file
compile 'net.jodah:failsafe:0.9.3'

The Dependency that got included is joda-time:2.9.4

Consider adding support for matching listeners to particular failures

Ex:

onFailedAttempt(ConnectException.class, e -> log.error("Failed to connect", e));

Failure instances of ConnectException will trigger the listeners, else the listener is not triggered. Currently users can do:

onFailedAttempt(e -> {
  if (e instanceof ConnectException)
    log.error("Failed to connect", e);
});

Not sure if this is worth all of the new overloaded methods it will introduce...

OSGi support

In the next release you can be added support for OSGi?

Thank you,
Giuseppe

Add onFailedAttempt(Exception) listeners

...for cases where the user doesn't care about the result, only the exception. Ex:

onFailedAttempt(Exception);
onFailure(Exception);
onRetry(Exception);

Support failure cause

Need a way for event handlers to distinguish between a failure caused by retries exceeded and a failure caused by an exception that did not match the retry policy.

ExecutionContext.isRetryPolicyExceeded
ExecutionContext.isCircuitBreakerOpen

These could be added to a subclass of ExecutionContext that is specific to each handler, such as FailureEvent.

This approach allows the onFailure handler to be used for different types of failures and situations. A failure is a failure - we just need to know the state of our failure handlers when the failure occurred.

Add additional fallback handlers

Runnable, BiConsumer, Consumer, etc. Strive for parity with get/run.

whenSuccess is getting called twice when using AsyncListeners.

Hi,

I'm running 0.5.0 and want to have a retry listener. From what I understand when calling Recurrent.get in async mode you can only inject AsyncListeners and no Listeners. I want to be able to log on retries and do magic on success. The following code calls whenSuccess twice:

    final AsyncListeners<Object> pullerAsyncListeners = new AsyncListeners<Object>()
        .whenRetryAsync((result, failure, stats) -> {
          log.error("Attempt #{} to start failed: {}",
                    stats.getAttemptCount(),
                    failure.getMessage());
        })

        Recurrent.get(() -> pull(sub), retryPolicy, retryExecutor, pullerAsyncListeners).whenSuccess(p -> log.info("Puller started");

Same if I add the whenSuccess block on the AsyncListener and cast it to an AsyncListener again (since whenSuccess returns a Listener).

This code works fine however:

Recurrent.get(() -> pull(sub), retryPolicy, retryExecutor).whenSuccess(p -> log.info("Puller started");

Java 7 port

The library is quite cool, but given the lack of hard dependencies on Java 8 APIs the port could be quite simple.

This would help adopt it in Android environments, where we're currently pushing hard for RxJava. This could be a great stepstone.

Execution.canRetryOn

Consider deprecating Execution.canRetryOn. We should be able to just perform retry attempts if isComplete is false.

Create CircuitBreaker and check for allowsExecution

CircuitBreaker breaker = new CircuitBreaker();
breaker.allowsExecution();

This code throws a NPE, as the state is still null.

CircuitBreaker broken for 0.9.3

After updated to 0.9.3, some of my tests failed:

FAIL in (test-circuit-breaker) (core_test.clj:185)
circuit open
expected: (instance? CircuitBreakerOpenException e)
  actual: java.lang.NullPointerException

lein test :only diehard.core-test/test-circuit-breaker

FAIL in (test-circuit-breaker) (core_test.clj:185)
circuit open
expected: (instance? CircuitBreakerOpenException e)
  actual: java.lang.IllegalStateException

lein test :only diehard.core-test/test-circuit-breaker

FAIL in (test-circuit-breaker) (core_test.clj:185)
circuit open
expected: (instance? CircuitBreakerOpenException e)
  actual: java.lang.IllegalStateException

Those situation expected to throw CircuitBreakerOpenException now doesn't work as expected.

I assume there shouldn't be API break change in a patch release. So this might be some regression issue?

Separate AsyncExecution and Execution impls

AsyncExecution doesn't need to inherit all of the methods in Execution, and should be based on a separate common base class.

ExecutionTest#shouldSupportMaxDuration() breaks because of indeterministic scheduling

The test ExecutionTest#shouldSupportMaxDuration() breaks randomly on windows with oracle jdk 1.8.0_91
Reason seems to be the Thread.sleep is exactly as long as the max duration but it actually does not sleep for that long time.

Fails

public void shouldSupportMaxDuration() throws Exception {
    Execution exec = new Execution(new RetryPolicy().withMaxDuration(100, TimeUnit.MILLISECONDS));
    assertTrue(exec.canRetryOn(e));
    assertTrue(exec.canRetryOn(e));
    Thread.sleep(100); // <--
    assertFalse(exec.canRetryOn(e));
    assertTrue(exec.isComplete());
  }

Succeeds (although still a little ugly)

public void shouldSupportMaxDuration() throws Exception {
    Execution exec = new Execution(new RetryPolicy().withMaxDuration(100, TimeUnit.MILLISECONDS));
    assertTrue(exec.canRetryOn(e));
    assertTrue(exec.canRetryOn(e));
    Thread.sleep(105); // <--
    assertFalse(exec.canRetryOn(e));
    assertTrue(exec.isComplete());
  }

I suggest one of the following fixes:

1. increase the Thread.sleep time, not a very elegant fix
2. abstract the time source by passing a clock in the constructor and use this to get current time, you can then control the time in your tests
3. use powermock or something to change the behaviour of the time source which is used. This does not require to modify the production code but makes the test code a little uglier.

Which one of the 3 suggestions do you prefer? or do you have another one?

Add support for failing on particular results/failures

failOn, failWhen etc.

Add possibility for ErrorHandler

It would be nice to have the possibility to add an ErrorHandler to a Retrypolicy in order to avoid that a 'FailsafeException' is thrown. would be great to throw your self defined exceptions there.

Our current workaround is to wrap the Failsafe calls in try catch blocks and add the handling there, which produces redundant code snippets and makes it kind of ugly.

Add circuit breaker support

Add support for breaking a circuit.

Need related event listeners.
Need support for a fallback executable when the circuit is open.
Track failure rates for different exception/failure types types.

RetryPolicy.fallbackOn, fallbackWhen

Recurrent.with(fallback)

Order retries / circuit breaker in the order they're configured

If retry policy is configured first, retry first then break, else visa versa.

Howto cancel async retries

Can you add a short example howto cancel async retries.

I was surprised that canceling the returned CompletableFuture does not cancel the retries.
Am I using the library wrong or is this intended?
How do I cancel the retry in the following example, do I need to add a canceled flag and check this in the future method?

scheduler = Executors.newScheduledThreadPool(1);
executor = Executors.newCachedThreadPool();

CompletableFuture<Proxy> proxyFuture = Failsafe.with(proxyRetryPolicy)
                       .with(scheduler)
                       .future(asyncExecution -> {
                           return CompletableFuture.supplyAsync(proxySupplier, executor);
                       });

// does not cancel the retry .. keeps retrying
proxyFuture.cancel(false);
proxyFuture.cancel(true);

Asynchronous retries 'whenFailure' broken in 0.4?

Hi,
when the retry policy is exceeded, the failure function is not called. Can you make a new release of the master brunch? - (perhaps the bug is corrected.)

retryPolicy = new RetryPolicy()
         .withDelay(100, TimeUnit.MILLISECONDS)
         .withMaxDuration(2, TimeUnit.SECONDS)
         .withMaxRetries(3);

RecurrentFuture<?> run = Recurrent.get(() -> {
  doAnything();
  return <<>>;
}, retryPolicy, executer);
run.whenFailure ( error -> System.out.println("timeout"));

Stability status.

Hi, thanks for this awesome lib! The API looks quite interesting.

We'd like to use it in production. We have a couple of questions here: How stable is Failsafe? Is it used in production already?

0.4.1 release?

Hi,
Looks like the examples in README are written for 0.4.1 as they show, among other things, the use of Listeners. When do you think 0.4.1 becomes publicly available, e.g. on maven central?
Thanks!

IllegalStateException: Cannot record result for open circuit

I'm using circuit breaker in a high concurrent environment, and I'm getting this IllegalStateException:

java.lang.IllegalStateException: Cannot record result for open circuit
  at net.jodah.failsafe.internal.OpenState.recordFailure(OpenState.java:34)
  at net.jodah.failsafe.CircuitBreaker.recordFailure(CircuitBreaker.java:436)
  at net.jodah.failsafe.AbstractExecution.complete(AbstractExecution.java:87)
  at net.jodah.failsafe.SyncFailsafe.call(SyncFailsafe.java:170)
  at net.jodah.failsafe.SyncFailsafe.get(SyncFailsafe.java:42)

Bind listener types to result types

...by creating UntypedSyncFailsafe and TypedSyncFailsafe (same for async) classes where the untyped variant infers a return type:

<T> T get(callable)

and the typed variant uses a return type received from the with(Listeners<T>) call:

T get(callable)

This will mean two different SyncFailsafe, AsyncFailsafe, and possible listener implementations as well, and so far it hasn't seemed worth the added complexity....

Allow listeners to be configured directly on SyncFailsafe or AsyncFailsafe instances

Allow cummulative retryOn, retryIf, etc. calls in RetryPolicy

Allow cummulative calls to methods in RetryPolicy:

retryIf(predicate1).retryIf(predicate2);
retryWhen(null).retryWhen(-1);

This would logically OR the supplied values.

Port failsafe to Python

I find your retry design through policies pleasant and have some interest in having the same library in Python.
Just want to check if this work is not already started?

Add onAbort event listeners

Add elapsed time tracking

Should return total elapsed time since started and start time.

Simplify Listeners API

Currently the listeners API requires a separate object instantiation for all cases:

Failsafe.with(retryPolicy)
  .with(new Listeners()
    .onFailure(e -> log.error(e))
    .onSuccess(cxn -> log.info("created connection: {}", cxn);
  .get(this::connect);

The idea behind this was that a set of Listeners could be reused and shared, but I'm not sure there's much of a use case for that since Listeners tend to be execution specific. That said, I think it makes sense to streamline this by allowing listener registration directly at the top level API:

Failsafe.with(retryPolicy)
  .onFailure(e -> log.error(e))
  .onSuccess(cxn -> log.info("created connection: {}", cxn);
  .run(this::connect);

In this setup, individual listeners could still be reused. This change will also remove the listener methods from FailsafeFuture since they can be registered instead via the Failsafe API.

A Listeners class will still be retained for Java 6 / 7 compatibility via the with(Listeners) method, but will not be used for listener registration, only overrides:

Failsafe.with(retryPolicy)
  .with(new Listeners<Connection>() {
    public void onFailure(Throwable failure) {
      log.error(e);
    }
  })
  .get(this::connect);