Coder Social home page Coder Social logo

softwaremill / retry Goto Github PK

View Code? Open in Web Editor NEW
348.0 36.0 36.0 200 KB

because you should never give up, at least not on the first try

Home Page: https://softwaremill.com/open-source

License: MIT License

Scala 100.00%
scala retry future scalajs

retry's Introduction

retry

Build Status

don't give up

install

With sbt, add the following to your project's build.sbt

libraryDependencies += "com.softwaremill.retry" %% "retry" % "0.3.6"

usage

Applications fail. Network connections drop. Connections timeout. Bad things happen.

Failure to address this will cause other bad things to happen. Effort is the measurement of how hard you try.

You can give your application perseverance with retry.

Retry provides interfaces for common retry strategies that operate on Scala Futures.

Basic usage requires three things

  • an implicit execution context for executing futures
  • a definition of Success encode what "success" means for the type of your future
  • a block of code that results in a Scala Future.

Depending on your strategy for retrying a future you may also need an odelay.Timer for asynchronously scheduling followup attempts

Retry provides a set of defaults that provide retry.Success definitions for Option, Either, Try, and a partial function (defined with Success.definedAt(partialFunction)) out of the box.

import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future

retry.Backoff().apply(() => Future {
  // something that can "fail"
})

Defining success

Retry needs to know what success means in the context of your Future in order to know when to retry an operation.

It does this through a generic Success[-T](pred: T => Boolean) type class, where T matches the type your Future will resolve to.

Retry looks for this definition within implicit scope of the retry.

You may wish define an application-specific definition of what "success" means for your future. You can do so by specifying the following in scope of the retry.

implicit val perfectTen = Success[Int](_ == 10)

If your future completes with anything other than 10, it will be considered a failure and will be retried. Here's to you, tiger mom!

Success values may also be composed with and and or semantics

// will be considered a success when the preconditions of both successA and successB are met
val successC = successA.and(successB)

// will be considered a success when the predconditions of either successC or successD are met
val successE = successC.or(successD)

Sleep schedules

Rather than blocking a thread, retry attempts are scheduled using Timers. Your application may run within a platform that provides its own way for scheduling tasks. If an odelay.jdk.JdkTimer isn't what you're looking for, you may wish to use the odelay.Timer for netty, odelay.netty.Timer in the odelay-netty module or an odelay.twitter.TwitterTimer available in the odelay-twitter module.

See the odelay docs for defining your own timer. If none of these aren't what you're looking for, please open a pull request!

According to Policy

Retry logic is implemented in modules whose behavior vary but all produce a common interface: a retry.Policy.

trait Policy {
  def apply[T](promise: () => Future[T])
     (implicit success: Success[T],
      executor: ExecutionContext): Future[T]
}

Directly

The retry.Directly module defines interfaces for retrying a future directly after a failed attempt.

// retry 4 times
val future = retry.Directly(4) { () =>
  attempt
}

Pause

The retry.Pause module defines interfaces for retrying a future with a configurable pause in between attempts

// retry 3 times pausing 30 seconds in between attempts
val future = retry.Pause(3, 30.seconds).apply { () =>
  attempt
}

Backoff

The retry.Backoff modules defines interfaces for retrying a future with a configureable pause and exponential backoff factor.

// retry 4 times with a delay of 1 second which will be multipled
// by 2 on every attempt
val future = retry.Backoff(4, 1.second).apply { () =>
  attempt
}

When

All of the retry strategies above assume you are representing failure in your Future's result type. In cases where the result of your future is "exceptional". You can use the When module which takes a PartialFunction of Any to Policy.

val policy = retry.When {
  case NonFatal(e) => retry.Pause(3, 1.second)
}

policy(execptionalAttempt)

Note, The domain of the PartialFunction passed to When may cover both the exception thrown or the successful result of the future.

FailFast

retry.FailFast allows you to wrap any of the above policies and define which failures should immediately stop the retries.

The difference between retry.FailFast and retry.When with a partial function for Throwables is that retry.When passes the execution to another policy after the first retry, whereas retry.FailFast uses the inner policy logic for each retry. For instance, it allows using a policy that retries forever together with a fail fast logic on some irrecoverable exceptions.

val innerPolicy = retry.Backoff.forever
val policy = retry.FailFast(innerPolicy) {
  case e: FooException     => true
  case e: RuntimeException => isFatal(e.getCause)
}

policy(issueRequest)

When the provided partial function is not defined at a particular Throwable, the retry logic is defined by the wrapped policy.

Suggested library usage

Since all retry modules now produce a generic interface, a retry.Policy, if you wish to write clients of services you may wish to make define a Success for the type of that service and capture an configurable reference to a Policy so that clients may swap policies based on use case.

case class Client(retryPolicy: retry.Policy = retry.Directly()) {
  def request = retryPolicy(mkRequest)
}

val defaultClient = Client()

val customClient = defaultClient.copy(
  retryPolicy = retry.Backoff()
)

Credits

Originally created by Doug Tangren, maintained by SoftwareMill.

retry's People

Contributors

aappddeevv avatar adamw avatar antonkulaga avatar avapl avatar daddykotex avatar drob avatar eberle1080 avatar ldrygala avatar lolgab avatar mergify[bot] avatar mkubala avatar pask423 avatar softprops avatar tkawachi avatar zuchos avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

retry's Issues

Implicit success

Without an implicit success it says

Cannot find an implicit retry.Success for the given type of Future, either require one yourself or import retry.Success._

however even after importing retry.Success._ nothing changes, is there a simple way to say a future is successful in every case it doesn't fail (e.g. is handled with recover)?

Default implicit Success values can lead to unintentional retries

Hi,
Currently the Success object provides default Success implicits for various standard types:

implicit def either[A,B]: Success[Either[A,B]] =
  Success(_.isRight)
implicit def option[A]: Success[Option[A]] =
  Success(!_.isEmpty)
implicit def tried[A]: Success[Try[A]] =
  Success(_.isSuccess)

These are available even without explicit import. I found that this can lead to unintentional retries when forgetting to add your own implicit Success value.

For example, when querying a database I have a result type of Future[Option[???]], where an empty Option stands for a missing value in the database. In some use cases, this is a legitimate return value, and retrying the query won't change anything.

The correct behavior here would be to explicitly have:

implicit val always = Success.always

in scope.

But if you forget to add this line, everything still compiles and the option implicit from the Success object is picked up, leading to unwanted retries.

I would suggest removing the implicit modifier from defs above to avoid the unintentional retries.

Thanks

context free -> discount context

at the moment, this version is api compatible with dispatches embedded retry library. One feature request that I expect will pop soon is the ability for the function that returns the future to somehow "know" about the context of execution with respect to the retry and or have the ability to inform the retry attempt. I think something like this could be doable with an implicit environment exposing the retry information. Alternatively we can try to do this more explicitly by expose a folding interface

retry.Policy.fold(init)(params)((last) => next)

The semantics of this would be to make the function passed to retry act like a catamophism over the last failed attempt. Needs so more thought but just jotting this down to track ideas for now.

The backoff samples examples don't compile

This code doesn't compile on scala 2.11 library version 0.2.1

implicit val ec = ExecutionContext.global
    implicit val success: Success[String] = Success[String](_.isEmpty)
    implicit val format: Formats = DefaultFormats
    implicit val timer = odelay.jdk.JdkTimer.newTimer

    retry.Backoff(4, 1.second) {
      Future {
        ""
      }
    }

though this does

 val withBackOff: Policy = retry.Backoff(4, 1.second)

    withBackOff {
      Future {
        ""
      }
    }

is the readme just out of date?

Is it ok to store the Policy as a instance variable and invoke it multiple times with the closure. I think i actually prefer it this way as its more readable but none of the examples seem to work like this

One final bit of feedback is why do you require the user to wrap the operation in a future, couldnt you accept any T and wrap it in a futre internally

Directly policy - too much retries

Direct policy retries one more time than specified. There is test which shows that:

    it ("should deal with future failures") {
      implicit val success = Success.always
      val policy = retry.Directly(3)
      val counter = new AtomicInteger()
      val future = policy { () =>
        counter.incrementAndGet()
        Future.failed(new RuntimeException("always failing"))
      }
      Await.ready(future, Duration.Inf)
      assert(counter.get() === 4)
    }

In documentation there is

// retry 4 times
val future = retry.Directly(4) {
  attempt
}

which is inconsistent with test

Randomized exponential backoff

I was looking through the codebase and didn't find(maybe just search not thourough enough) an option to set a value which will be used to calculate an addition to delay using randomness. In retry.JitterBackoff I noticed that it uses delay itself for such puprose.
What I mean is following...
Here is the definition of Backoff's apply:
def apply(max: Int = 8, delay: FiniteDuration = Defaults.delay, base: Int = 2). So retry.Backoff(3, 1.second) means that 1st retry will be exactly after 1 second, 2nd retry after 2 seconds, and 3rd retry after 4 seconds.
What I want to achieve is next: 1st retry after (1 + random(0, N)) second, 2nd retry after 2*(1 + random(0, N)) + random(0, N) seconds, and 3rd retry after 2*( 2*(1 + random(0, N)) + random(0, N) ) + random(0, N) seconds.
There is a JitterBackoff but seems for aformentioned N it uses initial delay value.
Is such functionality supported?
PS. I'm neglacting cap here

Scala 3

Would you be open to a PR to support Scala 3 ?

I would happily work on it if you tell me it can be merged and published.

Syntactic sugar for retries.

Thanks for a great library!

Would you be open to a PR that allowed policies to take Future[T]s by-name? This would be in addition to the () => Future[T] option offered now.

This would let clients write:

retry.Directly(3) { attemptSomething() }

instead of

retry.Directly(3) { () => {
  attemptSomething()
}}

I believe I can do this with a small addition to the Policy trait and with no breaking changes.

How to do graceful shutdown?

How can I do graceful shutdown using this library? Say I have some code that is doing retry on some HTTP call. Then the application starts shutting down. Can I somehow register that this should be gracefully shut down?

Defaults.jitter() throws NullPointerException

Defaults.jitter throws NullPointerException when apply()

java.lang.NullPointerException:
at retry.Jitter$$anon$2.next(Jitter.scala:99)
at retry.Jitter.apply(Jitter.scala:14)
at retry.Jitter.apply$(Jitter.scala:10)
at retry.Jitter$$anon$2.apply(Jitter.scala:93)

Retry is making an extra attempt

I'm not sure why but I'm getting an extra call to my promise
Looks like the retry is occurring correctly since logged "Retrying request" matches my retry attempt.
The last attempt is the one that looks extraneous. Would you be able to advise why?

Therefore if I use Directly(1) I'm seeting my request sent 3 times instead of 2.

def sendWithRetry[U](promise: => Future[U], retryPolicy: retry.Policy): Future[U] = {
    val retryOnSpecifiedFailurePolicy = retry.When {
      case CustomException() =>
        logger.warn("Retrying request...")
        retryPolicy
    }

    retryOnSpecifiedFailurePolicy(promise)
  }

I'm calling the above like this
sendWithRetry(send(message), retry.Directly(1))
and I have this defined in class scope
implicit val always: retry.Success[Any] = retry.Success.always

Success condition can be a failure

I have a method that I want to retry that could result in multiple exceptions. Only one of them I want to retry on the other ones I want to consider as a success and throw that exception.
The Success condition for the Policy is only checked within a .map of the future being evaluated so a failed future will always result in a retry.
Is there a way to only retry on certain failures without having to wrap the content of the futures in a Try or Either?

Provide usage examples

This library looks great. Unfortunately lack of full examples is makes working with it quite difficult. Uncomplete code faragments are not helpfull.
For example:

import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future

retry.Backoff()(Future {
  // something that can "fail"
})

produce

Multiple markers at this line
    - type mismatch; found : scala.concurrent.Future[Int] required: 
     odelay.Timer
    - type mismatch; found : scala.concurrent.Future[Unit] required: 
     odelay.Timer

or

val future = retry.Directly(4) {
  attempt
}

Leads to type mismatch; found : X required: () => scala.concurrent.Future[?]

Could you provide some simple but full examples of usage? With:

  • definition (or import) of Success,
  • some dummy operation in retry
  • performing some operation when result is complete (or failed)
    ?

remove timer modules once odelay is published

the timer modules really only private timer implementations and have little to nothing to do with retry behavior but rather delayed execution. After odelay is published this library should depend on that instead.

No Implicits found for parameter success

Hi, I am trying to implement retry with backoff for a function which return postgres connection but i am keep getting error not found: type Success in Intellij IDE. Please see the code below and let me know what i am missing to make this code compile. Thanks

import retry.Success
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import scala.concurrent.Future

 def getPostgresConnectionRetry(jdbcUrl: String, connectionProperties: Properties, processName: String, customerId: String): Future[Connection] = {
    implicit val nonNullResponse: Connection = Success[Connection](conn => conn != null)
    retry.Backoff(NUM_RETRIES, INITIAL_RETRY_AFTER).apply(() =>
      Future[Connection] {
        try {
          DriverManager.getConnection(jdbcUrl, connectionProperties)
        }
        catch {
          case e: Throwable =>
            LOG.error(s"Error while connecting to postgres for customerId; $customerId, processName: $processName. Error: ${e.getMessage}")
            throw e
        }
      }
    )
  }

The Connection is of type java.sql.connection.

Thanks

Successes should be composable

like so

val a = new Success[T](...)
val b = new Success[T](...)
a.or(b) // == new Success[T](v => a.predicate(v) || b.predicate(v))
a.or(const) // == new Success[T](a.predicate(_) || const)
a.and(b) // == new Success[T](v => a.predicate(v) && b.predicate(v))

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.