A kotlin library for carefully refactoring critical paths in your application.
This library is inspired by the ruby gem scientist.
Let's pretend you're changing the way you handle permissions in a large web app. Tests can help guide your refactoring, but you really want to compare the current and refactored behaviors under load.
fun isAllowed(user: User): Boolean = scientist<Boolean, Unit>() conduct {
experiment { "widget-permissions" }
control { user.isAllowedOldWay() }
candidate { user.isAllowedNewWay() }
}
Wrap a control
lambda around the code's original behavior, and wrap candidate
around the new behavior. When conducting the experiment conduct
will always return whatever the control
lambda returns, but it does a bunch of stuff behind the scenes:
- It decides whether or not to run the
candidate
lambda, - Randomizes the order in which
control
andcandidate
lambdas are run, - Measures the durations of all behaviors,
- Compares the result of
candidate
to the result ofcontrol
, - Swallows (but records) any exceptions thrown in the
candidate
lambda, and - Publishes all this information.
Compared to other scientist libraries this library separates the concepts of a scientist and the experiment. Which in turn gives you more freedom and flexibility to compose and reuse scientists and experiments (especially with dependency injection frameworks).
val scientist = scientist<Boolean, Unit>()
val experiment = experiment<Boolean, Unit() {
control { true }
}
val result = scientist conduct experiment
The scientist is responsible for setting up the environment of an experiment and conducting it.
The examples above will run, but they're not really doing anything. The candidate
lambdas run every time and none of the results get published. Add a publisher to control the result reporting:
val scientist = scientist<Boolean, Unit> {
publisher { result -> logger.info(result) }
}
You can also extend the publisher typealias
which then can be used as a parameter of the publisher lambda:
val logger = loggerFactory.call()
class LoggingPublisher(val logger: Logger) : Publisher<Boolean, Unit> {
override fun invoke(result: Result<Boolean, Unit>) {
logger.info(result)
}
}
val loggingPublisher = LoggingPublisher(logger)
val scientist = scientist<Boolean, Unit> {
publisher(loggingPublisher)
}
Scientist compares if control and candidate values have matched by using ==
. To override this behavior, use match
to define how to compare observed values instead:
val scientist = scientist<Boolean, Unit> {
match { candidate, control -> candidate != control }
}
candidate
and control
are both of type Outcome
(a sealed class) which either can be Success
or Failure
. As an example take a look at the default implementation:
class DefaultMatcher<in T> : Matcher<T> {
override fun invoke(candidate: Outcome<T>, control: Outcome<T>): Boolean = when(candidate) {
is Success -> when(control) {
is Success -> candidate.value == control.value
is Failure -> false
}
is Failure -> when(control) {
is Success -> false
is Failure -> candidate.errorMessage == control.errorMessage
}
}
}
A Success
outcome contains the value that has been evaluated. A Failure
outcome contains the exception that was caught while evaluating a control
or candidate
statement.
To provide additional data to the scientist Result
and Experiments
you can use the context
lambda to add a context provider:
val scientist = scientist<Boolean, Map<String, Boolean>> {
context { mapOf("yes" to true, "no" to false) }
}
The context is evaluated lazily and is exposed to the publishable Result
by evaluating val context = result.contextProvider()
and in the experiments conductibleIf
lambda that will be described further down the page.
During the early stages of an experiment, it's possible that some of your code will always generate a mismatch for reasons you know and understand but haven't yet fixed. Instead of these known cases always showing up as mismatches in your metrics or analysis, you can tell the scientist whether or not to ignore a mismatch using the ignore
lambda. You may include more than one lambda if needed:
val scientist = scientist<Boolean, Map<String, Boolean>> {
ignore { candidate, control -> candidate.isFailure() }
}
Like in match
candidate and control are of type Outcome
.
When running your test suite, it's helpful to know that the experimental results always match. To help with testing, Scientist defines a throwOnMismatches
field. Only do this in your test suite!
To throw on mismatches:
val scientist = scientist<Boolean, Map<String, Boolean>> {
throwOnMismatches { true }
}
Scientist will throw a MismatchException
exception if any observations don't match.
With an experiment you are setting up tests for the critical paths of your application by specifying a control
and candidate
lambda.
fun experiment = experiment<Boolean, Unit> {
name { "widget-permissions" }
control { user.isAllowedOldWay() }
candidate { user.isAllowedNewWay() }
}
Sometimes you don't want an experiment to run. Say, disabling a new code path for anyone who isn't member. You can disable an experiment by setting a conductibleIf
lambda. If this returns false
, the experiment will merely return the control value.
experiment<Boolean, Unit> {
// ...
conductibleIf { user.isMember() }
}
The conductibleIf
lambda can also take a contextProvider
as a parameter:
experiment<Boolean, Map<String, Boolean>> {
// ...
conductibleIf { context -> context()["externalCondition"]!! }
}
Scientist catches and tracks all exceptions thrown in a control
or candidate
lambda. To catch a more restrictive set of exceptions add a catches
lambda to your experiment setup:
experiment<Boolean, Unit> {
// ...
catches { e -> e is NullPointerException }
}
This tells the scientist to catch NullPointerException
and throw any other exception when running the control
and candidate
lambdas.
You can compose and execute experiments by putting conduct
between scientist
and an experiment context:
val result = scientist<Double, Unit> {
publisher { result -> println(result) }
} conduct { // or conduct experiment {
experiment { "experiment-dsl" }
control { Math.random() }
candidate { Math.random() }
}
The Java interoperability can certainly be improved but should be sufficient for now:
public boolean isAllowed(User user) {
Scientist<Boolean, String> scientist = Setup.scientist(setup -> setup
.context(() -> "execute")
);
Experiment<Boolean, String> experiment = Setup.experiment(setup -> setup
.name(() -> "experiment-name")
.control("test-control", () -> user.isAllowedOldWay())
.candidate("test-candidate", () -> user.isAllowedNewWay())
.conductibleIf((contextProvider) -> contextProvider.invoke().equals("execute"))
);
return scientist.evaluate(experiment);
}
Maven:
<dependency>
<groupId>com.github.spoptchev</groupId>
<artifactId>scientist</artifactId>
<version>1.0.2</version>
</dependency>
Gradle:
compile 'com.github.spoptchev:scientist:1.0.2'