wix-incubator / wix-http-testkit

Tools for testing HTTP services

HTTP Testkit

Overview

Wix Http Testkit is a library that will address many of the End-to-end testing concerns you might encounter.

Wix HTTP Testkit aims to be:

• Simple Testing REST services or starting mock/stub servers is very simple and requires very few lines of code.
• Fast Leveraging Akka-Http infrastructure, starting servers takes milliseconds.
• Integrated: Other than providing a set of DSLs to support composing and executing REST HTTP calls and creating and configuring web servers, it also contains out of the box matcher libraries for Specs² to easily validate each aspect of the tested flow.

Getting Started

Testing Client

Import DSL

import com.wix.e2e.http.client.sync._

Issue Call

    val somePort = 99123 /// any port
    implicit val baseUri = BaseUri(port = somePort)


    get("/somePath", 
        but = withParam("param1" -> "value") 
          and header("header" -> "value") 
          and withCookie("cookie" -> "cookieValue"))

Use Specs2 Matcher suite to match response

    import com.wix.e2e.http.matchers.ResponseMatchers._

    put("/anotherPath") must haveBodyWith("someBody")

For more info see Http Client Documentation and Response Matchers Suite.

Web Servers

Import Factory

    import com.wix.e2e.http.server.WebServerFactory._

Run an easily programmable web server

    val handler: RequestHandler = { case r: HttpRequest => HttpResponse()  }
    val server = aMockWebServerWith(handler).build
                                            .start()

Or run a programmable that will record all incoming messages

    val server = aStubWebServer.build
                               .start()

Match against recorded requests

  import com.wix.e2e.http.matchers.RequestMatchers._
  
  
  server must receivedAnyRequestThat(must = beGet)

For more info see Web Server Documentation and Request Matchers Suite.

Usage

HTTP-testkit version '0.1.25' is available on Maven Central Repository. Scala versions 2.11.x, 2.12.x and 2.13.x are supported.

SBT

Simply add the wix-http-testkit module to your build settings:

libraryDependencies += "com.wix" %% "http-testkit" % "0.1.25"

Maven

<dependencies>
  <dependency>
    <groupId>com.wix</groupId>
    <artifactId>http-testkit_${scala.dependencies.version}</artifactId>
    <version>0.1.25</version>
  </dependency>
</dependencies>

Documentation

• Rest Client: a declarative REST client Documentation.
• Simplicator Web Servers: Easily configurable web servers Documentation.
• Specs² Matchers Suite: Comprehensive matcher suites Response Matchers and Request Matchers.

Contribute

Ideas and feature requests welcome! Report an issue or contact the maintainer directly.

License

This project is licensed under MIT License.

wix-http-testkit's Issues

hash fragments are not properly handled in the client

CC: @avgar

Change scheme. Don't use port

I have endpoint for example https://github.com/wix/wix-http-testkit/
Can I use http-testkit with https scheme and without port?

create build for 2.13

for now blocked till akka-http will have a 2.13 version

Mocker server does not support modifying handlers

In my tests:

• Stub server supports altering handlers.
• Mock server does not support altering handlers.

Same handlers, when used as initial handlers, work just fine with mock server. Trait and methods exist but they seem to have no effect at runtime.

fixed main readme

need some more quick examples on the main readme

MarshallerErrorException description

The error looks like this []
Failed to unmarshall: [$content]. The content is a json string and "[" is a valid json character symbolizing an array. When I saw this message I tried to figure out why my json is being marshalled as an array and it confused me. I think we should remove the brackets here to not have this confusion.

build script issues

change artifacts, we should publish:

• http-testkit_2.11, 2.12
• http-testkit-specs2_2.11/2.12

so we need to assemble modules together: client/server/core to a single jar,
not publish - contract-tests, examples

allow better matching against redirection url

param order is not working for this example:

http://tfeabpnpvuv99fu3dhf7/apps/ecommerce/mobile/v1/752988c3-7967-4648-83ce-21f7e72f1abd/af87a67d-5d8e-4cea-b72c-8b546c9ef69d?Filter[0][Relation]=Equal&Paging[PageNumber]=3430&Filter[0][Value]=VNPLaUDmDtWTyEMTK9yk&X-OPERATION=GetItemList&Filter[0][Field]=CategoryId&Paging[ElementsPerPage]=42'
http://tfeabpnpvuv99fu3dhf7/apps/ecommerce/mobile/v1/752988c3-7967-4648-83ce-21f7e72f1abd/af87a67d-5d8e-4cea-b72c-8b546c9ef69d?X-OPERATION=GetItemList&Filter[0][Field]=CategoryId&Filter[0][Relation]=Equal&Filter[0][Value]=VNPLaUDmDtWTyEMTK9yk&Paging[ElementsPerPage]=42&Paging[PageNumber]=3430' }

Tests freeze when connection depletion happens

Current limit: 32 connections & 4 pipelines.

If tests do not consume response body, it gets depleted. Need to:
a) mitigate the problem
b) increase limits

consider scalatest matcher suite as well

consider adding matchers for OSS proposes issues.

Handlers are immutable

The previous version allowed to define handlers and reset them and define new ones.
This version does not support this (due to the underlying akka http code), question is, do we really need that ?
creating a new instance takes 20 millis and they all use the same actor system which makes things simpler.

provide extractors for request entity

server side does not have proper extractors

Path resolution is not intuitive

In path resolution, there are two factors:

• context path coming from BaseUri
• path coming from method parameter

For example:

context=/abc/
path=more/stuff
/abc/more/stuff

context=/abc/
path=stuff
/abc/stuff

There are a couple related problems / unintuitive (unexpected behaviours):

context=None
path=cats
/cats

context=None
path=/cats
//cats

Additionally, while context is supposed to be something you don't have to think about in tests, you actually care about it:

context=/my
path=cats
/mycats

context=/my
path=/cats
/my/cats

context=/my/
path=cats
/my/cats

context=/my/
path=/cats
/my//cats

So, looking at input => result, it's easy to understand how it works but it's not so intuitive when using the API.

What I would expect from the API is to handle these two paths like file system paths. Basically:

"" + "a" -> "/a"
"" + "/a" -> "/a"
"/" + "a" -> "/a"
"/" + "/a" -> "/a"
"/b" + "a" -> "/b/a"
"/b" + "/a" -> "/b/a"
"/b/" + "a" -> "/b/a"
"/b/" + "/a" -> "/b/a"

There also are some edge-cases which, I think, might also be important for some:

"/b" + "" -> "/b/"
"/b/" + "" -> "/b/"
"/b" + "/" -> "/b/"
"/b/" + "/" -> "/b/"
"/b" + "/a" -> "/b/a"
"/b" + "/a/" -> "/b/a/"

The way I see, there are two ways to go around it:
a) implement flexible API which introduces opinions (joins path based on rules defined above)
b) implement strict path checks, for example:

• context must/mustn't not end with /
• path must/mustn't start with /
  which would simplify implementation and avoid silly mistakes and time spent debugging.

waitFor in BlockingRequestManager should not be inifinite

On this line the await timeout for the future is set to Duration.Inf, which assumes that akka-http will honor the request timeout passed to it (withTimeout).
Unfortunately sometimes there are bugs in akka-http and the future may never be fulfilled.
This happened in akka-http 10.1.0-RC1.
It is better for future await to be limited to withTimeout as well (or slightly longer).

jackson dependency

right now the testkit will define a jackson marshaller,
the main problem is that if the user site has a different marshaller they will collide.
I need to extract the marshaller to an external module, if user will want it, he will import it explicitly, otherwise he will need to write one on his own.

Implement multi-part request transformer

Cannot detect response encoding

I found no way to detect whether response from the server had Transfer-Encoding: chunked.

create different packages

test kit should have several modules:

1. http testkit: marshaller/no marshaller * jackson version * testkit
2. specs2 matchers: marshaller/no marshaller * jackson version * specs2 version * testkit

all should be packaged without marshaller and with marshaller
marshallers should come with different jackson bundle

fix readme once all is implemented

Support file upload request transformer

marshaller issues

when more than one marshaller exists, prefer the custom one over the testkit marshaller

when matching with entity

check if the param is not a matcher, if so, fail the request and thrown a meaningful error

java proof library

allow usage in the library from java 8
matchers is an issue so i might not support java matchers (perhaps hamcrest ???)

allow usage of akka routing dsl in mock/stub web server

this will allow a much more convenient way of constructing stubs.
https://doc.akka.io/docs/akka-http/current/routing-dsl/index.html

specs2 matchers for server

mostly request matchers and web server request matchers

add an examples module

with common usecases for all rest usages

Be able to use WixJsonMarshaller with custom object mapper

Hi! Our team is currently in progress of making a shift from spray to the test kit and we want to say thank you for a great effort! However, while doing a refactoring, I found a few issues, that I can't create my own WixJsonMarshaller with custom object mapper, but we do use custom one in our code. It would be great to be able to construct WixJsonMarshaller, passing objectMapper in the constructor.
You can see an example how I did that here https://github.com/wix-private/wix-ecommerce-server/pull/386/files#diff-a3b74c8ec1eb524f74004f2e81fbb1a1R25. If you want, I would be glad submit a PR with the fix. Also, I will open another ticket about the other small issue I've found.

Unexpected behavior of withMultipartData with FileRequestPart

I'd expect this code:

withMultipartData(
  "file" -> FileRequestPart(myFile, ContentTypes.`application/json`),
  "bool" -> PlainRequestPart("true")
)

to be converted to something like:

r.copy(entity = Multipart.FormData(
  Multipart.FormData.BodyPart(
    "file",
    HttpEntity.fromPath(ContentTypes.`application/json`, myFile.toPath),
    _additionalDispositionParams = Map("filename" -> "file.js")
  ),
  Multipart.FormData.BodyPart("createStore", HttpEntity("true"))
).toEntity)

But it turns out, that FileRequestPart transform to just Binary entity without filename.

Throw exception if header matcher is used to check value of `Content-Length`

rename all packages

create proper package names for all (not decided yet)
rework all object names as well

Cannot set custom Host header

When sending a request with an arbitrary Host header - the Host header gets overridden with the value from URL.

Example:

get("http://localhost:12345/somePath", but = withHeaders("Host" -> "somehost.com"))

The request is sent with Host header = localhost:12345 instead of somehost.com

split testkit to modules

• wix-http-testkit (main module to import) - depends on all, contain all builders
• http-testkit-api (contain all common api's between server/client)
• http-testkit-client (client impl)
• http-testkit-server (server impl)
• http-testkit-specs2-matchers (matcher suit for server/client)
• http-testkit-commons (common classes for both modules)
• http-testkit-contract-tests (contract tests for client and server)

header matchers does not represent values changes correctly

if a header has a different value (but same header name) it will fail with a message that it couldn't find the header, i need to print a proper error, found the header but value was different (a != b)

Throw exception if header matchers are used to check `Transfer-Encoding`

Transfer-Encoding is not exposed by Akka as a header. Developers should be warned about it right away.

support extractAs

both implementation blocking/non-blocking

Head requests received as GET by the mock/stub server

This test fails

    "support head requests" in new ctx {
      head("/")
      server must receivedAnyRequestThat(beHead)
    }

This seems to be an akka http server issue, not the client. Probably related to the akka.http.server.transparent-head-requests config as explained here

WixHttpTestkitResources.xmlPrinter is not thread-safe and thus can’t be shared

Hi! While doing a shift from spray to akka-http I found one small issue, calling:

HttpClientRequestBodyTransformers#withPayload(xml: Node): RequestTransformer

in multi-threaded test env, this method uses WixHttpTestkitResources.xmlPrinter, implementation of which is not thread-safe and thus can't be a shared variable. I would propose changing the definition of xmlPrinter from lazy val to def or inlining it here since there is only one use of it. If you would like I can open a PR.

matcher suit for client

port all matchers, add tests and refactor

Wix Modules

create a module that will contain wix concerns to be used internally
implement an easy to migrate Probe -> whatever to allow easy migration

Common handlers

for stub and mock there are no utilities to allow quick generation of the server

I think there should be common handlers for:

return string/entity
match against specific path/paramters
store some in memory data
all request handlers are PURE, they cannot have a state.

allow to inject an external marshaller

We want to have a way to the user to supply us with a custom marshaller.
akka http wants marshaler per type, right now there is a global jackson module.

Possible solutions:

1. leave current implementation, user can supply custom marshaller via implicit resolution
2. config value, allow user to override the default marshaller
3. look in class path for implementations of marshaller and if found one, use it.

build sbt script

build all modules
publish to ???
cross build for 2.11, 2.12

add testkit headers

client should add client-http-testkit and proper version
server should add server-http-testkit and proper version

generate version info using sbt plugin: https://github.com/sbt/sbt-buildinfo

query param assumes '='

https://github.com/wix/wix-http-testkit/blob/master/http-testkit-client/src/main/scala/com/wix/e2e/http/client/internals/package.scala#L46

if the query params do not contain '=' then tests fail with match error.

Leaky abstraction ?

we are using akka/spray on our api's, the main reason for that is that spray modeled http requests for us and did it well, so why should we expose an abstraction on top of another abstraction ?
on the other hand, why should we expose our internals outside ?

Stub server doesn't support chunked entity on request

Create a stub server, send request with chunked entity.
The verification of recorded request entity against the original fails.

Test making a REST call throws exception "akka.http.scaladsl.model.IllegalResponseException: The server-side HTTP version is not supported”

I have a REST endpoint which currently renders a velocity vm
We are starting to develop an SSR version of the page, and so as an interim solution, depending on some business related decision, it will either render the velocity vm as usual, or it will proxy the request to a NodeJS server that will perform the SSR.

In order to IT test, I have used aMockWebServerWith to fake the NodeJS service.
This fake service returns the request it receives, so that in the test I can assert that the request was correct.

This test is successful, but the next test to run always fails with "akka.http.scaladsl.model.IllegalResponseException: The server-side HTTP version is not supported" when it does a get request

Storing response as intermediate value doesn't always work

Response cannot be stored as intermediate value because it doesn't always work.

// does not work
"not be gzipped for a path that's in the exclude list" in {
      val response = get(nonGzip, withAcceptEncodingGZip and withAccept(mimeType = "application/json"))
      response must beSuccessfulWith(Dog())
      response must beNotHaveContentEncodingGzipHeader
    }

// works
"not be gzipped for a path that's in the exclude list" in {
      get(nonGzip, withAcceptEncodingGZip and withAccept(mimeType = "application/json")) must beSuccessfulWith(Dog()) and beNotHaveContentEncodingGzipHeader
    }