quii / mockingjay-server

Fake server, Consumer Driven Contracts and help with testing performance from one configuration file with zero system dependencies and no coding whatsoever

mockingjay server

Mockingjay lets you define the contract between a consumer and producer and with just a configuration file you get:

• A fast to launch fake server for your integration tests
• Configurable to simulate the eratic nature of calling other services
• Consumer driven contracts (CDCs) to run against your real downstream services.

Mockingjay makes it really easy to check your HTTP integration points. It's fast, requires no coding and is better than other solutions because it will ensure your mock servers and real integration points are consistent so that you never have a green build but failing software.

• Installation - Download a binary, use a Docker image or go get
• Rationale
• See how mockingjay can easily fit into your workflow to make integration testing really easy and robust

Running a fake server

---
 - name: My very important integration point
   request:
     uri: /hello
     method: POST
     body: "Chris" # * matches any body
   response:
     code: 200
     body: '{"message": "hello, Chris"}'   # * matches any body
     headers:
       content-type: application/json

# define as many as you need...

$ mockingjay-server -config=example.yaml -port=1234 &
2015/04/13 14:27:54 Serving 3 endpoints defined from example.yaml on port 1234
$ curl http://localhost:1234/hello
{"message": "hello, world"}

Check configuration is compatible with a real server

$ mockingjay-server -config=example.yaml -realURL=http://some-real-api.com
2015/04/13 21:06:06 Test endpoint (GET /hello) is incompatible with http://some-real-api - Couldn't reach real server
2015/04/13 21:06:06 Test endpoint 2 (DELETE /world) is incompatible with http://some-real-api - Couldn't reach real server
2015/04/13 21:06:06 Failing endpoint (POST /card) is incompatible with http://some-real-api - Couldn't reach real server
2015/04/13 21:06:06 At least one endpoint was incompatible with the real URL supplied

This ensures your integration test is working against a reliable fake.

Inspect what requests mockingjay has received

 http://{mockingjayhost}:{port}/requests

Calling this will return you a JSON list of requests

Make your fake server flaky

Mockingjay has an annoying friend, a monkey. Given a monkey configuration you can make your fake service misbehave. This can be useful for performance tests where you want to simulate a more realistic scenario (i.e all integration points are painful).

---
# Writes a different body 50% of the time
- body: "This is wrong :( "
  frequency: 0.5

# Delays initial writing of response by a second 20% of the time
- delay: 1000
  frequency: 0.2

# Returns a 404 30% of the time
- status: 404
  frequency: 0.3

# Write 10,000,000 garbage bytes 9% of the time
- garbage: 10000000
  frequency: 0.09

$ mockingjay-server -config=examples/example.yaml -monkeyConfig=examples/monkey-business.yaml
2015/04/17 14:19:53 Serving 3 endpoints defined from examples/example.yaml on port 9090
2015/04/17 14:19:53 Monkey config loaded
2015/04/17 14:19:53 50% of the time | Body: This is wrong :(
2015/04/17 14:19:53 20% of the time | Delay: 1s
2015/04/17 14:19:53 30% of the time | Status: 404
2015/04/17 14:19:53  9% of the time | Garbage bytes: 10000000

Building

Requirements

• Go 1.3+ installed ($GOPATH set, et al)
• golint https://github.com/golang/lint

Build application

$ go get github.com/quii/mockingjay-server
$ cd $GOPATH/src/github.com/quii/mockingjay-server
$ ./build.sh

MIT license

mockingjay-server's Issues

There were duplicated requests found ...

There were duplicated requests found because request body limited 50 bytes

mockingjay-server/mockingjay/request.go

Line 154 in 4a9da10

if len(r.Body) > 0 && len(r.Body) > 50 {

Add expvar support

https://golang.org/pkg/expvar/

Be able to get config from a URL

Currently you have to provide a filepath to a config file.

This is fine for local development, but when you're trying to distribute your config to another team to run a CDC it becomes painful from a CI perspective.

It would be easier if the consumer could serve their config over HTTP and then the producer can just point mockingjay at a URL to run their consumer's CDCs.

[feature-request] Ability to specify wildcards for config in CDC mode

mockingjay-server -config=*.yaml -realURL=http://my.awesomeservice.com

Wouldn't it be nice to be able to run this? 😄

A pattern I see my team using more and more often, is to have a single yaml config file per consumer, and run them all against the same instance of our service.
Right now, we use run mockingjay once per config (in a loop), but wildcard support would be great.

If request body is JSON parse it and use that for equivalence

It's annoying having to provide a precise string representation of a request body, see if the request is JSON and if so parse both to see if equal

Regex/Glob matching for fake server

Background

People in the past have asked for wildcards/regex/glob for matching request URLs when running a fake server. I have had to say no because it's difficult to then keep the CDC functionality working.

e.g.

uri: /foo/{id}

What "id" do I send to the real server in CDC mode?

Possible solution

Have a second, optional field in the request section which defines a regex of the URL.

---
 - name: Test endpoint
   request:
     uri: /hello/world
     regex: /hello/{A-Z} 
     method: GET
   response:
     code: 200
     body: '{"message": "hello, world"}'
     headers:
       content-type: text/json

Note: Glob might be friendlier, and could be an option. But for now will assume regex.

On startup

• Make sure that the normal URI matches the regex.
• This ensures that when you're running as a fake server with variable parameters that it is still close to what call the CDC will make.
• That way you cant run in to a scenario where all your tests are hitting a "variable" URL that doesn't actually exist.

In fake mode

When a request arrives, obey the usual matching orders but if the URI doesnt match the "concrete" one, then see if it matches the regex. Need to make sure exact URLs are checked first.

---
 - name: Test endpoint2
   request:
     uri: /hello/world
     regex: /hello/{A-Z} 
     method: GET
   response:
     code: 200
     body: '{"message": "hello, world"}'
     headers:
       content-type: text/json

 - name: Test endpoint
   request:
     uri: /hello/chris
     method: GET
   response:
     code: 200
     body: '{"message": "hello, chris"}'
     headers:
       content-type: text/json

In this scenario if you send /hello/chris it should be the second one that "wins", not the first one by virtue of the regex

CDC mode

No changes

Credit to @chandley and @thecodeite

Handle basic auth

Need to do stuff like

user:[email protected]/foo/bar

Check XML structure when applicable

Like the JSON structure check.

Try and make output better when JSON body is incompatible

Currently it just spams out the output of the JSON it got when it's incompatible. If it's a large piece of JSON this isn't very helpful

This will involve updating https://github.com/quii/jsonequaliser

Support partial matches on bodies for compatability

Currently if you are just checking the compatibility of a body it needs to be an exact match.

It would be nice if you could somehow look for a particular substring within the downstream response.

Maybe expand on the wildcard stuff

body : '*foo*'

Add flag to configuration to disable CDC call

For some configurations it might be impossible for the downstream server to pass the consumer driven contract but you still want to model it in MJ.

For instance, you might want a configuration where given a certain resource

foo.com/stuff/failme

That it returns a 500 Internal Server Error. You wouldnt be able to make the CDC pass in this instance without help from the developers of your downstream server.

So a flag should be added to configuration to let a user disable CDC for that particular endpoint.

It should still log on the CDC section IGNORED.

runtime error: index out of range

panic: runtime error: index out of range

goroutine 20 [running]:
panic(0x2d5820, 0xc4200121c0)
	/home/travis/.gimme/versions/go1.7.linux.amd64/src/runtime/panic.go:500 +0x1a1
github.com/quii/mockingjay-server/vendor/github.com/quii/jsonequaliser.getJSONNodeFromString(0xc420013ee0, 0x2, 0xc4200ce900, 0x2, 0x8)
	/home/travis/gopath/src/github.com/quii/mockingjay-server/vendor/github.com/quii/jsonequaliser/json.go:14 +0x216
github.com/quii/mockingjay-server/vendor/github.com/quii/jsonequaliser.IsCompatible(0xc420013ee0, 0x2, 0xc4200ce7e8, 0x2, 0x0, 0xc42002c400, 0xc42000f7c0)
	/home/travis/gopath/src/github.com/quii/mockingjay-server/vendor/github.com/quii/jsonequaliser/compatability.go:13 +0x39
github.com/quii/mockingjay-server/mockingjay.checkBody(0xc4200ce7e8, 0x2, 0xc420013ee0, 0x2, 0xc4200ce7e8, 0x2, 0x0)
	/home/travis/gopath/src/github.com/quii/mockingjay-server/mockingjay/compatibility.go:147 +0xca
github.com/quii/mockingjay-server/mockingjay.(*CompatibilityChecker).check(0xc4200c2aa0, 0xc420486f18, 0x7fff5fbff712, 0x22, 0x0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/quii/mockingjay-server/mockingjay/compatibility.go:114 +0x4b2
github.com/quii/mockingjay-server/mockingjay.(*CompatibilityChecker).compatabilityWorker(0xc4200c2aa0, 0xc420774720, 0x1e, 0x0, 0xc42000b2f0, 0x28, 0x0, 0xc420013ea8, 0x3, 0x0, ...)
	/home/travis/gopath/src/github.com/quii/mockingjay-server/mockingjay/compatibility.go:70 +0x72
created by github.com/quii/mockingjay-server/mockingjay.(*CompatibilityChecker).CheckCompatibility
	/home/travis/gopath/src/github.com/quii/mockingjay-server/mockingjay/compatibility.go:56 +0x2c0

Autoreload configuration

Would be nice if you edit the YAML file that it wouldnt require a restart.

Especially nice in Dockerland.

Wildcard on request URL

Useful if you want to just return a canned response regardless of the URL

No usage message if non-existent yaml file supplied in CDC mode

If a non-existent yaml file is provided as config, mockingjay exits without any indication of what happened.

bash-3.2$ mockingjay-server -config=imaginary.yaml -realURL=http://my.awesomeservice.com
bash-3.2$ mockingjay-server -config=imaginary.yaml -realURL=http://my.awesomeservice.com
bash-3.2$

It would be nice if it pointed out that imaginary.yaml is make-believe.

Running contract tests against the Producer

I understand how I can use this to mock the producer, and run tests on the consumer.
But what about the other way around - running tests on the producer against the same contracts and making sure the producer does break them?

Proposal: Allow returning simple code results

I'd love to randomize the results - no different than allowing simple code execution. In this case, it might look like the following

---
 - name: Test endpoint
   request:
     uri: /hello
     method: GET
   response:
     code: 200
     body: '{"message": "hello, world", "temperatureOutside": `fmt.Sprintf("%s", rand.Intn(100)))`}'
     headers:
       content-type: text/json

There is a much much much less ugly way to do it than the above, but you get the idea.

Make field type error less awful

JSON err on field 'counterparts->entities->headquarters->state->value' : Field is not a float in other JSON

This is nice, but what is the type of the field?

Be able to define different response body for different response code

So when it's sending 400 response defined in the monkey-business.yaml instead of normal 200, it would be nice to have server return different response body

Some kind of locking issues

Seems on windows at least there are problems with the configuration file getting locked.

Probably not closing a file handle or something

How to delay a specific request

Not adding delay randomly to all requests.

@quii Is it doable?

Reference a file as the response body instead of inline JSON

We want to return large data sets in our application and have trouble editing the inline JSON inside of the YAML config file.

Can mockingjay-server include an optional reference to a separate JSON file in an endpoint that means we can edit the JSON in a JSON formatted file, instead of editing inline JSON in a YAML file.

e.g. the config could look like this

- name: My endpoint
  request:
    uri: /api/count
    method: POST
    headers:
      content-type: application/json
    body: '{
      "data": [],
    }'
  response:
    code: 200
    # Replace the `body` property with a reference to a JSON file
    bodyAsFile: './my-endpoint-response.json'

The monkey shouldnt slow down / tamper with the debugging routes

i.e /requests or the APIs for creating new endpoints

When there are Regex problems, the output doesn't give information about where that problem is

Example output:

service_name                | 2020/04/15 14:49:29 A regex defined in the request does not pass against it's defined URI

Sometimes it can be a bit difficult to figure out which config has the regex issue. Could it be updated so that the output says which config had the issue?

e.g

service_name                | 2020/04/15 14:49:29 A regex defined in the request for config 'Name of the config' does not pass against it's defined URI

go get throws assetFS undefined

../../quii/mockingjay-server/main.go:45: undefined: assetFS

`/requests` endpoint does not indicate if the request was successfully matched or not

Could also provide the served response body, which would be very helpful with an application that sends lots of requests to a single server.

Better error message for empty array in example JSON

When mockingjay-server is being used to run CDC tests, and the body element of the config contains JSON data which has an empty array, there is an exception thrown by a Go panic. It would be good if there was a clean error message to the user. It would be even better if the message pointed out exactly which element of the JSON was wrong.

Matching request header names are case sensitive

X-API-KEY should match with X-API-Key

Canned Response Headers cannot contain spaces

Neither the key or the value of the headers in the canned response can contain spaces.

This blocks using cookies that contain expiry dates.

The code responsible for this might be in https://github.com/quii/mockingjay-server/blob/master/mockingjay/validationHelpers.go

Example:

 - name: render Login form
   request:
     uri: /login
     regexuri: \/login.*
     method: GET
   response:
     code: 200
     body: 'non-empty body'
     Set-Cookie: 'Expires=Tue, 28 Apr 2037 13:46:30 GMT'

Manage releases better

Whilst Travis is creating binaries on every successful build, it is putting them inside "pre-release" which is a bit confusing. Would be good if it could make minor releases every time maybe?

Sending form data doesn't seem to work well

---
 - name: Response from Mailgun receiving message on Activism
   cdcdisabled: false
   request:
     uri: /v3/aslive.activistmonitor.com/messages
     method: POST
     body: 'from=info%40sandbox47ad8d7815c24338afa44c16a9492f36.mailgun.org&to=test.notifications%40mergermarket.com&subject=send%20a%20message&text=this%20is%20a%20message&html=foo&v%3Aenv=local&v%3AmatchId=cj&v%3Aapplication=notifications'
   response:
     code: 200

This doesn't seem to send the data correctly to mailgun as we get a 400. The interface here for users sucks too, perhaps adding a formdata field for the YAML will improve things

Headers check is too strict

Given

---

---
 - name: Issue 42 is where extraneous headers sent will trigger a 404. It should ignore unknown headers
   request:
     uri: /test
     method: POST
     headers:
      content-type: application/json
   response:
     code: 200

If a request is sent with other headers, such as Content-Length it is seen as not matching.

This is because mockingjay/request.go hopes that the two header arrays match (deepEqual). https://github.com/quii/mockingjay-server/blob/master/mockingjay/request.go#L95

Instead it should see if the incoming requests just contains all the ones we expect, ignoring others.

Automated tests are in server_test.go . The test for this should probably go in request_test.go

Writing log messages to stderr

I'm not familiar with the go language, but it appears log.Printf prints to stderr instead of the expected stdout for info-level log messages. I created a gulp task to start my mock api server when I startup my dev server:

    gulp.task('serve:api', [], function () {
      var serverPath = './linux_amd64_mockingjay-server';
      var mockPath = path.join(__dirname, './mock.yml');
      var mockingjay = spawn(serverPath, ['-config=./mock.yml']);

      mockingjay.stdout.on('data', function (data) {
        gutil.log('stdout: ' + data);
      });

      mockingjay.stderr.on('data', function (data) {
        gutil.log(gutil.colors.red('[Mockingjay Error]: ' + data));
      });

      mockingjay.on('close', function (code) {
        gutil.log('Mockingjay exited with code ' + code);
      });
    });

when running I get the message Serving 2 endpoints defined from ./mock.yml on port 9090 through stderr:

Refactor: Replace anonymous struct initialisers with named ones

Request{testURL, nil, "GET", contentTypes, ""} Bad

Request{ URI: "", Method: "POST", Headers: nil, Body: ""}

Good

If you put a URL escaped in the config YAML, it gets unescaped when making request to the real server

/document/10.1007%2Fs00414-006-0114-x

/document/10.1007/s00414-006-0114-x

swagger support

it would be really nice to pass MJ a swagger schema, along with some of the flakiness parameters and have MJ serve the endpoints in the schema according to the specified flakiness.

Request headers not printed on 404 response from server

When comparing request bodies (and probably response), if it is JSON it shouldn't be whitespace sensitive

---
    - name: Valid login details
      request:
        uri: /user-token
        method: POST
        body: '{"email":"[email protected]","password":"xxx"}'
      response:
        code: 200
        body: '{"token": "1234abc"}'

Should be compatible with a request body that has spacing in it. Currently it isn't.

Monkey: http: multiple response.WriteHeader calls

When monkeying around, monkey is overriding whatever it is changing, which is causing "http: multiple response.WriteHeader calls" to be logged.

Don't be concerned with order of query string params

If you have a request URL defined like

/books?id=1&id=2

It should match if the caller sends /books?id=2&id=1

Investigate a more standard test output

Currently the CDC check is just some console output (with a non 0 exit code for failure).

This is probably enough, but maybe there is a nicer way that will integrate with some CI systems

Environment variable? API Key?

Hello!

Is there a way to have the body read from environment variable?

We need this for example to add an API KEY inside the yaml file, so we can run the verify against the real API.

Body doesn't seem to be sent in request when doing compatibility check

---

- name: Valid login details
  request:
    uri: /user-token
    method: POST
    body: '{"email":"[email protected]","password":"secret"}'
    headers:
     X-Api-Key: supersecret
  response:
    code: 200
    body: '{"token": "1234abc"}'

[feature request] Can we make a request timeout configurable

Return list of requests received

Make a nice web UI to view the configuration

Currently if you hit an unknown endpoint you get a text based, boring and hard to read representation of your YAML.

It would be better if MJ had a nice UI to view the endpoints, be able to try the tests out and even run the CDC part in the browser.

Eventually this could lead to MJ configuration being totally editable via a UI instead of editing a YAML file, which might make it a bit more accessible.

Duplicate URLs

IS BAD

Compatibility check doesn't verify HTTP response headers

The wiki at https://github.com/quii/mockingjay-server/wiki/How-does-compatibility-work%3F
says:
"Mockingjay will check easy things such as status codes, URLs and headers to see if they match. If you define you want a particular header and it isn't in the response, then it will fail."

However, it's not checking HTTP response headers.

Set up two mocking servers - one to be the 'real' server, and one to be the 'mock'.
Both are identical, except that we expect a response header of "content-type: application/json".
So, we specify the expectation in the mock server config, and omit it from the real server config.

hello-mock.txt:

• name: My very important integration point
  request:
  uri: /hello
  method: POST
  body: "Chris" # * matches any body
  response:
  code: 200
  body: '{"message": "hello, Chris"}' # * matches any body
  headers:
  content-type: application/json

  hello-real.txt:

• name: My very important integration point
  request:
  uri: /hello
  method: POST
  body: "Chris" # * matches any body
  response:
  code: 200
  body: '{"message": "hello, Chris"}' # * matches any body

And then run the 'real' server via:
windows_amd64_mockingjay-server.exe -config=./hello-real.txt -port 9091

Run the mock server via:

C:\dev\2015\mockingjay>windows_amd64_mockingjay-server.exe -config=./hello-mock.txt -port 9090 -realURL=http://localhost:9091
2015/06/09 23:10:27 ? My very important integration point (POST /hello)
2015/06/09 23:10:27 All endpoints are compatible

The server reports that all endpoints are compatible, even though the 'real' server did not provide a content-type header.

Expected result: report an incompatibilty

Output from the 'real' server - showing that it is not returning a Content-Type header:
C:\dev\2015\mockingjay> curl -H "Content- Type: application/json" -X POST -d "Chris" -i http://localhost:9091/hello
HTTP/1.1 200 OK
Date: Wed, 10 Jun 2015 05:21:45 GMT
Content-Length: 27
Content-Type: text/plain; charset=utf-8

{"message": "hello, Chris"}

Output from the 'mock' server - showing the Content-Type header is returned:
C:\dev\2015\mockingjay> curl -H "Content- Type: application/json" -X POST -d "Chris" -i http://localhost:9090/hello
HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 10 Jun 2015 05:24:36 GMT
Content-Length: 27

{"message": "hello, Chris"}

For the go-uninititated, could you provide a little more installation info please?

Using "go get ...." might have worked (maybe), but I have no $mockingjay-server application available on my terminal after sourcing all various profiles etc. Please can you update with any other steps required. I'm on MAC OS X. Thanks

Possible leaked fd

you have:
https://github.com/quii/mockingjay-server/blob/master/compatability.go#L79

If an unexpected response code comes in, you return prior to closing the response body. Easy fix: just move that line up to before the if-status-code check

cheers