hypermedia-app / hypertest

Hypermedia-driven API tests

Hypertest DSL

Domain-Specific Language for testing APIs, hypermedia-style

❗ Early version; work in progress 🚧

Introduction

Hypertest DSL is a tool which lets API authors build end-to-end testing suites as a series of client-server interactions which resemble the implementation of a hypermedia-driven applications.

In other words, the tests are solely executing based on links and forms, all of which should be provided by the server in resoruce representations.

The DSL itself aims to be agnostic of the media type. A test runner could be implemented to run against any RESTful API such as SIREN, Hydra, NARWHL, HAL-FORMS, etc.

Quick start

Tests are written down as .api files. Here's an all too-simple test document:

# tests/scenario.api

With Class "Person" {
  Expect Property "name"
} 

It will instruct the runner to check that any ocurrence of the Person class has a name property.

More examples are available on testing.hypermedia.app.

Install compiler

Runners will work with a JSON representation of the test documents. A compiler can be used to transform the .api files to JSON:

npm i -g @hydrofoil/hypertest

Run it by passing a directory. You may try it on the samples directory in this repository:

hypertest-compiler ./api-example

It will produce a .api.json for each test scenario document.

Developing and running locally

Setting up development environment

The DSL and compiler are implemented in Java using Xtext.

• Make sure to install EditorConfig plugin. For example this one
• npm install to set up git hooks
• Install a Java JDK (Java 8)
• Install Eclipse IDE for Java and DSL Developers. Version 2019.03. Newer may not work.
• (In Eclipse) Add git repo:
  • Show the "Git Repositories" view: Window | Show View | Other... | Git | Git Repositories
  • Add git repo: Add an (existing) Repository in the "Git Repositories" view
    • Select ~/git/hypertest, Add
• (In Eclipse) Import Projects: File | Import ... | Git | Projects from Git, Next
  • Select Existing local repository, Next
  • Select hypertest, Next
  • Select Import existing Eclipse Projects and verify that the Working Tree top-node is selected, Next
  • Verify that all projects are selected, Finish

The projects should now be imported and the build finish without errors.

Run simple web editor

Follow the instructions from this blog post to set up a local Tomcat instance running the Language Server and simple web UI.

Running locally

Easiest way to build a running compiler is to build with Maven from command line and have jDeploy wire up an executable in local bin path:

npm i -g jdeploy
mvn package
jdeploy install
hypertest-compiler api-examples

Publishing a newer version to NPM

@hydrofoil/hypertest is published using jDeploy which packages the executable JAR in an easy to use node package.

npm run release
git push --follow-tags origin master

Travis does the rest.

hypertest's Issues

Refactor generators to use a JSON builder

The generators should be implemented with a builder instead of string templates. That way the methods will cleaner, not duplicated as much and reusable.

Create language highlighting style for VuePress

Operation should be usable as statement

Currently operation can only appear as a block.

It should be possible to only check that an operation exists. It will still be invoked if a matching top-level block exists.

Add validation error when a variable is not found in current block

With Class "http://localhost:5678/api#Installation" {
    Expect Operation "http://localhost:5678/api#InstallOperation" {
        Invoke {
            Expect Status 201
            Expect Header Location [entrypoint]

            Follow [entry]
        }
    }
}

In the snippet above the variable entry used does not match the entrypoint being "declared". It should be validated and an error shown guarding against such errors

Property statement should handle more than just string

Current the property rule only accepts STRING values

ExpectModifier 'Property' name=Identifier (expectation=STRING)?

I would like to at least also support

• integers, decimals, booleans
• numeric comparison
• regex

Expect Property "foo" true
Expect Property "age" Less Than 21
Expect Property "extension" Matching "$.docx?"

Additionally hydra could have special type assertion check rdf:type property, used with URIs

Expect Type hydra:Collection

Some steps should be repeatable

Currently the implicit behaviour of most steps is to execute only once, the first time an appropriate representation is received.

It should be possible to add an option for them to be processed more than once. For example to always follow a link:

With Link "up" {
    Follow always
}

Generator should not throw on unmapped prefix

Currently using a prefixed name with an unknown prefix causes the generator to throw.

The exception should be avoided by implementing validation or cross-references for prefix declaration rule.

Prefixed name cannot contain a hyphen

This does not parse but should

PREFIX mov: <https://hydra-movies.herokuapp.com/doc#>

Expect Operation mov:create-movie {}

Deployment workflow for the generator

I'd like get to a point where we actually publish something working pretty soon.

I would like to be able to run a generator to pipe the generated JSON directly into the node test runner. Preferably so that the runner consumes the DSL file and no JSON is exposed by default:

hydra-validator e2e --entrypoint http://my.hydra.api --scenarios my.hydra.api

And if not from node directly, then at least piping, like:

api-scenario-parser tests.api | hydra-validator e2e --entrypoint http://my.hydra.api

Hydra should use URI/QName terminals instead of string

Create VS Code plugin

https://blogs.itemis.com/en/integrating-xtext-language-support-in-visual-studio-code

Setting up operation invoke (request)

The request input should be customisable in a given Invoke block. At least two options are necessary: body and headers.

Here's a possible structure, adding a second block for setting up the request a'la HTTP syntax

With Operation "CreateMovie" {
    Invoke {
        // headers like in HTTP
        Content-Type text/turtle

        // body in backticks like markdown
        ```
        <> a mov:Movie ; schema:title "Rocky 8" .
        ```
    } => {
        // response & representation checks
    }
}

Property value expectations

Intro

We should be able to inspect properties on resources and optionally, their values. Comparing values can become rather difficult.

DSL Snippet

With Class http://schema.org/Person {
    # There should be any value of that property
    Expect Property http://schema.org/age

    # There should be a specific string value (quotes required?)
    Expect Property http://schema.org/name "Tomasz"

    # Test boolean with true/false
    Expect Property http://schema.org/alive true
}

Not sure about URL properties. It's a bit RDF/LD-specific, but maybe we could do angel brackets Expect Property knows <http://example.org/another-person>?

Generated JSON

{
  "steps": [
    {
      "type": "Class",
      "children": [
        {
          "type": "Property",
          "propertyId": "http://schema.org/age"
        }, 
        {
          "type": "Property",
          "propertyId": "http://schema.org/name",
          "value": "Tomasz"
        },
        {
          "type": "Property",
          "propertyId": "http://schema.org/alive",
          "value": true
        }
      ]
    }
  ]
}

Add Link block

Intro

Link is one of the elementary hypermedia controls. We need to support it on top level, as well as nested inside other blocks.

Implementation note

The semantics of a link block is to perform a request to fetch the linked representation and execute all matching scenarios nested in the link itself as well as top-level scenarios.

Top-level Link

• on top level a link must have children.
• on top level a link is never mandatory

With Link "canonical" {
    Expect Property "x" "value"
}

Nested link

• nested link can be preceded by With or Expectmodifiers
• Expect link does not need children
• With link requires children (doesn't make sense without them)

With Class "Klass" {
    # Mandatory, strict check, only verify existence, will not dereference
    Expect Link "canonical"

    # Mandatory
    Expect Link "next" {
        Expect Status 200
    }

    # Optional; dereferences if present; ignores otherwise
    With Link "up" {
        Expect Link "down"
    }
}

Generated JSON

{
  "type": "Link",
  "relation": "canonical",
  "strict": true,
}

strict is true when Expect modifier is in effect.

Explain the response and representation context

Most steps are either applicable to the HTTP response (such as inspecting response headers) or to resource representation (usually that means the body).

The documentation should be clear about it in the introductory pages.

Create JSON Schema for generated output

It could be good to have a JSON Schema to validate the generated output. Or, for the case when someone would wish to author such document by hand

Constraining blocks based on representation state

To branch the scenario depending on individual properties of the values, I would extend the syntax with a generic constraint concept and taking advantage of the fact that a Link or Property can occur multiple times too.

For example, to run child steps depending on a flag, one could check that different states of a resource have different links:

With Class "order" {
    When Property "state" Equals "incomplete"

    Expect Link "payment"
}

With Class "order" {
    When Property "state" Equals "complete"

    Expect Link "invoice"
}

Add outDir parameter

In the fix for hypermedia-app/hypertest-docker#1 I added a low-fi solution where the files are simply copied to a temporary directory and converted there.

A proper solution would be add an optional CLI parameter to select the output directory. The important detail is to preserve the original directory structure.

Also I would like to add some kind of library to safely handle the command line arguments.

Method statement

It could be useful to assert request's method to ensure that what we expect to be DELETE is actually that, etc.

For Hydra that would be

With Operation api:DeletePerson {
    Expect Method "DELETE"
}

It is impossible to check resource id

It should be possible to assert the URI of embedded resource objects within a representation.

Preferred solution

Create a specialised property just for Hydra in the form of

Expect Id <http://example.com/uri>

Recursive class block

Intro

Contents of a Class block should be able to appear under property assertions to create deep structure checks.

DSL snippet

The snippet below checks if a nested resource also has a name

With Class Person {
    Expect Property knows {
        Expect Property name
    }
}

The above should pass for a hypothetical JSON like

{
  "type": "Person",
  "knows": {
    "name": "Michael"
  }
}

Generated JSON

{
  "steps": [
    {
      "type": "Class",
      "classId": "Person",
      "children": [
        {
          "type": "Property",
          "propertyId": "knows",
          "children": [{
            "type": "Property",
            "propertyId": "name"
          }]
        }
      ]
    }
  ]
}

Support top level class block

Intro

To test resources, we need to be able to distinguish their shapes. Hydra and SIREN both come with a term Class, which we could use a common denominator for a "type of resource".

DSL snippet

With Class http://schema.org/Person {
}

Details

Note that the class id does not have to be a URI. Any string should be acceptable. No whitespace for now

Generated JSON

This should translate to a root level Class step

{
  "steps": [{
    "type": "Class",
    "classId": "http://schema.org/Person"
  }]
}

Handling multiple values of properties

Introduction

A representation property can have multiple values. Obvious example of this are collection resources but it could be anything. Both Link and Property thus need to support that in scenarios.

Proposal

Run all children against every value

My best idea is to keep the current syntax unchanged. If multiple values are found, a runner should simply apply all child steps to all values.

{
  "_links": {
    "friend": [ 
      { "href": "http://example.com/Fred" }, 
      { "href": "http://example.com/Barney" } 
    ]
  }
}

The following scenario would dereference both /Fred and /Barney and check the status code.

# Will dereference
With Link "friend" {
    Expect Status 200
}

Provide minimal instructions

We should provide some instructions on developing and using the DSL in the readme as soon as we can

Variable suggestion should come with brackets

When IDE module currently provides suggestion for a variable reference, it produces an invalid document:

On the screenshot above it should be

-variable VARIABLE
+[variable] VARIABLE

So that the inserted text is enclosed in square brackets, similarly to how the STRING terminal is proposed.

Remove src-gen

I'd expect generated code safety to .gitignore. This person seems to agree

Support userdata pass-through into JSON

Intro

To make it easy to try out different variations in the DSL, without having to do the roundtrip of [adjusting grammar, regenerating language infrastructure, adjusting generator], it should be possible to write text blocks in the DSL, that are passed-through as-is into the generated JSON.

This is meant to be used during the development of the DSL itself, in order to try out concepts that are not yet part of the proper grammar.

Markup for the text blocks (start/end): ++
The text foo would be written in the DSL as ++foo++

DSL snippet

With Class "http://schema.org/Person" {    
    ++,"foo": "bar", "baz": {"p": "v"}++        
}

Generated JSON

This should translate to

{
	"steps": [
		{
			"type": "Class",
			"classId": "http://schema.org/Person",
			"foo": "bar", "baz": {"p": "v"}
		}
	]
}

Using files as request body

Expanding on #32, it should be possible to use a file as request body. For example, sending a pdf from filesystem

Invoke {
  // request
  Header Content-Type "application/pdf"
  
  >>> @test/sample-docs/abstract.pdf
} => {
  // response
}

Inspired by this feature of vuepress (except I reversed the <<< because I find it pointing in the wrong direction 😉)

Case-sensitive or not?

In all my snippets I have been pretty much assuming the syntax to be case-sensitive. For example Expect Property is good but expect property is not.

Should we keep it that way? A more relaxed DSL would be better maybe?

Deploy sample web editor

• Build a Tomcat docker image running app.hypermedia.testing.dsl.web-1.0.0-SNAPSHOT.war
• Publish at https://editor.testing.hypermedia.app

CLI to recursively compile all sources

Given a directory, the CLI should descend into subdirectories

Add code coverage & codecov

https://github.com/codecov/example-java-maven