koodaamo / httpreverse

Reverse-engineer and use legacy HTTP APIs

httpreverse

Tool to help reverse-engineer legacy HTTP APIs.

Rationale and scope

This package was born of a need to be able to use multiple different kinds of existing, undocumented legacy HTTP APIs not following any kind of consistent, well planned design. It was also thought helpful to be able to add a little bit of documentation about what those legacy APIs do.

This package is NOT meant for defining new APIs. Use e.g. Swagger for that.

Why then not just use Swagger or some other such tool? They are really meant for creating new APIs from scratch and as such cater to a bit different use case. For example they tend to be geared toward the verbose. When reverse-engineering and documenting existing APIs, all the details are not that important. We just need to make it easy to use the APIs and be able to add an explanation of what they do, rather than documenting everything.

The examples hopefully clarify the difference and some of the benefits of this package.

Note that this package does NOT make HTTP requests using some client library. That is up to you; use something from the Python standard library, or the 'requests' package, or something asynchronous, whatever.

API specification examples ------------------------

Note: these examples are illustrative. For working examples, see the tests.

Simple example

An example API definition in YAML that specifies two operations for querying single and double rooms reservations, respectively:

label: Hotel API
description: An API to check room reservations

operations:

  list-singlerooms:
    label: List single room reservations
    description: List all reserved single rooms

    request:
      method: GET
      path: /hotel/reservations
      params:
        size: single

    response:
      type: application/json
      parser: hotelapi.util:parseresponse

  list-doublerooms:
    label: List double room reservations
    description: List all reserved double rooms

    request:
      method: GET
      path: /hotel/reservations
      params:
        size: double

    response:
      type: application/json
      parser: hotelapi.parseresponse

This is similar to how many specification syntaxes express HTTP APIs. Clear, but often lots of boilerplate and repetition. Parses into a plain dict using PyYaml as-is. Now let's see how to save some effort.

Using Jinja templating for API spec expansion

The API document can be expanded using Jinja2 templating. Using our room reservation example, we could generate an API operation for each room size variation:

operations:

  {% for size in sizes %}

  list-{{size}}-rooms:
    label: List {{size}} room reservations
    description: List all reserved {{size}} rooms
      request:
        method: GET
        path: /hotel/reservations
        params:
          size: {{size}}

  {% endfor %}

Two different API operations would be generated, such as with this code, assuming the api spec has been read into a string variable called 'yamlsource':

>>> from httpreverse import expand_jinja >>> expanded = expand_jinja(yamlsource, context={"sizes":["single", "double"]}) >>>

For blunt copying of parts of the YAML document to another place, the standard YAML anchor/alias mechanism can of course be used as well.

Templated request specifications

Besides Jinja templating, a custom templating mechanism is provided for request and response specification convenience. Here's an example with a roomapi request/response template that is used to move repetitive request and response specifications into a common template, referred to from the actual specs:

label: Hotel API
description: An API to check room reservations

templates:

  roomapi:
    request:
      method: GET
      path: /hotel/reservations
    response:
      type: application/json
      parser: hotelapi.parseresponse

operations:

  list-singlerooms:
    label: List single room reservations
    description: List all reserved single rooms
    template: roomapi
    request:
      params:
        size: single

  list-doublerooms:
    label: List double room reservations
    description: List all reserved double rooms
    template: roomapi
    request:
      params:
        size: double

Here's how to apply the request/response template in Python:

>>> from httpreverse import apply_template >>> api = yaml.load(yamlsource) >>> templates = api["templates"] >>> operation = api["operations"]["list-doublerooms"] >>> applied = apply_template(operation, templates) >>>

Simple parametrization

The API definitions can also be parametrized for convenient run-time use. The parametrization function accepts an optional context argument that is simply a dictionary that is used to assign values to all the named parameters found in the operations. Parameters are prefixed with the dollar sign ('$'). So it would be possible to also specify a single dynamically invoked operation for listing the rooms:

operations:

  list-rooms:
    label: List room reservations
    description: List reserved rooms
    template: roomapi
    request:
      params:
        size: $size

By passing either {"size":"single"} or {"size": "double"} as context, room size values would then be assigned:

>>> from httpreverse import parametrize >>> api = yaml.load(yamlsource) >>> operation = api["operations"]["list-rooms"] >>> parametrized = parametrize(operation, context={"size":single}) >>>

More complex parametrizations are possible using the same simple mechanism:

operations:

  add-reservation:
    label: Add reservation
    description: Add a room reservation
    template: roomapi
    request:
      method: POST
      body:
        value: {"size": $roomsize, "customers": $customers}
        type: application/json

The context would then have to include both the room size and occupants, ie. {"roomsize":"double", "customers":["John Doe", "Jane Doe"]}.

Consult the YAML documentation for more on what kind of data structures are possible to express.

When a type + value is given for a parameter or body (as above), the value is automatically marshaled to the given type (json in above example). If a parameter or body is given directly (no type+value syntax), a default must be given thus:

defaults:

structured_param_type: json structured_body_type: json

The above API snippet would specify that whenever a structured parameter or body value is encountered (such as a container or mapping), it will be marshalled to json. Simple values (strings, numbers etc) are used as-is.

Request generator and response parser loading

There are two convenience functions, _load_generator for loading the request generator and _load_parser for loading the response parser:

>>> from httpreverse import _load_parser >>> api = yaml.load(yamlsource) >>> parser = _load_parser(api["list-rooms"]) >>>

Recommended API operations spec generation and use

Typically, when using httpreverse to e.g. make http requests using whatever http client you have, you might want to first run just the Jinja expansion first and parse the resulting YAML string. Then, apply the request/response templates for the operations you expect to be using (or maybe all of them). Keep a copy of the the result. Finally, for each HTTP request, just parametrize the API operation being used, marshal parameters and body and fire away!

httpreverse's Issues

support indicating "required" values

When an API is wrapped, it can be useful to be able to provide a list of variables that the user or higher-level app should provide at minimum for the call to work. Sort of "required" values that the API spec itself cannot itself provide.

There could be one common required key or one for params and one for the body. Perhaps one common setting would suffice though.

define repeat operation

Define how to run an operation repeatedly, possibly with context taken from an iterator for each run.

handle parameter marshaling & change body marshaling

YAML does a lot of basic type (de)marshaling for us. However this is not enough for complex data structures that need to be marshaled as JSON or XML.

The marshaling should take part after parametrization (of body and/or params) and added as an optional included step in the parametrization.

HTTP body marshaling indication should be changed to follow the same syntax as well.

Suggested syntax using type and valuekeys:

params:
  myparamname:
    type: JSON
    value: [{"a": 1}]

The implementation should allow for specifying default encodings for both body and params separately, thus allowing for omitting the extra type and value structure. In such a case marshaling nevertheless changes the body into the canonical form (having type and value)

support response parser specification

Make it possible to specify a response parser thus:

<opname>:
  response:
    parser: pkg.module:parsercallable 

Only provide support for importing and returning the callable. It's the job of the application to hook the parser into the http client, for example.

support request generator specification

Make it possible to specify a request generator thus:

<opname>:
  request:
    generator: pkg.module:generatorcallable 

Provide support for importing and returning the callable, optionally running it destructively (the callable produces the request on its own) or constructively (the callable only amends the request).

It's the job of the application to hook the generator into the http client, for example.

handle arbitrary data structure parametrization

Implement a recursive traverser to parametrize arbitrary request parameter or body structures.

support parameter subelement accessors

Make more advanced parametrization possible by using a slash to indicate a subelement of a parameter data structure, thus: $parameter/subelement, or $parameter/0. The first case would return a subelement of a mapping, by key, and the latter of a sequence, by index (starting from zero).

For implementation, it may be sufficient to just split the parameter at slashes and iterate over them and the parameter data structure.

support conversion of request body to JSON/XML

When request body is given, convert as per content type setting. Use stdlib json support, and xmltodict for Python-to-XML conversion.

support specifying a static context within the API spec

Make it possible to specify and use the context thus (at YAML root level).

Specification:

contexts:
  hotelapi_test1:
    size: double
    occupants: {["John Doe", "Jane Doe"]}
    ...

Use:

operations:
  list-reservations:
    context: hotelapi_test1
    ...

Note that the statically defined context is NOT available to Jinja template expansion.

define composite operation

Define how to run multiple different API operations in one shot, with the possibility of using values from the result of a previous run as input context for the next.