3scale / apicast

3scale API Gateway

APIcast

APIcast is an API gateway built on top of NGINX. It is part of the Red Hat 3scale API Management Platform.

• Getting started
• Features
• Development
• Documentation
• License

Getting started

master branch is not recommended for production use. For the latest release, go to the Releases page

Docker

You need to specify an ACCESS_TOKEN, that you can get from the 3scale admin portal, and also your ADMIN_PORTAL_DOMAIN. If you are using SaaS, it is YOUR_ACCOUNT-admin.3scale.net.

docker run --name apicast --rm -p 8080:8080 -e THREESCALE_PORTAL_ENDPOINT=https://ACCESS_TOKEN@ADMIN_PORTAL_DOMAIN quay.io/3scale/apicast:master

You can use a JSON configuration file instead:

docker run --name apicast --rm -p 8080:8080 -v $(pwd)/config.json:/opt/app/config.json:ro -e THREESCALE_CONFIG_FILE=/opt/app/config.json quay.io/3scale/apicast:master

In this example config.json is located in the same directory where the docker command is executed, and it is mounted as a volume at /opt/app/config.json. :ro indicates that the volume will be read-only.

The JSON file needs to follow the schema, see an example file with the fields that are used by APIcast.

Openshift

You need to create a secret with your ACCESS_TOKEN and your ADMIN_PORTAL_DOMAIN:

oc create secret generic apicast-configuration-url-secret \
   --from-literal=password=https://ACCESS_TOKEN@ADMIN_PORTAL_DOMAIN \
   --type=kubernetes.io/basic-auth
oc new-app -f https://raw.githubusercontent.com/3scale/apicast/master/openshift/apicast-template.yml

Features

• Performance: it is fast because it's built on top of NGINX and uses LuaJIT.
• Scalability: APIcast is stateless, so it scales horizontally.
• Request transformation: allows to modify the headers, the path and the arguments of a request.
• Rate-limit: can apply limits based on a header, JWT claims, the IP of the request and many more.
• Modular and extensible: thanks to the APIcast policies framework.
• Monitoring with Prometheus.
• NGINX instrumentation using OpenTelemetry. Works with Jaeger.
• Can be deployed in Openshift.
• Integrates with IDPs like Keycloak to provide authentication based on OIDC.

Development

Using Docker you just need to run:

make development

That will create a Docker container and run bash inside it. The project's source code will be available in the container and sync'ed with your local apicast directory, so you can edit files in your preferred environment and still be able to run whatever you need inside the Docker container.

To install the dependencies inside the container run:

make dependencies

To run the unit tests inside the container:

make busted

To run the integration tests inside the container:

make prove

To learn about the other available make targets:

make help

APIcast uses:

• OpenResty: a platform that includes NGINX, LuaJIT and Lua modules.
• busted: for the unit tests.
• Test::Nginx: for the integration tests.

More info can be found in the development specific doc.

Documentation

• Configuration parameters
• Get started with Lua and OpenResty
• Learn about policies and write your own
• Available Prometheus metrics
• Management API
• Contribute

License

Apache 2.0

apicast's Issues

Change license to Apache2.0

Based in legal input, Apache2.0 is slightly better than MIT, and while all the contributions to the repo are 3scalers (and hence copyright is owned) we can easily switch the license.

So, I propose we do it now - before we accept any 3rd party contributions that complicate the switch.

Add a "reload from 3scale" endpoint to the Management API

Currently the Management API can update the configuration with a JSON configuration file passed as body in POST /config or PUT /config.

It would be good to have a method that would invoke downloading the configuration from 3scale and applying it to the running gateway.

/dev/std* not accessible by non-root on docker < 1.9.0

nginx fails to read the conf file with access_log set to /dev/stdout

Steps To Reproduce

1. install centos 6
2. install docker (1.7 is the latest supported by the old kernel)
3. create a container using :latest

Current Result

nginx starts but does not forward any requests to the private URL

Expected Result

nginx forwards requests

Additional Information

• command to run docker: docker run --detach --publish=127.0.0.1:319:8080 --name=3scale --env=THREESCALE_PROVIDER_KEY=$THREESCALE_PROVIDER_KEY --env=THREESCALE_ADMIN_URL=https://tripgo-admin.3scale.net quay.io/3scale/gateway
• relevant nginx output: 2016/09/13 05:48:17 [emerg] 7#0: open() "/dev/stdout" failed (13: Permission denied)

Workaround

maybe this one in /entrypoint.sh: moby/moby#6880 (comment) then point nginx to log to /tmp/logpipe

[v2] http.d/resolver.conf is not included correctly by nginx.conf

related to #48

Create tests for the gateway in OAuth mode

We don't have the necessary test coverage of APIcast in OAuth mode (see #6593)

We should have integration tests with real IAM
We could consider something like https://github.com/doorkeeper-gem/doorkeeper
or http://www.ory.am/hydra

[v2] Openresty lua is failing to validate SSL certificates

2016/09/06 23:32:53 [error] 23#23: *1 lua ssl certificate verify error: (20: unable to get local issuer certificate), client: 172.17.0.4, server: _, request: "GET / HTTP/1.1", host: "testing.com"
2016/09/06 23:32:53 [error] 23#23: *1 lua entry thread aborted: runtime error: /opt/app/src/configuration.lua:186: missing configuration
stack traceback:
coroutine 0:
    [C]: in function 'error'
    /opt/app/src/configuration.lua:186: in function 'boot'
    access_by_lua(apicast.conf:52):5: in function <access_by_lua(apicast.conf:52):1>, client: 172.17.0.4, server: _, request: "GET / HTTP/1.1", host: "testing.com"

Rename to APIcast

Let's rename this repository to APIcast.

• rename this repository to apicast
• rename quay.io/3scale/gateway to quay.io/3scale/apicast ?

@vramosp do you agree?

Rename THREESCALE_ENDPOINT in OpenShift template to THREESCALE_ADMIN_URL

Rename THREESCALE_ENDPOINT in OpenShift template to THREESCALE_ADMIN_URL as that better described what it is, is more consistent with other references we have - and we tend to reserve the "ENDPOINT" term for our APIs, not for a url of the UI.

Please do this asap and let me know, as the Tutorial text and screenshots all need updating.

use opm instead of luarocks

Luarocks is trouble. It needs to be installed with proper settings. Passed proper flags.

We can use soon to be released openresty package manager opm.

Expose APICAST_RELOAD_CONFIG as OpenShift template parameter

APICAST_RELOAD_CONFIG can control reloading of the configuration, but it is not exposed as OpenShift parameter.

Steps To Reproduce

1. oc new-app -f ... -p APICAST_RELOAD_CONFIG=1

Current Result

Not reloading.

Expected Result

Reloading config on every request.

Info for supported config

There are a number of questions that are asked frequently about apicast, we can help answer them proactively with a FAQ in the repo. I would put in docs folder and also link to it from the readme.

Deployment options

• Customer direct deploy: The customer must deploy supported version of NGINX/OpenResty, and then clone a GA tagged release of the repo. [doc]
• Docker: Customer runs Docker Image from the the official Red Hat Docker Registry, using a supported version of Docker [doc]
• OpenShift: Customer deploys on a supported version of OpenShift a released and unmodified OpenShift template for APIcast, which will run the official Docker images mentioned above

What OS is this project tested on (not implying support)

What version of OpenResty is recommended and tested against?

Failover. If the 3scale Service Management API is unreachable, does the API Gateway continue to work, how?

Add a description and a License file

Add a short description of this repo
Add MIT License file

Port in api_backend defaults to 80 even though schema is HTTPS

service.api_backend normally (when created with UI) includes the port (i.e. https://echo-api.3scale.net:443), however, when the port is not specified, it will not be set here, and as a result the balancer will fall back to the port 80, even if the schema is https. This causes a 502 Bad Gateway error with the following in the error log:

2016/12/22 11:12:17 [error] 52363#0: *5 SSL_do_handshake() failed (SSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol) while SSL handshaking to upstream, client: 127.0.0.1, server: _, request: "GET /test?user_key=asdf HTTP/1.1", upstream: "https://52.54.57.126:80/test?user_key=asdf", host: "echo-api.3scale.net"

To reproduce, take a working configuration with valid credentials and set api_backend to https://echo-api.3scale.net:

{
  "id": 1234567890987,
  "provider_key": "provider-key",
  "services": [
    {
      "id": 654321,
      "backend_version": "1",
      "backend_authentication_type": "service_token",
      "backend_authentication_value": "service-token-1234567890",
      "proxy": {
        "api_backend": "https://echo-api.3scale.net",
        "auth_app_key": "app_key",
        "auth_app_id": "app_id",

For 3scale backend this is solved this way.

For API backend, we could follow the same way.

Or, alternatively, we could change this line to:

local peers = balancer:peers(ngx.ctx[ngx.var.proxy_host], ngx.var.proxy_port)

proxy_port will be set to the protocol's default port. Custom ports should still work, because of this precedence, but maybe I'm missing something.

[Meta] Enable squash merging on repo

Can the repo administrator enable squash merging on repo as per this article? Not sure I like how #150 is currently looking :)

Redundant location block "client_secret_matches"

In apicast.conf there is a client_secret_matches location block that is not currently used or needed:

https://github.com/3scale/apicast/blob/c306f49655caa6cae98479ff53fad41afd1b57ea/apicast/conf.d/apicast.conf#L121-L127

Handle API errors when getting the configuration from 3scale

Currently when making the call to /admin/api/nginx/spec.json the API can return: {"error":"Access denied"} in case the access token is not correct, or {"status":"Not found"} if the URL is wrong. This body is used to set provider.contents.

download() and curl() in configuration.lua should check for the status code of the response and if it's not 200 probably set the provider.contents to null.

Ability to only download relevant service configurations only based on

Currently if you add in the provider key and admin portal, the nginx configuration will be downloaded for ALL services.

Given that we now have admin rights by service which allow us to cherry pick which configurations get downloaded based on the user rights, would it make sense to update the docker image to use access_tokens or provider_key to download the relevant configs.

Create example OAuth2 integration

We need an easy way to demo OAuth2 integration of APIcast. Not only demo, but to try in a browser after making changes to verify all works.

Steps

1. create JSON configuration using example OAuth server
2. create html page using that configuration
3. write step by step guide to try whole flow

Expected Result

Everyone having the necessary software installed should be able to go through full OAuth flow on their local machine.

Use JSON response content type for GET /config

Currently the GET /config call to the management API returns a serialized JSON with text/plain content type. It would be better to return it as application/json.

Bake functionality into V2 that deals effectively with 3scale being unreachable

A production ready version of this:
https://support.3scale.net/codehub/use-prebuilt-cache

Should check the cache for dynamically added keys/tokens - rather than hard code them in whitelist.lua

Should avoid the long in-band wait associated with a 503 with the current solution. Background job?

Avoid dependency on REDIS

I understand that with OAuth additions we have made that we have introduced a (optional) dependency on REDIS in apicast?

Can someone confirm/deny/explain it a bit more and the need.

I think it would be great it we could find other ways to solve the same need and keep the simple apicast deployment as simple as possible.

Fix OAuth

[provide a description of the issue]
I am led to understand that OAuth is not working correctly in apicast at the moment.

Can anyone verify that?

Version

Steps To Reproduce

1. [step 1 (json configuration file, if applies)]
2. [step 2 (curl commands to reproduce, if applies)]
3. [step 3]

Current Result

Expected Result

Additional Information

• [Gist with minimal reproducible configuration, see guidelines for contributing for details]
• [Gist with nginx log output]

Make placeholder in template match convention in Support site

Make the THREESCALE_ADMIN_URL placeholder that appears in the UI (in the new app form from the template) consistent with the support site "standard" (i.e https://MYDOMAIN-admin.3scale.net/).

Performance hit in module.lua because of unnecessary calls to require()

I'm writing a module for Apicast v2. I used stapxx to profile it, and realized that there was something wrong. Around 75-80% of the time was consumed on the method prequire() of module.lua.

My module defines rewrite(), access(), etc. instead of call(). The problem is that in the load() method of module.lua there's this:

  local files = {
    [ name .. '.' .. phase ] = 'call', -- like apicast.init exposing .call
    [ name ] = phase -- like apicast exposing .init
  }

  for file, method in pairs(files) do
    local ok, ret = prequire(file)

and prequire() only caches a result when it's successful. In other words, as I haven't defined call() in my module, prequire() fails, and it does not cache that it failed, so it retries in every single request. This is a huge performance hit. When I fixed this, I noticed that the time spent prequire() went down from 75-80% to almost none.

[v2] THREESCALE_CONFIG_FILE is used even when is the env var is empty

setting THREESCALE_CONFIG_FILE="" causes the container to fail due bad env checks.

add option to disable lua code cache and enable code/configuration reloading

We need to have an option to disable lua code cache and enable code reloading.
That will also make configuration reloading work. Later we might need two different options.
One for code and second for configuration.

Version

[provide output of the nginx -V or openresty -V command from openshift/local terminal]
[provide timestamp of the docker image from docker inspect --format='{{.Created}}' quay.io/3scale/apicast:v2 ]

Steps To Reproduce

1. bin/apicast -C
2. do a request
3. change code and configuration
4. see configuration and code reloaded

Current Result

does not reload the configuration or code

Expected Result

should reload the configuration and code from filesystem/api

Additional Information

the -C option is just suggestion. can be something else.

Track gateway usage

It seems that with latest releases we've lost the ability of using the user agent header to track source of calls to backend and we'd like to bring it back.

In addition, it'd be useful to also being able to track:

• Deployment method
• Modules used

Is v2 backwards compatible?

We are using quay.io/3scale/gateway:latest at the moment, which is from 4 months ago. I'd like to use the v2 tag (seems to be quite a lot of development). Anything I should be worried about or do before upgrading? Thanks.

Improve logging

Would be nice to get some feedback on how we should improve logging.

From debugging and troubleshooting to logging request times.

Merge openshift branch into master

@jmprusi . Any reason to not merge the https://github.com/3scale/docker-gateway/tree/openshift branch into Master?

NOTE: If you do so, please do early on Tuesday and let me know, as I will need to change the tutorial to refer to master branch content.

Support path based routing between services

The docker gateway works great with one service.

If we have multiple service support, the gateway always routes to call to first server block on nginx conf file and then to upstream backend.

Below is the file format 3scale generates

http {

  upstream backend_abc {
  ...
}
  server {
    server_name $hostname;

    location /{
    set $service_id 2555555;
    ...
    ...
    }

  }

  upstream backend_def {
  ...
}
  server {
    server_name $hostname;

    location /{
    set $service_id 2444444;
    ...
    ...
    }

  }

  upstream backend_xyz {
  ...
}
  server {
    server_name $hostname;

    location /{
    set $service_id 233333;
    ...
    ...
    }

  }
}

Since the hostname always remain the same, the call always get routed to service_id 2555555 of the first server block and ultimately to upstream backend_abc.

Let me know if you need more information.

Add APIcast 2.0 OpenShift template to ansible-openshift installer

Add the APIcast 2.0 OpenShift template to the Ansible-Openshift installer, so when the customer installs OpenShift, the APIcast templates gets automatically installed.

The GitHub repository is this:
https://github.com/openshift/openshift-ansible

Process
From @sdodson: We just sync the file from your upstream repo into the installer repo. Your teams can either open pull requests against openshift-ansible or email me and let me know the new tag to
include and we'll update it then.

[v2] Missing error message when resolver is not set

We should output the error message from this part:
https://github.com/3scale/docker-gateway/blob/v2/src/configuration.lua#L232-L241

We should get a more descriptive message like this:

2016/09/07 22:24:40 [error] 24#24: *1 lua entry thread aborted: runtime error: 
/opt/app/src/configuration.lua:243: no resolver defined to resolve "3scaletesting-admin.3scale.net"

instead of:
2016/09/07 22:16:22 [error] 23#23: *1 lua entry thread aborted: runtime error: /opt/app/src/configuration.lua:193: missing configuration

Change response code returned to client when they are over limits

Current Result

1. Client sends bad key --> got 403 Authentication failed
2. Client sends valid key, but is over rate limits --> got 403 Authentication failed.

This makes errors difficult to debug, as the two different causes are not discriminated.

Requested Result

Return 429 "Too many requests"

There maybe documentation pages that will required modification to explain this code has changed in this (insert version of apicast that implements this here) or apicast

Additional Information

I believe that this is when Service Management API returns a 409 Conflict with authorizing.

We do not at the moment contemplate the need to customize this code and error message via Admin Portal - so a fixed code and message are all that's requested in this issue.

Use balancer_by_lua to use upstream with keepalive

Using proxy_pass $var will not use upstream and keep-alive connection.

Either we have to use cosockets and lua-resty-http to stream API request and responses, or use balancer_by_lua in upstream.

Don't use provider_key (currently) in downloaded config

Currently the downloaded config from MT API includes a provider_key in the JSON, and this is used by apicast for debugging calls/headers.

To reduce possible leakage of the global provider_key the proposal is that this key is not used (there will be a separate issue in System dependant on this one, to stop sending it) and the provider_key or access_token that is used in the deploy of the gateway (the one used to fetch the config from MT API in the first place) is used in it's place.

I will assign to the RHAMP 2.0 release milestone, as the MT part of this is breaking the API (JSON file format) - and we want to reduce the legacy versions out there that will need to be updated to this newer version of apicast.

Improve path-based routing

This PR introduced routing the requests to different services based on the path, which allows having the same Public Base URL in multiple services.

This works, but the way it works is not very efficient. In the worst case it will iterate over all services and all mapping rules, and do regex matching for each one of them. After getting the service id, the results of regex matching are discarded, and it will be iterated again for all the rules of the service.

I think this can be improved.

As the 1st step it's probably worth doing a benchmark to understand better how serious the problem is.

Then, we can think about how this can be improved. I am thinking of two ways:

1. Use the find_service_cascade function as it is now, but store the results of usage and matched_patterns to reuse them later and avoiding regex matching again.

2. On configuration load build some kind of index based on the mapping rules patterns (across all services), which will be used for service id lookup on incoming requests.

Add TROUBLESHOOTING

While getting apicast running and integrated with 3scale API Management Platform a number of issues can arise that are not familiar or easy to debug for new users. We wish to make this start-up and integration process as easy and painless as possible for new users.

To do so, we would like to add to this upstream project a document where possible sources of problems are documented as a resource for people to use to help them get going.

I suggest adding in doc folder and linking from the README.

This would be an upstream document, focused on apicast alone, based on learnings of 3scale over the years captured in this product document.
https://support.3scale.net/docs/api-devops/troubleshooting

When resolving issues, or finding new causes, anyone can PR this document to add cases that may help other users int he community solve their problems.

Service ids should be strings, not integers

The schema for importing data states that service ids are integers, but they are opaque strings for all intents and purposes within Apicast, and that is bad because we end up converting them everywhere.

There are two ways to fix this. One is changing this line in the configuration schema so it says string instead of integer, and changing the configuration generation as needed.

The other way is using this schema but making sure it is loaded with the converted service id at initialization time.

I think this should be considered before making the schema final.

Basic description what APIcast is to the top of the readme

Add a basic description what APIcast is to the top of the readme so visitors know what this repository is about.

No Mapping Rule matched - with multiple Services

[provide a description of the issue]
I have an issue, first seen in Openshift, where endpoints in the 2nd service I declare aren't picked up - despite setting the host header.
It's not Openshift specific, I've setup a 3scale AMI which you can access and API requests routed through it see the same issue.
ssh -i ./_3scale-us-east.pem [email protected] (pem is available to the 3scale dev team by emailing me - actually it's on an earlier email)
this restart commend give you the location of the configs:

sudo /opt/openresty/nginx/sbin/nginx -p /opt/openresty/nginx/ -c /opt/openresty/nginx/conf/nginx.conf -s reload

I've uploaded the auto-gen'd config files - zero modifications.

NOTE: I have added this to the Integration page in 3scale as suggested:
Public Base URL http://ec2-107-22-18-40.compute-1.amazonaws.com:80

See JSON
https://cathay-poc-os-admin.3scale.net/admin/api/nginx/spec.json

By contrast, the other call in the other service (flights) works:

https://cathay-poc-os-admin.3scale.net/admin/api_docs/services/55457/preview

Steps To Reproduce

1. go here and Try it Out:
   https://cathay-poc-os-admin.3scale.net/admin/api_docs/services/55458/preview
   keep the default host and enter the baggage api key. Reports no mapping where clearly it is mapped.

Current Result

No Mapping Rule matched
I get this in the Nginx console on server startup
nginx: [warn] conflicting server name "ec2-107-22-18-40.compute-1.amazonaws.com" on 0.0.0.0:80, ignored

Expected Result

Same is this returns (remember it's a free heroku so first call every hour is slow):
http://two-methods-prints-headers.herokuapp.com/baggage/intl/items

Additional Information

Configs are unchanged - you can get them from 3scale.

access_token is not taken from the Authorization header in OAuth flow

Currently the access_token for the calls using OAuth flow is taken:

1. from the query parameters (for GET) or body (for POST and other methods), in case the service is configured in 3scale to use headers for credentials locations configured.

2. from access_token header

While 1 is OK, 2 is not compliant with the RFC 6750, according to which the access_token should be taken from the Authorization header:

Authorization: Bearer xxyyzz

This should be fixed.

Another question is whether APIcast should actually check for both locations (thus disregarding the setting in the 3scale configuration) so that it can supports any OAuth client.

Ability to slot Nginx Custom Configuration in when building Nginx on Openshift

Close to 100% of large complex on-boardings require some degree of customization of their Nginx configuration. Typically, changes are required around the server and location blocks inside Nginx.conf. Lua is less subject to change, though we frequently need to modify that as well. Anything from
-using URL Path to dictate which Service is used to
-adding custom headers to
-adding custom paths to IDP Servers to
-JWT validation etc.

Nginx Openshift needs a back door to slot in custom configuration.

Document APIcast parameters

There is a number of parameters that can be set for APIcast that modify its behavior, normally they are set via environment variables (e.g. AUTO_UPDATE_INTERVAL , APICAST_SERVICES, APICAST_MISSING_CONFIGURATION etc.)

All these parameters need to be documented: the name, what they do, what are the allowed values/format.

I think this needs to be a separate reference document.

Add documentation to make getting started with apicast easier

We wish to make it easier for people new to apicast (who maybe also new to nginx, openresty and lua) to get started, use it and add their customizations.

Here is a list of customization examples that we would like to have explained briefly how to do.

• Adding a new server with own server name (ssl, nginx config)
• Adding some nginx http block configuration (like resolver)
• Adding custom lua config to override some existing functions
• Adding a custom module (e.g. XC)

These could be short documents on each one, in the corresponding examples sub-folder or small sections in an overall "customizations" document in examples root, to accompany any code examples for the same in the examples folder

Add documentation to customize APIcast

With the new modular architecture, it'd be good to create some documentation which explains how to add custom behavior to the gateway by creating a module.

cc @andrewdavidmackenzie

Mapping rules matching for GET vs POST

Mapping rules defined in 3scale API Manager can have additional query parameters which are taken into account for mapping:

For GET requests it's very straightforward – the call:

curl -XGET "http://localhost:8080/test?key=hello&another=parameter"

will match the method get as expected.

But for POST the behavior is a bit strange. The reason is that for "non-GET" requests the "query parameters" are taken from the body.

So, this request will not match:

curl -XPOST "http://localhost:8080/test?key=hello" -d "another=parameter" 

while this one will match the pattern /test?key={value} (post method):

curl -XPOST "http://localhost:8080/test" -d "key=hello&another=parameter" 

I am trying to understand if this is by design, or this behavior should be change.
I'd say the current behavior is not intuitive, and would only be partly useful if application/x-www-form-urlencoded content type is used, as it will not work for JSON payloads for example.

I believe only actual query string needs to be taken into account for mapping rules matching.

[v2] 404 for all hosts unless "localhost"

Any request sent to the gateway will return 404 unless the Host header is set to "localhost" even when the service has been configured with "testing.com".

Provide a generic plugin to avoid code duplication across custom plugins

To add a custom plugin, we only need to define a lua module that implements a few methods like rewrite(), access(), etc. that are called in the appropriate nginx phases.

The problem is that as it is defined right now, the plugin system provided by Apicast forces custom plugins to define all those methods, even if they do not have anything to do with the functionality that the custom plugin adds. We should implement some changes so methods that are not likely to change from plugin to plugin do not need to be duplicated across all the custom plugins.

Support https in the OpenShift apicast template

There should be an OpenShift template for deploying the nginx gateway on OpenShift with a service supporting https. The key and certificate for the ssl config come from the volume of an OpenShift secret.