zalando / skipper

An HTTP router and reverse proxy for service composition, including use cases like Kubernetes Ingress

Skipper

Skipper is an HTTP router and reverse proxy for service composition. It's designed to handle >300k HTTP route definitions with detailed lookup conditions, and flexible augmentation of the request flow with filters. It can be used out of the box or extended with custom lookup, filter logic and configuration sources.

Main features:

An overview of deployments and data-clients shows some use cases to run skipper.

Skipper

• identifies routes based on the requests' properties, such as path, method, host and headers
• allows modification of the requests and responses with filters that are independently configured for each route
• simultaneously streams incoming requests and backend responses
• optionally acts as a final endpoint (shunt), e.g. as a static file server or a mock backend for diagnostics
• updates routing rules without downtime, while supporting multiple types of data sources — including etcd, Kubernetes Ingress, static files, route string and custom configuration sources
• can serve as a Kubernetes Ingress controller without reloads. You can use it in combination with a controller that will route public traffic to your skipper fleet; see AWS example
• shipped with
  • eskip: a descriptive configuration language designed for routing rules
  • routesrv: proxy to omit kube-apiserver overload leveraging Etag header to reduce amount of CPU used in your skipper data plane
  • webhook: Kubernetes validation webhook to make sure your manifests are deployed safely

Skipper provides a default executable command with a few built-in filters. However, its primary use case is to be extended with custom filters, predicates or data sources. Go here for additional documentation.

A few examples for extending Skipper:

• Example proxy with custom filter https://github.com/szuecs/skipper-example-proxy
• Image server https://github.com/zalando-stups/skrop
• Plugins repository https://github.com/skipper-plugins/, plugin docs

Getting Started

Prerequisites/Requirements

In order to build and run Skipper, only the latest version of Go needs to be installed. Skipper can use Innkeeper or Etcd as data sources for routes, or for the simplest cases, a local configuration file. See more details in the documentation: https://pkg.go.dev/github.com/zalando/skipper

Installation

From Binary

Download binary tgz from https://github.com/zalando/skipper/releases/latest

Example, assumes that you have $GOBIN set to a directory that exists and is in your $PATH:

% curl -LO https://github.com/zalando/skipper/releases/download/v0.14.8/skipper-v0.14.8-linux-amd64.tar.gz
% tar xzf skipper-v0.14.8-linux-amd64.tar.gz
% mv skipper-v0.14.8-linux-amd64/* $GOBIN/
% skipper -version
Skipper version v0.14.8 (commit: 95057948, runtime: go1.19.1)

From Source

% git clone https://github.com/zalando/skipper.git
% make
% ./bin/skipper -version
Skipper version v0.14.8 (commit: 95057948, runtime: go1.19.3)

Running

Create a file with a route:

echo 'hello: Path("/hello") -> "https://www.example.org"' > example.eskip

Optionally, verify the file's syntax:

eskip check example.eskip

If no errors are detected nothing is logged, else a descriptive error is logged.

Start Skipper and make an HTTP request:

skipper -routes-file example.eskip &
curl localhost:9090/hello

Docker

To run the latest Docker container:

docker run registry.opensource.zalan.do/teapot/skipper:latest

To run eskip you first mount the .eskip file, into the container, and run the command

docker run \
  -v $(PWD)/doc-docker-intro.eskip:/doc-docker-intro.eskip \
  registry.opensource.zalan.do/teapot/skipper:latest eskip print doc-docker-intro.eskip

To run skipper you first mount the .eskip file, into the container, expose the ports and run the command

docker run -it \
    -v $(PWD)/doc-docker-intro.eskip:/doc-docker-intro.eskip \
    -p 9090:9090 \
    -p 9911:9911 \
    registry.opensource.zalan.do/teapot/skipper:latest skipper -routes-file doc-docker-intro.eskip

Skipper will then be available on http://localhost:9090

Authentication Proxy

Skipper can be used as an authentication proxy, to check incoming requests with Basic auth or an OAuth2 provider or an OpenID Connect provider including audit logging. See the documentation at: https://pkg.go.dev/github.com/zalando/skipper/filters/auth.

Working with the code

Getting the code with the test dependencies (-t switch):

git clone https://github.com/zalando/skipper.git
cd skipper

Build and test all packages:

make deps
make install
make lint
make shortcheck

On Mac the tests may fail because of low max open file limit. Please make sure you have correct limits setup by following these instructions.

Working from IntelliJ / GoLand

To run or debug skipper from IntelliJ IDEA or GoLand, you need to create this configuration:

Parameter	Value
Template	Go Build
Run kind	Directory
Directory	skipper source dir + /cmd/skipper
Working directory	skipper source dir (usually the default)

Kubernetes Ingress

Skipper can be used to run as an Kubernetes Ingress controller. Details with examples of Skipper's capabilities and an overview you will can be found in our ingress-controller deployment docs.

For AWS integration, we provide an ingress controller https://github.com/zalando-incubator/kube-ingress-aws-controller, that manage ALBs or NLBs in front of your skipper deployment. A production example for skipper and a production example for kube-ingress-aws-controller, can be found in our Kubernetes configuration https://github.com/zalando-incubator/kubernetes-on-aws.

• Comparison with other Ingress controllers
• Comparison with service-mesh

Documentation

Skipper's Documentation and Godoc developer documentation, includes information about deployment use cases and detailed information on these topics:

• The Routing Mechanism
• Matching Requests
• Filters - Augmenting Requests and Responses
• Predicates - additional predicates to match a route
• Service Backends
• Route Definitions fetched by dataclients:
  • route string
  • eskip file
  • remote eskip
  • etcd
  • kubernetes
• Circuit Breakers
• Extending It with Custom Predicates, Filters, can be done by building your own proxy, Plugins or Lua Scripts
• Proxy Package
• Logging and Metrics
• Operations guide
• Authentication and Authorization
• Load Shedders
• Rate Limiters
• Opentracing tracers or extend create your own

1 Minute Skipper introduction

The following example shows a skipper routes file in eskip format, that has 3 named routes: baidu, google and yandex.

% cat doc-1min-intro.eskip
baidu:
        Path("/baidu")
        -> setRequestHeader("Host", "www.baidu.com")
        -> setPath("/s")
        -> setQuery("wd", "godoc skipper")
        -> "http://www.baidu.com";
google:
        *
        -> setPath("/search")
        -> setQuery("q", "godoc skipper")
        -> "https://www.google.com";
yandex:
        * && Cookie("yandex", "true")
        -> setPath("/search/")
        -> setQuery("text", "godoc skipper")
        -> tee("http://127.0.0.1:12345/")
        -> "https://yandex.ru";

Matching the route:

• baidu is using Path() matching to differentiate the HTTP requests to select the route.
• google is the default matching with wildcard *
• yandex is the default matching with wildcard * if you have a cookie yandex=true

Request Filters:

• If baidu is selected, skipper sets the Host header, changes the path and sets a query string to the http request to the backend "http://www.baidu.com".
• If google is selected, skipper changes the path and sets a query string to the http request to the backend "https://www.google.com".
• If yandex is selected, skipper changes the path and sets a query string to the http request to the backend "https://yandex.ru". The modified request will be copied to "http://127.0.0.1:12345/"

Run skipper with the routes file doc-1min-intro.eskip shown above

% skipper -routes-file doc-1min-intro.eskip

To test each route you can use curl:

% curl -v localhost:9090/baidu
% curl -v localhost:9090/
% curl -v --cookie "yandex=true" localhost:9090/

To see the shadow traffic request that is made by the tee() filter you can use nc:

[terminal1]% nc -l 12345
[terminal2]% curl -v --cookie "yandex=true" localhost:9090/

3 Minutes Skipper in Kubernetes introduction

This introduction was moved to ingress controller documentation.

For More details, please check out our Kubernetes ingress controller docs, our ingress usage and how to handle common backend problems in Kubernetes.

Packaging support

See https://github.com/zalando/skipper/blob/master/packaging/readme.md

In case you want to implement and link your own modules into your skipper, there is https://github.com/skipper-plugins organization to enable you to do so. In order to explain you the build process with custom Go modules there is https://github.com/skipper-plugins/skipper-tracing-build, that was used to build skipper's opentracing package. We moved the opentracing plugin source into the tracing package, so there is no need to use plugins for this case.

Because Go plugins are not very well supported by Go itself we do not recommend to use plugins, but you can extend skipper and build your own proxy.

Community

User or developer questions can be asked in our public Google Group

We have a slack channel #skipper in gophers.slack.com. Get an invite. If for some reason this link doesn't work, you can find more information about the gophers communities here.

The preferred communication channel is the slack channel, because the google group is a manual process to add members. Feel also free to create an issue, if you dislike chat and post your questions there.

Proposals

We do our proposals open in Skipper's Google drive. If you want to make a proposal feel free to create an issue and if it is a bigger change we will invite you to a document, such that we can work together.

Users

Zalando used this project as shop frontend http router with 350000 routes. We use it as Kubernetes ingress controller in more than 100 production clusters. With every day traffic between 500k and 7M RPS serving 15000 ingress and 3750 RouteGroups at less than ¢5/1M requests. We also run several custom skipper instances that use skipper as library.

Sergio Ballesteros from spotahome said 2018:

We also ran tests with several ingress controllers and skipper gave us the more reliable results. Currently we are running skipper since almost 2 years with like 20K Ingress rules. The fact that skipper is written in go let us understand the code, add features and fix bugs since all of our infra stack is golang.

In the media

Blog posts:

• opensource.com - Try this Kubernetes HTTP router and reverse proxy
• opensource.com - An open source HTTP router to increase your network visibility
• Building our own open source http routing solution: Giving some context about why Skipper was created in the first place.
• Kubernetes in production @ ShopGun
• Hacker News Skipper – An HTTP router and reverse proxy for service composition

Conference/Meetups talks

• LISA 2018 - modern HTTP routing

Version promise

Skipper will update the minor version in case we have either:

• a significant change
• a Go version requirement change (go directive in go.mod change)
• a dependency change that adds or removes a replace directive in go.mod file (requires library users to add or remove the same directive in their go.mod file)
• a change that require attention to users, for example Kubernetes RBAC changes required to deploy https://github.com/zalando/skipper/releases/tag/v0.18.0
• a feature removal like Kubernetes ingress v1beta1 https://github.com/zalando/skipper/releases/tag/v0.15.0
• an API change of a function that is marked experimental example

We expect that skipper library users will use skipper.Run(skipper.Options{}) as main interface that we do not want to break. Besides the Kubernetes v1beta1 removal there was never a change that removed an option. We also do not want to break generic useful packages like net. Sometimes we mark library functions, that we expect to be useful as experimental, because we want to try and learn over time if this is a good API decision or if this limits us.

This promise we hold considering the main, filter, predicate, dataclient, eskip interfaces and generic packages. For other packages, we have more weak promise with backwards compatibility as these are more internal packages. We try to omit breaking changes also in internal packages. If this would mean too much work or impossible to build new functionality as we would like, we will do a breaking change considering strictly semantic versioning rules.

How to update

Every update that changes the minor version (the m in v0.m.p), should be done by +1 only. So v0.N.x to v0.N+1.y and you should read v0.N+1.0 release page to see what can break and what you have to do in order to have no issues while updating.

skipper's Issues

Delete for Innkeeper

Implement the delete operation in eskip for Innkeeper

remove default data client

version output for the skipper and eskip commands

Improve linting

As of current version (08b8adf), skipper needs some linting love: it currently has 278 "problems", mostly style mistakes like proxy/proxy.go:228:6: func cloneUrl should be cloneURL.
I would go through all the source code and make it compliant. Please @lmineiro @aryszka have a look at the current output and share what you think.

Eskip refactor: convert post-yacc processing of predicates into yacc rules.

eskip tool nil pointers when etcd not specified for delete command

handle oauth failures properly

provide standard proxy headers

Can we add some simple steps/details here? Then we can push as a "Help Wanted."

fix the time sensitivity of the tests

tests are randomly failing due to the non-sync initial propagation of the route settings. however, this is not an issue in normal operation, it causes annoyances in dev time and ci/cd.

header filter behavior and naming

currently the requestHeader and responseHeader filters use the stdlib add method, which in some cases may be what the user expects and in some cases may not.

• enable set besides add
• improve the naming

Refactor RouteInfo

Either remove RouteInfo from everything but etcd, either implement route parsing errors for innkeeper

'routeInfos' here is visible in the docs, due to LoadAndParseAll is public, info doesn't have plural form. Please, give it a better name.

add confirmation and force mode for more dangerous eskip operations

static filter: can make a log warning if the local path is not available

#28 (comment)

glob standard for free wildcards?

it could be tested if /some/path/** works, and then it could be mentioned in the documentation.

#28 (comment)
#28 (comment)

explicit types for different request/response objects during filtering and maybe predicates

Pasting on behalf of @mo-gr :

The incoming req/resp and the outgoing req/resp in some filters can be confusing. I believe, the likelyhood of such issues can be greatly reduced by some type-trickery.
package main

import (
"fmt"
"net/http"
)

type IncomingRequest http.Request
type OutgoingRequest http.Request

func foo(r http.Request) int {
    return 1
}

func Filter(r IncomingRequest) int {
    return 2
}

func main() {
    fmt.Println(Filter(IncomingRequest{}))
    // fmt.Println(Filter(OutgoingRequest{})) // fails
    fmt.Println(foo(http.Request(IncomingRequest{})))
}

This would make it clear, which is which and you could write functions, that are only callable with inbound requests (or outbound responses or whatever)
I am aware that this is a rather huge refactoring. And I'm not 100% sure if the benefits are worth it
Most annoyingly, it would completely break the Filter/Predicate API
Maybe something to consider for a version 2?

Expose the backend url from the route in the FilterContext

Some filters need access to the backend url from the route to whitelist/blacklist certain features

Currently there is no way to do it.

The FilterContext should include a copy of that element from the route

Improve metrics

We need a global request metric and also a routing failure metric

Add a CONTRIBUTING.md

Would also identify 2-3 issues you really want help with that are good candidates for community to-do's, and add them in the README. I could do the add, if you share the issue #'s here.

Host condition naming: currently it is called Host, but it is a regexp. Make consistent with other predicates.

#28 (comment)

Investigate if function need pre and post route filters

innkeeper/client.go:238

func convertJsonToEskip(data []*routeData, preRouteFilters, postRouteFilters []*eskip.Filter) ([]*eskip.Route, []string, string)

Add metrics to count errors

We should also expose counter metrics for, at least, the major errors

Support gzip compression when required

• Return gzip response when Accept-Encoding: gzip, deflate is set by the browser

make the filter instances identifiable for monitoring purpose

chaining filters

allow this:

chainedFilter(p1, p2): filter1("value", p1) -> filter2(1, p2, 2, 3);
noopFilter():;
route1: noopFilter() -> chainedFilter("param1", 1.5) -> otherFilter() -> ":8080"

involves:

• eskip syntax
• runtime route table construction
• etcd: (with prefix /filters on the same level as /routes)
• innkeeper
• eskip tool

Using regex delimiter fails in character list in eskip

This should work

PathRegexp(/[/]/)

but it fails in eskip

include the requested host in the access log

should be similar to setting %{Host}i in apache

Declarative validations for filters and predicates

Right now, as soon as a predicate or filter is constructed, there are some checks done.

Those checks can be done in a more general, declarative way. For example, for a status filter, we could have:

status(float)

Based on this, we know that status should accept one parameter, and that that parameter should be a float.

Make Innkeeper WriteClient secure

Write now, we create it with the insecure flag. writeclient.go:32

Surprising route behaviour with more specific sub-path routes

If I have the following three routes:

a: Path("/foo/*_") -> "https://foo.org"
b: Path("/foo/bar") && Traffic(0.5, "bar", "foobar") -> "https://bar.org"
z: * -> "https://catch.all"

I would expect a request with the path /foo/bar to hit foo.org in half the time and bar.org the other half. However, all the requests, that don't hit bar.org end up being route to catch.all.

This is probably due to the way, the path-match tree is constructed, but it is very surprising and dangerous behaviour.

Handle all non success status codes from Innkeeper

zalando/skipper/innkeeper/client.go:346

In case of a 500, the calling code will believe that the request was successful, and can't even make a log entry or anything.

Add X-Powered-By and Server headers

Skipper needs some branding

Create a Github DataClient implementation

Github provides a good source of route definitions that can be easily reviewed and versioned

Add a filter to generate a request id

Often there is a flow id that helps gather events and log entries from multiple systems originated for the same customer request.

Such flow id needs to be generated at the earliest and sent down the request path so that the following systems can use it for those purposes.

I suggest we use a common HTTP header X-Flow-Id that is created for every request. Generation of this id should favor performance and relax on collision probability since the time window where this is relevant is limited

static filter signature: file system path first, http path prefix optional

#28 (comment)

header matching is case sensitive

Add pre-route and post-route filters

Pre-Route filters allows us to always go through a set of filters that are relevant enough to be applied to each and every route. Owners of the routes don't need to include them and they also can't opt-out.

Post-Route filters can be used to blacklist given headers / cookies, for ex. or override specific headers like 'X-Powered-By', for ex.

consider <shunt> symbol to be implicit

more cleanup for the tests: apply "table-driven" tests, where they are more expressive

Skipper should support SSL/TLS

• It should be possible to turn on and off.
• It should be possible to specify Key, CA, Certificate.

prioritize multiple data clients (innkeeper, overridden by etcd, overridden by file source)

the debug proxy seems to handle the outgoing host wrong, sending the incoming host, as if the preserve host switch was always on

eskip parsing fails when a comment is the last token

Refactor The Command Map

For

src/github.com/zalando/skipper/cmd/eskip/media.go:137

Refactor the map keeping in mind @mo-gr's comment

To me, those two maps (commandToValidations and commands in eskip.go) smell a lot like some 
weird inside-out data structures (call it OO if you insist). I think, command should not be a type alias for 
a string anymore but instead be a proper struct containing a command-name (the current command), a 
validation function pointer and a commandFunc function pointer. This way, the weird map lookups 
would go away and we could do proper validations and command execution dispatched on the actual
command.

Apply Gocover.io so I can add this to Awesome Go

To add Skipper to https://github.com/avelino/awesome-go, it needs a Gocover link. Can you add?

fine tune the proxy response streaming buffer size with benchmarks

as an extra, may check if a buffer pool pays back the cost of the additional complexity

introduce versioning

provide tests for the etcd endpoint rotation

smarter header request and response header, with similar replace capabilities as modPath

eskip pretty printing option

consider if this should be done by eskip, or just provide an awk example

r.PathRegexps should have only one path

 innkeeper/eskiptojson.go:18
 innkeeper/eskiptojson.go:37