vulcand / vulcand

Programmatic load balancer backed by Etcd

Vulcand

Vulcand is a programmatic extendable proxy for microservices and API management. It is inspired by Hystrix and powers Mailgun microservices infrastructure.

Focus and priorities

Vulcand is focused on microservices and API use-cases.

Features

• Uses etcd as a configuration backend.
• API and command line tool.
• Pluggable middlewares.
• Support for canary deployments, realtime metrics and resilience.

Project info

documentation	https://vulcand.github.io/
status	Used in production@Mailgun on moderate workloads. Under active development.
discussions	https://groups.google.com/d/forum/vulcan-proxy
roadmap	roadmap.md
build status

Opentracing Support

Vulcand has support for open tracing via the Jaeger client libraries. Users who wish to use tracing support should use the --enableJaegerTracing flag and must either run the Jaeger client listening on localhost:6831/udp or set the environment variables JAEGER_AGENT_HOST and JAEGER_AGENT_POST. (See the Jaeger client libraries for all available configuration environment variables.)

When enabled vulcand will create 2 spans: one span called vulcand which covers the entire downstream request and another span called middleware which only spans the processing of the middleware before the request is routed downstream.

Aliased Expressions

When running vulcand in a kubernetes DaemonSet vulcand needs to know requests from the local node can match Host("localhost") rules. This --aliases flag allows an author of a vulcand DaemonSet to tell vulcand the name of the node it's currently running on, such that vulcand correctly routes requests for Host("localhost"). The --aliases flag allows the user to pass in multiple aliases separated by commas.

Example

$ vulcand --aliases 'Host("localhost")=Host("192.168.1.1")'

vulcand's Issues

Migrate to godep -r for alternative builds

lightweight docker image

It'd be great to see a smaller docker image. vulcand image is roughly 2x the size of my other containers. I notice that the Go build environment is bundled with the current image...is that necessary?

An alternative approach may be to release precompiled binaries and pull them down at build time.

go-etcd

I notice that vulcand uses a forked version of go-etcd library rather than the upstream version. I'm guessing upstream broke at some point?

Surely it would be better to use Godeps to fix the version to a known good point, and submit patches upstream?

exit 1 on stopping watch

Could vulcand exit with 1 or higher if it stops watching etcd for changes? I keep running into this:

• I mangle the /vulcand configuration in etcd
• The vulcand processes stop working and stick their fingers in their ears
• I fix the /vulcand configuration in etcd
• The vulcand processes hear nothing
• I look at fleetctl list-units and docker ps
• All seems well
• I eventually check the journal
• I have to manually re-start the vulcand units

Example logs:

May 29 21:41:17 node1 docker[15114]: INFO  May 29 21:41:17.816: [etcdbackend.go:384] Etcd backend reading initial configuration
May 29 21:41:17 node1 docker[15114]: ERROR May 29 21:41:17.847: [etcdbackend.go:386] Failed to generate changes: Object not found, stopping watch.

vulcand exits when etcd watch fails

https://github.com/mailgun/vulcand/blob/master/backend/etcdbackend/etcdbackend.go#L399

results in the following on my coreos cluster:
Aug 04 02:17:25 coreos.novalocal bash[6109]: ERROR Aug 4 02:17:25.192: [etcdbackend.go:405] Stop watching: Etcd client error: unexpected end of JSON input
Aug 04 02:17:25 coreos.novalocal bash[6109]: INFO Aug 4 02:17:25.192: [service.go:121] Stopped watching changes with error: unexpected end of JSON input. Shutting down
with error
Aug 04 02:17:25 coreos.novalocal bash[6109]: INFO Aug 4 02:17:25.192: [service.go:111] Got request to shutdown with error: unexpected end of JSON input
Aug 04 02:17:25 coreos.novalocal bash[6109]: Service exited with error: Service exited with error: unexpected end of JSON input

i was told in coreos irc that the watch should be retried and not treated as a fatal error.

perhaps vulcand can catch json.SyntaxError (the type of the error passed) and not treat it as fatal, or ignore any error.

interacting with vulcand through an api

We are trying to programmatically add a host and endpoint configuration to a running vulcand but we cant seem to determine if you can do that over a rest interface. Any suggestions?

Make Documentation Public

Currently, the docs repo that powers vulcanproxy.com is private. Can we make this public so non-Mailgunners can contribute fixes?

Great project

Love the project, was wondering why etcd was chosen as service discovery platform. Consul (http://www.consul.io/intro/vs/zookeeper.html) seems like a better fit as it can be easily incorporated into the rest of the stack for a complete service discovery, k/v store and alerting platform.

Also would love to see SSL and Webscket features added in as seen in this Golang project: https://github.com/joinmytalk/drunken-hipster

Repeatable crash pushing to docker-registry via vulcand

Failed twice in a row at about the same place:

# docker tag 96256ff3db58 registry:80/vulcand:latest
# docker push registry:80/vulcand:latest
The push refers to a repository [registry:80/vulcand] (len: 1)
Sending image list
Pushing repository registry:80/vulcand (1 tags)
Image 511136ea3c5a already pushed, skipping
f10ebce2c0e1: Pushing [================>                                  ] 35.22 MB/109.6 MB 22s
2014/05/23 07:18:47 Failed to upload layer: Put http://registry:80/v1/images/f10ebce2c0e158af1eb0dc08c9e917cc0976e7d57319defbb06ea61191d29e76/layer: write tcp 127.0.0.1:80: broken pipe

Pushing directly worked fine:

# docker tag 96256ff3db58 localhost:49153/vulcand:latest
# docker push localhost:49153:80/vulcand:latest

Log:

May 23 07:15:55 hostname bash[6178]: 2014/05/23 07:15:55 http: panic serving 172.17.42.1:60810: runtime error: invalid memory address or nil pointer dereference
May 23 07:15:55 hostname bash[6178]: goroutine 360 [running]:
May 23 07:15:55 hostname bash[6178]: net/http.func<C2><B7>009()
May 23 07:15:55 hostname bash[6178]: /usr/local/go/src/pkg/net/http/server.go:1093 +0xae
May 23 07:15:55 hostname bash[6178]: runtime.panic(0x6c52e0, 0xa7b068)
May 23 07:15:55 hostname bash[6178]: /usr/local/go/src/pkg/runtime/panic.c:248 +0x106
May 23 07:15:55 hostname bash[6178]: github.com/mailgun/vulcan.(*Proxy).ServeHTTP(0xc21001ded0, 0x7f75b1d87608, 0xc210144140, 0xc210100270)
May 23 07:15:55 hostname bash[6178]: /home/goworld/src/github.com/mailgun/vulcan/proxy.go:38 +0x456
May 23 07:15:55 hostname bash[6178]: net/http.serverHandler.ServeHTTP(0xc21001ebe0, 0x7f75b1d87608, 0xc210144140, 0xc210100270)
May 23 07:15:55 hostname bash[6178]: /usr/local/go/src/pkg/net/http/server.go:1597 +0x16e
May 23 07:15:55 hostname bash[6178]: net/http.(*conn).serve(0xc21011e800)
May 23 07:15:55 hostname bash[6178]: /usr/local/go/src/pkg/net/http/server.go:1167 +0x7b7
May 23 07:15:55 hostname bash[6178]: created by net/http.(*Server).Serve
May 23 07:15:55 hostname bash[6178]: /usr/local/go/src/pkg/net/http/server.go:1644 +0x28b
May 23 07:15:55 hostname bash[6178]: ERROR May 23 07:15:55.334: [proxy.go:35] Request read error read tcp 172.17.42.1:60810: i/o timeout
May 23 07:18:47 hostname bash[6178]: ERROR May 23 07:18:47.592: [proxy.go:35] Request read error read tcp 172.17.42.1:60938: i/o timeout
May 23 07:18:47 hostname bash[6178]: 2014/05/23 07:18:47 http: panic serving 172.17.42.1:60938: runtime error: invalid memory address or nil pointer dereference
May 23 07:18:47 hostname bash[6178]: goroutine 371 [running]:
May 23 07:18:47 hostname bash[6178]: net/http.func<C2><B7>009()
May 23 07:18:47 hostname bash[6178]: /usr/local/go/src/pkg/net/http/server.go:1093 +0xae
May 23 07:18:47 hostname bash[6178]: runtime.panic(0x6c52e0, 0xa7b068)
May 23 07:18:47 hostname bash[6178]: /usr/local/go/src/pkg/runtime/panic.c:248 +0x106
May 23 07:18:47 hostname bash[6178]: github.com/mailgun/vulcan.(*Proxy).ServeHTTP(0xc21001ded0, 0x7f75b1d87608, 0xc210144140, 0xc21013c4e0)
May 23 07:18:47 hostname bash[6178]: /home/goworld/src/github.com/mailgun/vulcan/proxy.go:38 +0x456
May 23 07:18:47 hostname bash[6178]: net/http.serverHandler.ServeHTTP(0xc21001ebe0, 0x7f75b1d87608, 0xc210144140, 0xc21013c4e0)
May 23 07:18:47 hostname bash[6178]: /usr/local/go/src/pkg/net/http/server.go:1597 +0x16e
May 23 07:18:47 hostname bash[6178]: net/http.(*conn).serve(0xc21011ec80)
May 23 07:18:47 hostname bash[6178]: /usr/local/go/src/pkg/net/http/server.go:1167 +0x7b7
May 23 07:18:47 hostname bash[6178]: created by net/http.(*Server).Serve
May 23 07:18:47 hostname bash[6178]: /usr/local/go/src/pkg/net/http/server.go:1644 +0x28b

passing --vulcan to 'host add' command results in error

Trying to add a host using the method described in the readme:

docker run mailgun/vulcan /opt/vulcan/vulcanctl host add --name 'tittyballsci.f4tech.com' --vulcan 'http://bld-docker-02:8182'
Incorrect Usage.

NAME:
   vulcanctl - Command line interface to a running vulcan instance

USAGE:
   vulcanctl [global options] command [command options] [arguments...]

VERSION:
   0.0.0

COMMANDS:
   status, s    Show vulcan status and configuration
   location     Operations with vulcan locations
   host         Operations with vulcan hosts
   endpoint     Operations with vulcan endpoint
   upstream     Operations with vulcan upstreams
   ratelimit    Operations with ratelimits
   connlimit    Operations with connection limits
   help, h      Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --vulcan 'http://localhost:8182'     Url for vulcan server
   --version, -v                        print the version
   --help, -h                           show help

I'm not really sure what to do here - how do I specify the vulcan URL?

Get rid of EtcdKey and Id duplication

Having both EtcdKey and Id is confusing and redundant. Moreover, Id property is not unique.

Get rid of EtcdKey property and use the Id property everywhere. The value of the Id property should be equal to node's full path in Etcd.

This way Id property would be unique.

no error on bind fail

Failing to bind to the given port number (for either proxy or API) results in vulcand process starting without any error being printed. For example, binding to port 80 as non-root user is not allowed, yet vulcand starts anyhow.

I think vulcand should throw a fatal error in this case.

Add alternative backends

• The backends API should be highly documented.
• Any reference to Etcd from the Backend type must be removed. (Related issues: #5)
• Create a toturial explaining how to write a backend.
• Create a Consul backend.
• Create a Consul DNS backend.
• Create a SkyDNS2 backend.
• Create a ZooKeeper backend.

internal timeouts terminating the connection at strange interval

CASE 1
If you have a request which takes >10s, vulcand will complain that the connection timed out at 30s (?!?), won't terminate the client connection until 40s, and then without a proper response:

ERROR Jul 3 19:19:13.664: [proxy.go:50] Request(id=1, method=GET, url=http://localhost:8080/, attempts=3) failed: net/http: timeout awaiting response headers

time curl -i localhost:8181/
curl: (52) Empty reply from server

real 0m40.035s
user 0m0.006s
sys 0m0.008s

CASE 2
If you set the writeTimeout to > 30s, then things look a little better. Still complains at 30s, but we actually get a response:

time curl -i localhost:8181/
HTTP/1.1 502 Bad Gateway
Content-Type: application/json
Date: Thu, 03 Jul 2014 19:23:48 GMT
Content-Length: 23

{"error":"Bad Gateway"}
real 0m30.020s
user 0m0.006s
sys 0m0.007s

Setting the readTimeout seems to result in the mysterious 30s being 30s+readTimeout in the first case, and didn't affect the second case at all.

I would expect that vulcand would send a 502 exactly when the timeout is up (10s) instead of 30s. I'm not sure where that 30s comes from.

Additionally, I'd expect vulcand always to send a response and never just close the connection.

I submitted a PR to allow setting the internal 10s request timeout. It's a little clunky but gets the job done. Thanks!

Etcd expire

It's very useful to have ttl in etcd. Etcdbackend does not handle action == 'expire'.

Is there any use case where expiry should be different from delete?

Example in documentation fails unless -port 8181 is added

While attempting to follow the documentation on starting vulcand in a docker container, I found that it would not work unless the vulcand commandline had -port 8181 added. I am using the mailgun/vulcand image from the docker registry. I wonder if the default port had changed since the docs were written?

Not authenticated response reply

If authentication information is not provided in a request, the Vulcand responds with:

$ curl https://api.mailgun.net/v2/foo.com/events -v -k
< HTTP/1.1 407 Proxy Authentication Required
< Server: nginx/1.4.7
< Date: Mon, 04 Aug 2014 08:23:00 GMT
< Content-Type: application/json
< Content-Length: 41
< Connection: keep-alive

if authentication information is not valid in a request, then Vulcand responds with:

$ curl https://api.mailgun.net/v2/foo.com/events -v -k -u api:bar
< HTTP/1.1 403 Forbidden
< Server: nginx/1.4.7
< Date: Mon, 04 Aug 2014 08:21:48 GMT
< Content-Type: application/json
< Content-Length: 34
< Connection: keep-alive

At the same time the rest of Mailgun in either cases responds with:

$ curl https://api.mailgun.net/v2/realescort.com/bounces -v -k
< HTTP/1.1 401 UNAUTHORIZED
< Server: nginx/1.4.7
< Date: Mon, 04 Aug 2014 08:24:52 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 9
< Connection: keep-alive
< WWW-Authenticate: Basic realm="MG API"

Note the different return codes, and what is more important the presence of the WWW-Authenticate header in the latter case. So Vulcand should probably return 401 in all such cases and provide WWW-Authenticate header in responses.

location rewrite

I have a Jenkins server behind vulcand. It's complaining that my "reverse proxy set up is broken"

Jenkins has a built in test which sends a redirect back to the client with headers:

HTTP/1.1 302 Found
Location: http://myserver/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/a%2Fb/
[...]

The client requests the URL in the Location: field. Jenkins expects to see a request come in unmodified, but instead of the suffix a%2Fb/ it sees a/b/ which results in a 404, and the test fails.

A tcpdump confirms it is vulcand doing this rewrite.

See this post for more discussion

read timeout error

If the client takes too long to complete sending data and readTimeout value is exceeded, vulcand returns HTTP 400 and JSON body with {"error":"Bad Request"}

A more appropriate error might be HTTP 408 {"error":"Request Timeout"}

Relates to #35

Vulcand Dashboard

Build a realtime dashboard to help visualize status and detect anomalies with a Vulcand deployment. For reference, here are the dashboards provided by several other open source projects:

• HAProxy - http://blog.loadbalancer.org/wp-content/uploads/2013/04/HaproxyStats114000cons.png
• nginx - http://demo.nginx.com/status.html

Pluggable middleware

I really love the idea of middleware. It's very useful for us as we would like to write some middle where to call one of our internal REST services to make some decisions when routing to the appropriate upstreams.

I would love to see some way where we can have pluggable middleware so that we do not need to recompile vulcand.

We could then have a scenario where vulcand is installed using a vanilla package built by a trusted source and include the middleware as some sort of module.

Allow fallback through etcd configuration

If the specified upstream is down or not responding a fallback would be useful for common error pages.

Perhaps

/bin/etcdctl set /vulcand/hosts/www.mysite.com/locations/home/fallback myerrorsite

etcd client - empty body error

I'm seeing a weird issue where exactly 5mins after vulcand has started, it shuts down with error:

ERROR Jul 18 12:41:13.735: [etcdbackend.go:405] Stop watching: Etcd client error: unexpected end of JSON input

I've enabled some debug and discovered that the etcd client is getting a response back with an empty http body, hence the unmarshall error.

This only seems to have started since I updated CoreOS. The update includes a newer etcd version - 0.4.5

I'm starting vulcand as follows:

vulcand -apiInterface="0.0.0.0" -interface="0.0.0.0" -etcd="http://172.17.42.1:4001" -port=8181

Endpoint failover/retry not working as expected

I am using a basic setup where I am trying to load balance two Tomcat instances running in Docker containers. Each has an endpoint listening on a high port number. They are load balanced as expected in a round robin fashion when both are functioning. The problem I am encountering is when the connector is configured to only start on init (using bindOnInit). This makes it so that when a deployment is occurring, the Tomcat instance is not yet listening on the specified port number so it returns a Connection refused if you access it directly, bypassing vulcand.

When going through vulcand, I receive a {"error":"Bad Gateway"} response back to the client. When watching via vulcanctl status, I can see that it registers as a failure and in the vulcand logs, all I see is this:
ERROR Jul 24 14:53:47.800: [proxy.go:55] Request(id=55, method=POST, url=http://192.168.56.2:49162/login, attempts=1) failed: read tcp 192.168.56.2:49162: connection reset by peer

I would have expected it to retry on another endpoint instead of just returning the 502 back to the client. Is performing an automatic retry not expected here? I did not see any configuration options to change this behavior unless I missed something.

*edit: For clarity, to test this, you can simply add an endpoint pointing to a url where no service is listening and the same behavior is seen.

'object not found' error

Vulcand fails to start with the following messages:

INFO  Jul  4 00:19:01.292: [etcdbackend.go:384] Etcd backend reading initial configuration
ERROR Jul  4 00:19:01.359: [etcdbackend.go:386] Failed to generate changes: Object not found, stopping watch.
INFO  Jul  4 00:19:01.359: [service.go:117] Stopped watching changes with error: Object not found. Shutting down with error
INFO  Jul  4 00:19:01.359: [service.go:107] Got request to shutdown with error: Object not found
Service exited with error: Service exited with error: Object not found

Which object? vulcand should print which object it is failing to read! This time, I happen to know which object is missing, but that won't always be the case 😄

ssl

Could vulcand be given the ability to handle SSL terminations?

I think the only workable approach would be to use SNI (virtual hosts for SSL). This post outlines how it might be achieved.

Certificates could be stored in etcd backend but the challenge will be how to manage the private keys...

Support for setting TrustForwardHeader?

We are currently running vulcand behind an SSL offloader which also sets X-Forwarded-Proto.

The Vulcan library supports forwarding this header if location.options.TrustForwardHeader is set (See https://github.com/mailgun/vulcan/blob/master/location/httploc/httploc.go#L198).

Is it possible to set TrustForwardHeader in vulcand, or is that not (yet) implemented?

502 Bad Gateway

Way too short upstream timeout makes Vulcand prematurely quit waiting for a response from an upstream and request another upstream instance. That can repeat several times within one request timeout. As a result clients receive 502 Bad Gateway as a response and upstreams are DOSed by Vulcand.

Support consul in addition to etcd

Consul is another k/v store similar to etcd, but it has more functionality and does better in network partition testing via the jepsen tests. Bonus points that the hashicorp guys are amazing and do great stuff.

Please consider it as a backend for vulcand in addition to etcd please. Kelsey Hightower's confd is a tool that also supports both. Might give you some ideas perhaps.

Internal Server Error when upstream is not set.

Given: Vulcand Created from docker repo : "mailgun/vulcand" with no existing hosts.
And: I add upstream with endpoints, host, location but do not redirect the traffic of the location to the any of the upstreams.

When: I access Vulcand Hosts API

curl {{VULCAND_URL}}/v1/hosts

Then: I get "Internal Server Error" in response.

When: I redirect the traffic to the newly created upstream

Then: The error goes away.

Example used for test:
http://coreos.com/blog/zero-downtime-frontend-deploys-vulcand/

serverWriteTimeout default

Using the recently renamed flag serverWriteTimeout I'm seeing an issue where a large download (i.e. from backend server, through proxy to client) is exceeding the timeout threshold (i.e. download takes longer than 60s) and the connection is dropped. Smart clients will auto-resume, however many do not (including Firefox web browser) resulting in a truncated file.

Increasing serverWriteTimeout from the default fixes the issue, however I wonder if it wouldn't be better if the default was for either no timeout or at least a very large timeout (e.g. 1hr) ?

Enhancement: sticky sessions

Sticky sessions track a client, and if a new request can be sent to an endpoint that the client has talked with, will forward the request to that endpoint in preference over an endpoint the client has not spoken with before.

It'd be good to see this implemented such that each location can specify: global-sticky, location-sticky, or no-sticky: no-sticky is no stickiness, location-sticky only considers requests made against this location, and global sticky would track client<->endpoint url for all connections made.

Docker Trusted Build

Can you add a trusted build to index.docker.io?

Add option for using a trie instead of regex to route urls

Read a few of these for the reasoning why:

https://gdstechnology.blog.gov.uk/2013/12/05/building-a-new-router-for-gov-uk/
https://github.com/alphagov/router/tree/master/triemux

http://blog.ant0ine.com/typepad/2013/02/better-url-routing-golang-1.html
https://github.com/ant0ine/go-urlrouter

For lots and lots of urls, regular expressions tend to slow things down a bit.

not in conformance with go recommendations of lower-cased errors

see https://code.google.com/p/go-wiki/wiki/CodeReviewComments#Error_Strings for the convention on creating error strings in go. as i understand, error strings should be lowercased. however, it's OK for log messages to be uppercased.

Question: Load Balancer

Hi,
From reading the docs it doesn't seem like Vulcan could be used as a Load Balancer. Or is it implicit that having multiple endpoints under a upstream key will act as a Load Balancer? If so, is it doing simple round-robin or is there more to it? Any kind of state detection on the endpoint?
EDIT: Looking at http://vulcan.ghost.io/intro/ it's more clear now. Time to get hands on and try it.

Order of `--vulcan` matters even though it is a global option

Order of --vulcan matters even though it is a global option:

root@d4f5ad745101:/opt/vulcan# ./vulcanctl status --vulcan 'http://172.17.42.1:8182'

[hosts]

root@d4f5ad745101:/opt/vulcan# ./vulcanctl --vulcan 'http://172.17.42.1:8182' status
[ERROR]: Get http://localhost:8182/v1/hosts: dial tcp 127.0.0.1:8182: connection refused

readTimeout default

Please consider raising the default value for -readTimeout. The current default of 10s is easy to exceed on a moderately large POST and will catch people unaware. For reference, Nginx defaults to 60s for its proxy_read_timeout setting

Changing order of sets for location breaks config

This works:

etcdctl set /vulcand/hosts/localhost/locations/loc1/path "/home"
etcdctl set /vulcand/hosts/localhost/locations/loc1/upstream up1

While this does not:

etcdctl set /vulcand/hosts/localhost/locations/loc1/upstream up1
etcdctl set /vulcand/hosts/localhost/locations/loc1/path "/home"

Enhancement: vulcand health status check from loadbalancers

Currently vulcand status can be only checked http://172.17.8.99:8182/v1/status way, which does not work for AWS and GCE loadbalancers, where the check is done curl -L http://vulcand_ip/v1/status:8182
Would be possible to open an extra port just to work for loadbalancers?

Enhancement: Mongrel2 ZMQ transport backend

Vulcand as a front-end-server has to initiate a lot of connections to the back-end services.

Mongrel2 performs a similar front-end service, however instead of going HTTP to the backend services, it specifies a ZeroMQ protocol for talking with backend services:

First, the types of ZeroMQ sockets used are a ZMQ_PUSH socket for messages from Mongrel2 to Handlers, which means your Handler’s receive socket should be a ZMQ_PULL. Mongrel2 then uses a ZMQ_SUB socket for receiving responses, which means your Handlers should send on a ZMQ_PUB socket. This setup allows multiple handlers to connect to a Mongrel2 server, but only one Handler will get a message in a round-robin style. The PUB/SUB reply sockets, though, will let Handlers send back replies to a cluster of Mongrel2 servers, but only the one with the right subscription will process the request.

This allows for non-TCP based connectivity (ZeroMQ supports pipes and UDP for example) between the front & back end servers, it allows for two sockets to transmit all traffic between front/back end, and it allows for non-blocking use of the socket.

Alas mongrel2 has rather rudimentary configurability, and poor online configurability: both things where Vulcand is ideal.

Ideally I'd like to see vulcand be able to create locations or endpoints that create a mongrel2-speaking zeromq {push/pull request socket, pub/sub response socket} pair.

The docs on Mongrel2 are so so. http://mongrel2.org/manual/book-finalch6.html#x8-600005 has a bunch of implementation details for dealing with their ZeroMQ sockets, but isn't a great high level intro. There's a pretty wide range of existing app servers with Mongrel2 client implementations.

Unexpected error: 401: The event in requested index is outdated and cleared (the requested history has been cleared

I'm getting these errors:
[etcdbackend.go:429] Unexpected error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [5420728/3680799]) [5421727], reconnecting

And for some of my sites I get then {"error":"Bad Gateway"}, but these sites for sure have the right host:ip.
To get all sites working again I need to restart vulcand service. I'm on CoreOS latest alpha v402.2.0.

rewrite path

I'd like to be able to have a single host with 2 locations specified e.g.

vulcand/hosts/example.com/locations/loc1/path  "/bar"
vulcand/hosts/example.com/locations/loc2/path  "/foo"

Where:
http://example.com/bar/xyz is sent to upstream host1 as http://host1/bar/xyz
http://example.com/foo/xyz is sent to upstream host2 as http://host2/xyz i.e. the path specified in the path key is stripped off.

I realise this could be achieved with middleware, but I wonder if it isn't a common enough case to warrant adding a special type to the core, for example:

vulcand/hosts/example.com/locations/loc2/pathx  "/foo"

Where pathx indicates the special type which strips the path /foo (but leaves anything after it)

api creates should return json of the created object

If I create an upstream without submitting an ID using the web api the only json I get back is {"message":"Upstream added"}

It would be really helpful if the api returned that object so I could save it off and use it at a later time.

Creating etcd keys after vulcan is started ignores changes

Order of events:

1. Vulcan is started
2. Keys created via etcdctl
3. This log entry appears every second:

INFO  Apr 13 22:35:16.350: [etcdbackend.go:50] Got response: set /vulcand/upstreams/robszumski/endpoints/robszumski-3.service 85929 %!s(<nil>)
ERROR Apr 13 22:35:16.354: [etcdbackend.go:53] Failed to process change: Unsupported action on the endpoint: set, ignoring

Vulcan works properly when you create the keys before it starts:

1. Keys created via etcdctl
2. Vulcan is started

Regex to match all paths doesn't work

I'm trying to match everything after /, but I keep getting bad gateway on all of the stylesheets, images, etc.

$ etcdctl get /vulcand/hosts/robszumski.com/locations/home/path
/.*

Have you gotten this to work?

docker clean exit

In order for Docker to consider the container to have exited cleanly, it must shut down gracefully on a SIGTERM. If it does not respond, then a SIGKILL is sent and the shutdown is considered unclean.

Without a clean exit, a container that has been marked for auto-removal (with --rm flag) does not get deleted.

It would be good to have vulcand respond to SIGTERM

Configurable Timeouts

Running into an issue when hitting an API that takes a little bit longer to return a response with the following error after 3 attempts:

attempts=3) failed: net/http: timeout awaiting response headers

Are there plans to make the timeout setting configurable in the future? Great project btw, thanks!

Transparency

I have a service behind vulcan where i need the url to contain another url (encoded)

http://vulcan.site.com/log/http%3A%2F%2Fwww.site.com%2Fsomething

As mentioned in #40 vulcan rewrites this url to

http://vulcal.site.com/log/http://www.site.com/something

I'm aware that vulcan doesn't claim to be a transparent proxy, but as the docs (http://golang.org/pkg/net/url/) describe, the server could consult req.RequestURI and use URL{Host: "example.com", Opaque: "..." }

Is there any reasonable usecase where this whould not be the desired behavior?

Expose /health endpoint

It would be awesome if there was a url to hit that tracked vulcand health for use with cloud load balancer's health monitor.

Great work so far, I'm loving it.