rcrowley / goagain

Zero-downtime restarts in Go

goagain

Zero-downtime restarts in Go

The goagain package provides primitives for bringing zero-downtime restarts to Go applications that accept connections from a net.TCPListener or net.UnixListener.

Have a look at the examples because it isn't just a matter of importing the library and everything working. Your main function will have to accomodate the goagain protocols and your process will have to have some definition (however contrived you like) of a graceful shutdown process.

Installation

go get github.com/rcrowley/goagain

Usage

Send SIGUSR2 to a process using goagain and it will restart without downtime.

example/single/main.go: The Single strategy (named because it calls execve(2) once) operates similarly to Nginx and Unicorn. The parent forks a child, the child execs, and then the child kills the parent. This is easy to understand but doesn't play nicely with Upstart and similar direct-supervision init(8) daemons. It should play nicely with systemd.

example/double/main.go: The Double strategy (named because it calls execve(2) twice) is experimental so proceed with caution. The parent forks a child, the child execs, the child signals the parent, the parent execs, and finally the parent kills the child. This is regrettably much more complicated but plays nicely with Upstart and similar direct-supervision init(8) daemons.

goagain's Issues

relauch terminates when the executable is missing

While it's not likely that the user is going to signal the running app to -USR2 when the executable is not present... when the user does the app crashes. It should write an error to the log instead of stopping.

How to build a service which supports the 'restart' operation with goagain?

Hi,

Could you point me the direction how can I build the service which would support the "restart" operation?

The big goal of mine, is to develop an application which I would be able to restart using the command like
service apache2 restart
This case, when I have new binary uploaded, the new entity would be the updated application (am I right?)

As I understand the "goagain" is something I can use for this purpose. But I'm a bit lost about how should I use it.

I run the "single" example - I tried to kill it, to interrupt it, but neither operation has lead to the fact that the application was restarted.

1. How can I run the example app and see if it works?
2. Do you have any idea what I should research to be able to add graceful restart / stop and start to my application?

Thanks

Doesn't work in OSX

I'm really confused why this doesn't work in OSX, the sub process always says that "dup: bad file descriptor" . But I have it working fine on linux. Any ideas of what to look at?

Support UNIX domain sockets in addition to TCP sockets

This came up in #6 and #7 so I'd like to oblige.

Leaks file descriptors on graceful reload

The first time it runs.

goagain-e 9114 manuel 0u CHR 136,0 0t0 3 /dev/pts/0
goagain-e 9114 manuel 1u CHR 136,0 0t0 3 /dev/pts/0
goagain-e 9114 manuel 2u CHR 136,0 0t0 3 /dev/pts/0
goagain-e 9114 manuel 3u IPv4 678836 0t0 TCP 127.0.0.1:48879 (LISTEN)
goagain-e 9114 manuel 4r FIFO 0,8 0t0 678837 pipe
goagain-e 9114 manuel 5w FIFO 0,8 0t0 678837 pipe
goagain-e 9114 manuel 6u 0000 0,9 0 6869 anon_inode
goagain-e 9114 manuel 17r REG 0,3 0 410676 /proc/4972/auxv

Every subsequent time, it adds two more sockets.

goagain-e 9268 manuel 0u CHR 136,0 0t0 3 /dev/pts/0
goagain-e 9268 manuel 1u CHR 136,0 0t0 3 /dev/pts/0
goagain-e 9268 manuel 2u CHR 136,0 0t0 3 /dev/pts/0
goagain-e 9268 manuel 3u IPv4 678836 0t0 TCP 127.0.0.1:48879 (LISTEN)
goagain-e 9268 manuel 4u IPv4 678836 0t0 TCP 127.0.0.1:48879 (LISTEN)
goagain-e 9268 manuel 5r FIFO 0,8 0t0 688469 pipe
goagain-e 9268 manuel 6w FIFO 0,8 0t0 688469 pipe
goagain-e 9268 manuel 7u IPv4 678836 0t0 TCP 127.0.0.1:48879 (LISTEN)
goagain-e 9268 manuel 8u 0000 0,9 0 6869 anon_inode
goagain-e 9268 manuel 17r REG 0,3 0 410676 /proc/4972/auxv

Where are you reopening the socket?

Runtime go is the one that ships with Ubuntu 12.04.

Bad file descriptor when opening file after USR2

I'm experiencing an odd issue after issuing a USR2 signal to my process. I've got an OnSIGHUP handler which opens some files and reloads data from them. On the first HUP after USR2 I get

read file.txt: bad file descriptor

if I HUP the process again it all works fine. I've been able to hack around this by setting a global variable after USR2 and if its the first HUP after a USR2 and doing

f, err := os.Open(os.DevNull)
if err == nil {
    defer f.Close()
}

and then running the code to reload the data from the files. Is this anything you've seen before, is there likely to be anything common I've not done correctly.

My use of goagain effectively matches the examples/simple/main.go example with the addition of the OnSIGHUP handler.

I'll try to replicate the issue with the simple example.

Make it Upstart-friendly

After talking with Paul Hammond, the following scheme would make Goagain Upstart-friendly:

Open the *net.TCPListener in the parent process and then fork a child process. The child process execs and begins accepting connections on the inherited *net.TCPListener.

When signaled SIGUSR2, the parent forks a new child. The child process execs, which loads new code from disk, and resumes accepting connections on the inherited *net.TCPListener. When this new child is operational, it signals the old child SIGQUIT to exit gracefully.

The parent process should only exit on SIGTERM and the like.

single strategy error in mac

In Single strategy，setEnvs error in mac.

error message

panic: reflect: call of reflect.Value.Int on zero Value

goroutine 1 [running]:
reflect.Value.Int(...)
	/usr/local/go/src/reflect/value.go:961

solution

Replace with below code will solve this problems.

v := reflect.ValueOf(l).Elem().FieldByName("fd").Elem().FieldByName("pfd")
fd = uintptr(v.FieldByName("Sysfd").Int())

Go programs can handle SIGQUIT

This tool does not handle SIGQUIT by default because it claims that Go "doesn't seem to like handling SIGQUIT". That may have been true in prior versions of Go, but in Go 1 SIGQUIT can be handled just like any other signal. Programs run under the "go run" tool aren't able to intercept SIGQUIT, but programs compiled to an executable and run work fine. Since a goal seems to be to mimic the behavior of Unicorn and nginx, I'd suggest that SIGQUIT can be used instead of SIGTERM.

goagain does not work!

Running the following httperf command... almost 4000 transactions are lost because it takes about a second to relaunch the app.

httperf --num-calls=4000 --port= 48879 --uri=/mock --num-conns=10

Looking closely at the code it's not possible to have a zero downtime.

(a) goagain shares open connection
(b) there is no state migration
(c) during the swap no new connections can be made and therefore potential transactions lost

/r

Make stub functions for Windows (and possibly others)

Currently any project that includes goagain will be unbuildable on Windows due to syscall dependencies only present on some systems.

Would it be possible to create stubs that will be used for unsupported systems via build tags?

Of course there will not be any of the goagain functionality, but at least our developers on other platforms will be able to build projects using goagain.

PS. Thanks for an awesome project!

Multiple listeners support

I have multiple listeners (like two listening on port :8080 and 8081 and one listening to https on 443 port). That means I have now three listeners. However the nature of goagain doesn't allow to use it with several listeners. I can split my app that it runs like three binaries but this would be not a good choice.

I've hacked a little on goagain to support for multiple listeners, a sneak code:

for addr, l := range listeners {
    a := reflect.ValueOf(l)
    v := a.Elem().FieldByName("fd").Elem()
    fd := uintptr(v.FieldByName("sysfd").Int())
    if err := os.Setenv("GOAGAIN_FD"+addr, fmt.Sprint(fd)); nil != err {
        return err
    }

    if err := os.Setenv("GOAGAIN_NAME"+addr, fmt.Sprintf("tcp:%s->", l.Addr().String())); nil != err {
        return err
    }

    fmt.Println("FD number is", fd)
    files[fd] = os.NewFile(fd, string(v.FieldByName("sysfile").String()))
}

However I don't like the way the file descriptors are created. Because I have to create the array earlier to append the FD's to the files arary, like:

files := make([]*os.File, 30 )

Here, 30 is just an integer that's large enough. I've tried to use append on files array. However when I use append it's breaking goagain with this err:

 fork/exec ./go/bin/test_goagain: bad file descriptor

It seems it is working quite well. Do you have any plans to support multiple listeners?

Lack of examples of how to use it for use with http in the standard library

I like the idea of goagain, but I'm surprised that so much code has to live in my main(). Can't we have a goagain.HttpListenAndserve()? This would be much more user friendly and solve the common case that one just wants to use go's builtin http server.

Is there a good reason it is not like this?

Doesn't finish ongoing requests on reload

Unlike http://code.google.com/p/jra-go/source/browse/cmd/upgradable/main.go, the parent process doesn't actually stop listening or wait until it's done handling all requests, before it dies. This is because the child kills the parent and the parent exits unconditionally. This causes loss of requests.

goagain hijacks sigquit

by default in go, the quit signal is used to print stacktraces of running go programs (type ctrl+\ or use kill -QUIT <pid>). this is a very useful feature, but goagain overrides the quit signal.

Fails to compile: "cannot use int(fd) (type int) as type syscall.Handle in function argument"

Doing a go get github.com/rcrowley/goagain gives me:

C:\gd\src\github.com\ezbiz>go get github.com/rcrowley/goagain
# github.com/rcrowley/goagain
c:\gd\src\github.com\rcrowley\goagain\goagain.go:25: undefined: syscall.SIGUSR2
c:\gd\src\github.com\rcrowley\goagain\goagain.go:45: undefined: syscall.SIGUSR2
c:\gd\src\github.com\rcrowley\goagain\goagain.go:71: cannot use int(fd) (type int) as type syscall.Handle in function argument
c:\gd\src\github.com\rcrowley\goagain\goagain.go:92: undefined: syscall.Kill

This is go version go1.0.3 64-bit under Windows 7 64-bit. It's strange because syscall.Kill and syscall.SIGUSR2 are certainly "defined" under Go 1.0.3 as per the official pkg-docs..

Any idea what the issue could be here?

Drop-in replacement for net.Listen

Would it be possible to wrap some of the complexity of the handoff / setup code? With a similar signature as net.Listen, it should either resume the previous connection or set up a new one. This could be performed by calling goagain.Listener() and depending on whether it succeeds, either kill off the parent (or other appropriate cleanup) or return a new net.Listener by calling net.Listen(net, laddr).

Basically handling all of example/simple/simple.go#L20-44 (but just returning the listener). This would make the library a near drop-in replacement of the standard net.Listener.

It could be named goagain.Listen akin to net.Listen, goagain.ListenAgain, or maybe goagain.ReListen.

Remove or provide opt out for log messages

It would be nice if these log messages were removed or there was a way to turn logging off (package scoped flag?):

2012/05/23 00:33:11 user defined signal 2
2012/05/23 00:33:11 spawned child 3060