shazow / gohttplib

Go buildmodes prototype: Expose net/http.Server as a library with C bindings

gohttplib

Shared library that exposes Go's net/http.Server with externally-bindable handlers.

This is a silly project for experimenting with Go buildmodes. I gave a talk about this at PyCon 2016.

Status: Tiny subset of the http.HandlerFunc callback gets passed to a C handler callback. Python bindings are working, too.

Getting Started

Requirements:

• Go 1.5 or newer.
• C example: make, gcc
• Python example: cffi, python-cffi

$ git clone https://github.com/shazow/gohttplib/
$ cd gohttplib
$ make examples

Example: C

C example can be linked against a shared object (libgohttp.so generated by go build -buildmode=c-shared) or against a static library archive (libgohttp.a generated by go build -buildmode=c-archive). By default, we link against the shared object because that's what the Python example uses too.

Linked against our shared object:

$ make example-c
$ DYLD_LIBRARY_PATH=build/ ./build/gohttp-c

Note that you'll need to make sure that libgohttp.so is findable at runtime.

Now you can request http://localhost:8000/hello and the C handler in examples/c/main.c will handle it!

Linked against our static library archive:

$ make example-c-static
$ ./build/gohttp-c-static

The static archive gets built into the binary so it's more portable during runtime. Note the size differences:

8.8K  gohttp-c
5.1M  gohttp-c-static
7.5M  libgohttp.a
6.9M  libgohttp.so

Example: Python

The Python example uses python-cffi (you'll need python-cffi installed) to link against the shared object.

$ make example-python
$ cd examples/python
$ python -m gohttp
 * Running on http://127.0.0.1:5000/

Or write your own handler using this library:

from gohttp import route, run

@route('/')
def index(w, req):
    w.write("%s %s %s\n" % (req.method, req.host, req.url))
    w.write("Hello, world.\n")

run(host='127.0.0.1', port=5000)

References & Credit

• Go Execution Modes document.
• Every single StackOverflow thread tagged [cgo].
• github.com/jrick/buildmodes
• github.com/wolever/python-cffi-example
• Invaluable help with python-cffi from @wolever, @paultag, and @dstufft.

Sponsors

This project was made possible thanks to Glider Labs.

License

MIT

gohttplib's Issues

Server port is still in use after killing the application.

examples/c: Link against c-shared object

C example currently links against the go build -buildmode=c-archive object, need to see if it'll also link against the c-shared variant so that we can use the same one as the Python example.

[bug] memory leak through C allocated memory

The server leaks memory through C.* invocations that are not matched with a C.free()

examples/python: Need wrappers for lib

• HandleFunc
  https://cffi.readthedocs.org/en/latest/using.html#callbacks
• ListenAndServe (basically alias should do)

library: Implement C bridge for io.ReadWriteCloser interfaces.

... and possibly more io.* interfaces. These kinds of interfaces crop up all over Go code and it'd be really useful to have a reusable bridge for them.

examples/python: Segmentation fault on exit

$ cd examples/python
$ python -c "import gohttp"
Segmentation fault: 11

Presumably we need to tell Go that we're closing before we close, or something? Works fine from Python shell. Not sure why it's special.

>>> import gohttp
>>> exit()

Error while using library on mac.

I'm running Mac OS-X .
During the installation via pip there were no errors. But while running, got an OSError error.

Trace

gohttp/libgohttp.so: unknown file type, first eight bytes.

When I tried to solve the problem, I saw this stack overflow question.
Also here is a similar issue on another python library.

tutorial/automation to wrap standard Go functions

I found https://blog.heroku.com/see_python_see_python_go_go_python_go while looking at how can I call Go stdlib from Python. In many cases (time/date, exec, net) API seems more mature to me.

I found https://github.com/go-python/gopy but it is somewhat stalled and is not compatible with 1.6, so the question is - is it possible to write a tool that can wrap Go functions automatically, starting from more simple ones and gradually build up?

expose time.Now() gogogo

Then in Python just:

from gogogo import time
print time.Now().String()

Just a fantasy. =) I know that it is complicated, takes a lot of time and requires some specs to be written. Maybe there are already existing project like https://github.com/pybind/pybind11 that are more active.

examples/python: Ship wheels for all platforms for pypi

Right now https://pypi.python.org/pypi/gohttp ships with just an OSX-built shared object for gohttplib. Won't work on other architectures.

TypeError occurred when gohttp was run on Python 3.5 environment

I tried to run on py35 environment, but an error was occurred as below. However, it's running on py27 environment. Is there any issues on my environment?

(gohttp)$ make example-python
(gohttp)$ cd examples/python && python gohttp/build_gohttplib.py
(gohttp)$ python -m gohttp
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/t2y/work/external-repo/go/gohttplib/examples/python/gohttp/__main__.py", line 4, in <module>
    @route('/')
  File "/Users/t2y/work/external-repo/go/gohttplib/examples/python/gohttp/__init__.py", line 59, in wrapped
    lib.HandleFunc(pattern, handler)
TypeError: initializer for ctype 'char *' must be a bytes or list or tuple, not str

library: Expose Server.Shutdown interface

We want to expose an interface to https://golang.org/pkg/net/http/#Server.Shutdown

1. We currently use the package-global http.ListenAndServe helper. Reading the source, it's just a wrapper around creating a server instance and calling ListenAndServe() on it. We'll need access to that server instance to call Shutdown(context.Context) on it later, so we'll have to write our own ListenAndServe() one-liner instead of using the package-global one.

2. Once https://github.com/shazow/gohttplib/blob/master/gohttplib.go has its own server instance it can access, add a Shutdown() wrapper, something like:

var server *http.Server
...
func Shutdown() error {
  // We can skip passing in the context for now, could even skip returning an error if that's too annoying
  return server.Shutdown(context.Background())
}

3. Export it and make the bridge accessible from Python

net/http.Server linking failure

$ go version
go version devel +38e3427 Sat Jun 13 19:59:16 2015 +0000 darwin/amd64
$ go build -buildmode=c-archive
$ gcc -o gohttp-c examples/c/main.c gohttplib.a -lpthread
Undefined symbols for architecture x86_64:
  "_CFArrayGetCount", referenced from:
      _FetchPEMRoots in gohttplib.a(000003.o)
  "_CFArrayGetValueAtIndex", referenced from:
      _FetchPEMRoots in gohttplib.a(000003.o)
  "_CFDataAppendBytes", referenced from:
      _FetchPEMRoots in gohttplib.a(000003.o)
  "_CFDataCreateMutable", referenced from:
      _FetchPEMRoots in gohttplib.a(000003.o)
  "_CFDataGetBytePtr", referenced from:
      _FetchPEMRoots in gohttplib.a(000003.o)
      __cgo_6dbb806e9976_Cfunc_CFDataGetBytePtr in gohttplib.a(000003.o)
     (maybe you meant: __cgo_6dbb806e9976_Cfunc_CFDataGetBytePtr)
  "_CFDataGetLength", referenced from:
      _FetchPEMRoots in gohttplib.a(000003.o)
      __cgo_6dbb806e9976_Cfunc_CFDataGetLength in gohttplib.a(000003.o)
     (maybe you meant: __cgo_6dbb806e9976_Cfunc_CFDataGetLength)
  "_CFRelease", referenced from:
      _FetchPEMRoots in gohttplib.a(000003.o)
      __cgo_6dbb806e9976_Cfunc_CFRelease in gohttplib.a(000003.o)
     (maybe you meant: __cgo_6dbb806e9976_Cfunc_CFRelease)
  "_SecKeychainItemExport", referenced from:
      _FetchPEMRoots in gohttplib.a(000003.o)
  "_SecTrustCopyAnchorCertificates", referenced from:
      _FetchPEMRoots in gohttplib.a(000003.o)
  "_kCFAllocatorDefault", referenced from:
      _FetchPEMRoots in gohttplib.a(000003.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [example-c] Error 1

gohttplib supports concurrent request processing

This is an interesting feature not clearly mentioned in your blog, or the README.

gohttplib supports concurrent request processing out of the box. For example a method that takes a second for every 10th invocation (see below). Will process 100 requests concurrently:

> ab -n 100 -c 100 -s 5 -k http://127.0.0.1:5000/
...
Time taken for tests:   1.009 seconds
...

The same test with flask takes 10 seconds, unless you pass threaded=True

The test method

@route('/')
    def index(w, req):
        global count
        if count % 10 == 0:
            foo = time.time()
            while time.time() < foo + 1:
                print "doing work"
        count += 1
        w.set_status(200)

        w.write(b"Hello, world.\n")
        pass

examples/python: Link against c-archive?

We got our example working with go build -buildmode=c-shared but haven't been able to get it working with the file produced by go build -buildmode=c-archive. Would be nice to link against the same thing from both the C and Python examples.

Or at least figure out why it's not working.

library: expose request headers and body

I was trying out gohttplib after reading the blog post, which was very informative and detailed.

For the simple experiment I wanted to do (accept webhook triggers) I need access to the request headers and request body.

Unable to run benchmark past ~16,300 requests

As discussed in #11, I encountered the following issue: after about 16,300 requests created by the benchmark tool ab the server 'hangs'. This happens regardless of the concurrency

The failing command:

> ab -n 20000 -c 1 -s 5  http://127.0.0.1:5000/
...
Completed 16000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 16280 requests completed

When enabling keep-alive from ab the command does complete (-k), even with 10x more requests:

> ab -n 200000 -c 1 -s 5  -k http://127.0.0.1:5000/
...
Finished 200000 requests
...

Note, the memory leak #14 is not causing this issue.

library: Bridge the remainder of the API

• type ResponseWriter
  • Write
  • WriteHeader
  • Header
• type Request
  • Method
  • URL
  • Host
  • Header
  • Body (io)
  • ... Bunch of other stuff we may or may not care about?
• type Server
  • API for a straight-up handler or a custom Server configuration?
• Other stuff?