Coder Social home page Coder Social logo

zurl's Introduction

Zurl

Author: Justin Karneges [email protected]

Description

Zurl is an HTTP and WebSocket client daemon with a ZeroMQ interface. Send it a message to make an HTTP request.

For example, here's how to make a request using Python:

import json
import zmq

# set up zmq socket
sock = zmq.Context.instance().socket(zmq.REQ)
sock.connect('ipc:///tmp/zurl-req')

# send request
req = {
  'method': 'GET',
  'uri': 'http://example.com/path'
}
sock.send('J' + json.dumps(req))

# print response
print json.loads(sock.recv()[1:])

Since every language can already make HTTP requests directly, you might wonder what value there is in delegating the work to a separate process like this. Zurl is mainly useful for implementing Webhooks, because applications don't need to keep state nor worry about concurrency. Zurl even offers protection from evil URLs.

Zurl can also make sense as part of a greater ZeroMQ-based architecture, where you want to integrate HTTP itself into your pipeline.

See Fun With Zurl for some wild possibilities that a message-oriented HTTP client daemon can bring.

License

Zurl is offered under the GNU GPL. See the COPYING file.

Features

  • Request HTTP and HTTPS URLs
  • Connect to WS and WSS URLs for WebSockets
  • HTTP support based on Libcurl
  • Event-driven design can handle thousands of simultaneous connections
  • Two access methods: REQ and PUSH/SUB
  • Requests and responses can be buffered in single messages or streamed
  • Packet format can be JSON or TNetStrings
  • Set access policies (e.g. block requests to 10.*)

Requirements

  • qt >= 5.2
  • libzmq >= 2.0
  • libcurl >= 7.20

Setup

If accessing from Git, be sure to pull submodules:

git submodule init
git submodule update

Build:

./configure --qtselect=5
make

Run:

cp zurl.conf.example zurl.conf
./zurl --verbose --config=zurl.conf

Message Format

Requests and response messages are encoded in JSON or TNetStrings format. The format type is indicated by prefixing the encoded output with either a 'J' character or a 'T' character, respectively.

For example, a request message in JSON format might look like this:

J{"method":"GET","uri":"http://example.com/"}

Here's an example of the same request in TNetStrings format:

T44:6:method,3:GET,3:uri,19:http://example.com/,}

Here's what a response might look like:

J{"code":"200","reason":"OK","headers":[...],"body":"hello\n"}

Zurl always replies using the same format that was used in the request. If you need to send and receive binary content, you'll need to use TNetString format rather than JSON (Zurl does not attempt to Base64-encode binary content over JSON or anything like that).

Requests may have a number of fields. Here are the main ones:

  • id - Unique ID among requests sent.
  • method - The HTTP method to use.
  • uri - The full URI to make the request to, e.g. scheme://domain.com/path?query
  • headers - The request headers as a list of two-item lists.
  • body - The request body content.

Only method and uri are required. Headers are not strictly required, not even Content-Length as Zurl will set that header for you. If body is unspecified, it is assumed to be empty. If you are using a REQ socket to speak with Zurl, then you can probably get away with having no id field. However, if you use DEALER for multiplexing, then you'll need to ID your requests in order to be able to match them to responses.

Additional request fields:

  • user-data - Arbitrary data to be echoed back in the response message. It's a handy way to ship off state with the request, if the response will be handled by a separate process.
  • max-size - Don't accept a response body larger than this value.
  • connect-host - Override the host to connect to. The outgoing Host header will still be derived from the URI.
  • connect-port - Override the port to connect to.
  • ignore-policies - Ignore any rules about what requests are allowed (i.e. bypass Zurl's allow/deny rules).
  • ignore-tls-errors - Ignore the certificate of the server when using HTTPS or WSS.
  • follow-redirects - If a 3xx response code with a Location header is received, follow the redirect (up to 8 redirects before failing).
  • timeout - Maximum time in milliseconds for the entire request/response operation.

Responses may have the following fields:

  • id - The ID of the request.
  • type - Either ommitted or with value error, meaning the request failed in some way.
  • condition - In an error response, this is a short, machine-parsable string indicating the reason for the error.
  • code - The HTTP status code.
  • reason - The HTTP status reason (e.g. "OK").
  • headers - The response headers as a list of two-item lists.
  • body - The response body content.
  • user-data - If this field was specified in the request, then it will be included in the response.

Sockets

For basic usage, connect to Zurl's request-based interface using a REQ socket (ipc:///tmp/zurl-req by default, see your zurl.conf). To make a request, send a message over the socket. To receive the response, read from the socket.

For advanced usage you can connect to Zurl's streaming interface using PUSH, ROUTER, and SUB sockets. See tools/getstream.py as an example or check out the ZHTTP draft spec for details.

WebSockets

Creating a WebSocket connection through Zurl uses a variant of the ZHTTP protocol. Zurl's streaming interface must be used in this case. The protocol is not documented yet, but you can see tools/wsecho.py as an example.

zurl's People

Contributors

deweerdt avatar jannic avatar jkarneges avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

zurl's Issues

ability to check cert against overridden connect host

This can be useful if a server provides a valid certificate for the domain the client physically connected to rather than the logical domain provided in the URL.

There are a few ways we could try to support this:

  • An option to check cert against the specified connect host (or both the URI host and connect host).
  • An option to specify the host string to check against.
  • Always check against the URI host and connect host, without adding a new option.

zurl rejects OPTIONS requests with a body against spec

Per rfc2616: If the OPTIONS request includes an entity-body (as indicated by the presence of Content-Length or Transfer-Encoding), then the media type MUST be indicated by a Content-Type field. Although this specification does not define any use for such a body, future extensions to HTTP might use the OPTIONS body to make more detailed queries on the server. A server that does not support such an extension MAY discard the request body.

Compilation fails with zmq 3

Using libzmq 3.2.4 on Ubuntu 12.04. Works fine with 2.2.0.

$ configure
Configuring Zurl ...
Verifying Qt 4 build environment ... ok
Checking for Qt >= 4.7.0 ... yes
Checking for libcurl >= 7.20 ... yes
Checking for libzmq >= 2.0 ... yes
Checking for qjson ... yes

Good, your configure finished.  Now run /usr/bin/make.

$ make
cd src/pro/libzurl/ && /usr/bin/qmake-qt4 /home/mmior/apps/zurl/src/pro/libzurl/libzurl.pro -o Makefile
cd src/pro/libzurl/ && make -f Makefile 
make[1]: Entering directory `/home/mmior/apps/zurl/src/pro/libzurl'
gcc -c -m64 -pipe -O2 -fPIC -Wall -W -D_REENTRANT -DQT_WEBKIT -DNO_IRISNET -DUSE_CURL -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtNetwork -I/usr/include/qt4 -I/usr/include/qt4 -I../../../src -I../../jdns -I../../jdnsshared -I../../qzmq/src -I../../common -I_moc -o _obj/jdns_util.o ../../jdns/jdns_util.c
gcc -c -m64 -pipe -O2 -fPIC -Wall -W -D_REENTRANT -DQT_WEBKIT -DNO_IRISNET -DUSE_CURL -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtNetwork -I/usr/include/qt4 -I/usr/include/qt4 -I../../../src -I../../jdns -I../../jdnsshared -I../../qzmq/src -I../../common -I_moc -o _obj/jdns_packet.o ../../jdns/jdns_packet.c
gcc -c -m64 -pipe -O2 -fPIC -Wall -W -D_REENTRANT -DQT_WEBKIT -DNO_IRISNET -DUSE_CURL -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtNetwork -I/usr/include/qt4 -I/usr/include/qt4 -I../../../src -I../../jdns -I../../jdnsshared -I../../qzmq/src -I../../common -I_moc -o _obj/jdns_mdnsd.o ../../jdns/jdns_mdnsd.c
gcc -c -m64 -pipe -O2 -fPIC -Wall -W -D_REENTRANT -DQT_WEBKIT -DNO_IRISNET -DUSE_CURL -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtNetwork -I/usr/include/qt4 -I/usr/include/qt4 -I../../../src -I../../jdns -I../../jdnsshared -I../../qzmq/src -I../../common -I_moc -o _obj/jdns_sys.o ../../jdns/jdns_sys.c
gcc -c -m64 -pipe -O2 -fPIC -Wall -W -D_REENTRANT -DQT_WEBKIT -DNO_IRISNET -DUSE_CURL -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtNetwork -I/usr/include/qt4 -I/usr/include/qt4 -I../../../src -I../../jdns -I../../jdnsshared -I../../qzmq/src -I../../common -I_moc -o _obj/jdns.o ../../jdns/jdns.c
g++ -c -m64 -pipe -O2 -fPIC -Wall -W -D_REENTRANT -DQT_WEBKIT -DNO_IRISNET -DUSE_CURL -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtNetwork -I/usr/include/qt4 -I/usr/include/qt4 -I../../../src -I../../jdns -I../../jdnsshared -I../../qzmq/src -I../../common -I_moc -o _obj/qjdns_sock.o ../../jdns/qjdns_sock.cpp
/usr/bin/moc-qt4 -DQT_WEBKIT -DNO_IRISNET -DUSE_CURL -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtNetwork -I/usr/include/qt4 -I/usr/include/qt4 -I../../../src -I../../jdns -I../../jdnsshared -I../../qzmq/src -I../../common -I_moc ../../jdns/qjdns.cpp -o _moc/qjdns.moc
g++ -c -m64 -pipe -O2 -fPIC -Wall -W -D_REENTRANT -DQT_WEBKIT -DNO_IRISNET -DUSE_CURL -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtNetwork -I/usr/include/qt4 -I/usr/include/qt4 -I../../../src -I../../jdns -I../../jdnsshared -I../../qzmq/src -I../../common -I_moc -o _obj/qjdns.o ../../jdns/qjdns.cpp
In file included from /usr/include/qt4/QtCore/QtCore:92:0,
                 from ../../jdns/qjdns.h:29,
                 from ../../jdns/qjdns.cpp:24:
/usr/include/qt4/QtCore/qtconcurrentfilter.h: In function 'QtConcurrent::ThreadEngineStarter<void> QtConcurrent::filterInternal(Sequence&, KeepFunctor, ReduceFunctor)':
/usr/include/qt4/QtCore/qtconcurrentfilter.h:108:47: warning: typedef 'Iterator' locally defined but not used [-Wunused-local-typedefs]
     typedef typename Sequence::const_iterator Iterator;
                                               ^
/usr/bin/moc-qt4 -DQT_WEBKIT -DNO_IRISNET -DUSE_CURL -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtNetwork -I/usr/include/qt4 -I/usr/include/qt4 -I../../../src -I../../jdns -I../../jdnsshared -I../../qzmq/src -I../../common -I_moc ../../jdnsshared/jdnsshared.cpp -o _moc/jdnsshared.moc
g++ -c -m64 -pipe -O2 -fPIC -Wall -W -D_REENTRANT -DQT_WEBKIT -DNO_IRISNET -DUSE_CURL -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtNetwork -I/usr/include/qt4 -I/usr/include/qt4 -I../../../src -I../../jdns -I../../jdnsshared -I../../qzmq/src -I../../common -I_moc -o _obj/jdnsshared.o ../../jdnsshared/jdnsshared.cpp
In file included from /usr/include/qt4/QtCore/QtCore:92:0,
                 from ../../jdns/qjdns.h:29,
                 from ../../jdnsshared/jdnsshared.h:24,
                 from ../../jdnsshared/jdnsshared.cpp:36:
/usr/include/qt4/QtCore/qtconcurrentfilter.h: In function 'QtConcurrent::ThreadEngineStarter<void> QtConcurrent::filterInternal(Sequence&, KeepFunctor, ReduceFunctor)':
/usr/include/qt4/QtCore/qtconcurrentfilter.h:108:47: warning: typedef 'Iterator' locally defined but not used [-Wunused-local-typedefs]
     typedef typename Sequence::const_iterator Iterator;
                                               ^
g++ -c -m64 -pipe -O2 -fPIC -Wall -W -D_REENTRANT -DQT_WEBKIT -DNO_IRISNET -DUSE_CURL -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtNetwork -I/usr/include/qt4 -I/usr/include/qt4 -I../../../src -I../../jdns -I../../jdnsshared -I../../qzmq/src -I../../common -I_moc -o _obj/qzmqcontext.o ../../qzmq/src/qzmqcontext.cpp
/usr/bin/moc-qt4 -DQT_WEBKIT -DNO_IRISNET -DUSE_CURL -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtNetwork -I/usr/include/qt4 -I/usr/include/qt4 -I../../../src -I../../jdns -I../../jdnsshared -I../../qzmq/src -I../../common -I_moc ../../qzmq/src/qzmqsocket.cpp -o _moc/qzmqsocket.moc
g++ -c -m64 -pipe -O2 -fPIC -Wall -W -D_REENTRANT -DQT_WEBKIT -DNO_IRISNET -DUSE_CURL -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++-64 -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtNetwork -I/usr/include/qt4 -I/usr/include/qt4 -I../../../src -I../../jdns -I../../jdnsshared -I../../qzmq/src -I../../common -I_moc -o _obj/qzmqsocket.o ../../qzmq/src/qzmqsocket.cpp
../../qzmq/src/qzmqsocket.cpp: In function 'int QZmq::get_hwm(void*)':
../../qzmq/src/qzmqsocket.cpp:106:33: error: 'ZMQ_HWM' was not declared in this scope
  int ret = zmq_getsockopt(sock, ZMQ_HWM, &hwm, &opt_len);
                                 ^
../../qzmq/src/qzmqsocket.cpp: In function 'void QZmq::set_hwm(void*, int)':
../../qzmq/src/qzmqsocket.cpp:115:33: error: 'ZMQ_HWM' was not declared in this scope
  int ret = zmq_setsockopt(sock, ZMQ_HWM, &v, opt_len);
                                 ^
../../qzmq/src/qzmqsocket.cpp: In member function 'void QZmq::Socket::Private::write(const QList<QByteArray>&)':
../../qzmq/src/qzmqsocket.cpp:282:89: error: too few arguments to function 'int zmq_send(void*, const void*, size_t, int)'
     ret = zmq_send(sock, &msg, ZMQ_NOBLOCK | (n + 1 < message.count() ? ZMQ_SNDMORE : 0));
                                                                                         ^
In file included from ../../qzmq/src/qzmqsocket.cpp:32:0:
/usr/include/zmq.h:349:16: note: declared here
 ZMQ_EXPORT int zmq_send (void *s, const void *buf, size_t len, int flags);
                ^
../../qzmq/src/qzmqsocket.cpp: In member function 'bool QZmq::Socket::Private::tryWrite(int*)':
../../qzmq/src/qzmqsocket.cpp:337:84: error: too few arguments to function 'int zmq_send(void*, const void*, size_t, int)'
    ret = zmq_send(sock, &msg, ZMQ_NOBLOCK | (message.count() > 1 ? ZMQ_SNDMORE : 0));
                                                                                    ^
In file included from ../../qzmq/src/qzmqsocket.cpp:32:0:
/usr/include/zmq.h:349:16: note: declared here
 ZMQ_EXPORT int zmq_send (void *s, const void *buf, size_t len, int flags);
                ^
../../qzmq/src/qzmqsocket.cpp: In member function 'bool QZmq::Socket::Private::tryRead(bool*)':
../../qzmq/src/qzmqsocket.cpp:373:32: error: too few arguments to function 'int zmq_recv(void*, void*, size_t, int)'
    ret = zmq_recv(sock, &msg, 0);
                                ^
In file included from ../../qzmq/src/qzmqsocket.cpp:32:0:
/usr/include/zmq.h:350:16: note: declared here
 ZMQ_EXPORT int zmq_recv (void *s, void *buf, size_t len, int flags);
                ^
make[1]: *** [_obj/qzmqsocket.o] Error 1
make[1]: Leaving directory `/home/mmior/apps/zurl/src/pro/libzurl'
make: *** [sub-src-pro-libzurl-make_default] Error 2

crash on mac

[DEBUG] 2019-03-28 20:06:22.721 recv-init: { "credits": 200000, "seq": 0, "id": "a10728ef-4a6c-4223-a471-9b8e46a39a69", "headers": [ [ "User-Agent", "Pushpin-Updater" ] ], "method": "GET", "from": "pushpin-proxy_89809", "ignore-policies": true, "uri": "https://updates.fanout.io/check/?package=pushpin&version=1.20.1&os=mac&arch=64&id=...&cmax=1&cminutes=15&recv=12964&sent=12964&ops=25943", "stream": true, "ignore-tls-errors": true, "ext": { "multi": true } }
[INFO] 2019-03-28 20:06:22.721 IN id=a10728ef-4a6c-4223-a471-9b8e46a39a69, GET https://updates.fanout.io/check/?package=pushpin&version=1.20.1&os=mac&arch=64&id=...&cmax=1&cminutes=15&recv=12964&sent=12964&ops=25943
[DEBUG] 2019-03-28 20:06:22.722 timerFunction: wake up in 0ms
[DEBUG] 2019-03-28 20:06:22.722 curl: Connection 3 seems to be dead!
[DEBUG] 2019-03-28 20:06:22.722 curl: Closing connection 3
[DEBUG] 2019-03-28 20:06:22.722 curl: TLSv1.2 (OUT), TLS alert, Client hello (1):
[DEBUG] 2019-03-28 20:06:22.722 timerFunction: wake up in 0ms
[DEBUG] 2019-03-28 20:06:22.723 timerFunction: wake up in 1ms
[DEBUG] 2019-03-28 20:06:22.723 timerFunction: wake up in 1ms
[DEBUG] 2019-03-28 20:06:22.723 send: { "seq": 0, "id": "a10728ef-4a6c-4223-a471-9b8e46a39a69", "from": "{e6fa89e9-e44f-4bb1-a00d-0beea090db86}", "type": "keep-alive", "ext": { "multi": true } }
[DEBUG] 2019-03-28 20:06:22.724 timerFunction: wake up in 1ms
[DEBUG] 2019-03-28 20:06:22.725 timerFunction: wake up in 1ms
[DEBUG] 2019-03-28 20:06:22.726 timerFunction: wake up in 3ms
[DEBUG] 2019-03-28 20:06:22.730 timerFunction: wake up in 1ms
[DEBUG] 2019-03-28 20:06:22.732 timerFunction: wake up in 7ms
[DEBUG] 2019-03-28 20:06:22.739 timerFunction: wake up in 1ms
[DEBUG] 2019-03-28 20:06:22.740 timerFunction: wake up in 15ms
[DEBUG] 2019-03-28 20:06:22.756 timerFunction: wake up in 1ms
[DEBUG] 2019-03-28 20:06:22.757 timerFunction: wake up in 31ms
[DEBUG] 2019-03-28 20:06:22.789 timerFunction: wake up in 1ms
[DEBUG] 2019-03-28 20:06:22.790 timerFunction: wake up in 63ms
[DEBUG] 2019-03-28 20:06:22.854 timerFunction: wake up in 1ms
[DEBUG] 2019-03-28 20:06:22.856 trying 178.128.129.175
[DEBUG] 2019-03-28 20:06:22.856 curl: Trying 178.128.129.175...
[DEBUG] 2019-03-28 20:06:22.856 curl: TCP_NODELAY set
[DEBUG] 2019-03-28 20:06:22.856 socketFunction: CURL_POLL_OUT 33
[DEBUG] 2019-03-28 20:06:22.856 timerFunction: wake up in 200ms
[DEBUG] 2019-03-28 20:06:23.051 curl: Connected to updates.fanout.io (178.128.129.175) port 443 (#4)
[DEBUG] 2019-03-28 20:06:23.051 curl: ALPN, offering http/1.1
[DEBUG] 2019-03-28 20:06:23.051 curl: Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
[DEBUG] 2019-03-28 20:06:23.054 curl: successfully set certificate verify locations:
[DEBUG] 2019-03-28 20:06:23.054 curl: CAfile: /etc/ssl/cert.pem
CApath: none
Segmentation fault: 11

wsecho example undefined-condition

Hi, I am trying to get the wsecho.py example working but I am running into an issue.

I am running the example like this

python tools/wsecho.py wss://echo.websocket.org

and this is the output that I am receiving

OUT: {'credits': 200000, 'from': 'wsecho.py', 'id': '91eaff57-22a4-437b-adf4-dc87370128fb', 'seq': 0, 'uri': 'wss://echo.websocket.org'}
IN: {'from': '{e38917b3-369f-419f-90f3-fb75300dc714}', 'type': 'error', 'id': '91eaff57-22a4-437b-adf4-dc87370128fb', 'condition': 'undefined-condition', 'seq': 0L}

make trust-connect-host work on mac without openssl

The trust-connect-host feature relies on mucking with curl's openssl context. On Mac this feature doesn't work, because the system curl is not built with openssl, and Homebrew's default version of curl is not built with openssl either. Ideally there would be a way to do a custom cert host check using curl and the Mac security framework.

Support py3.12

While trying to migrate zurl to test against py3.12, ran into the following test failure

    zmq/backend/cython/_device.c: In function ‘__Pyx_Raise’:
    zmq/backend/cython/_device.c:4372:34: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
     4372 |         PyObject* tmp_tb = tstate->curexc_traceback;
          |                                  ^~
    zmq/backend/cython/_device.c:4375:19: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘curexc_traceback’
     4375 |             tstate->curexc_traceback = tb;
          |                   ^~
    zmq/backend/cython/_device.c: In function ‘__Pyx_PyInt_As_int’:
    zmq/backend/cython/_device.c:5408:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
     5408 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
          |                                                     ^~
    zmq/backend/cython/_device.c:5463:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
     5463 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
          |                                                     ^~
    zmq/backend/cython/_device.c: In function ‘__Pyx_PyInt_As_long’:
    zmq/backend/cython/_device.c:5680:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
     5680 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
          |                                                     ^~
    zmq/backend/cython/_device.c:5735:53: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
     5735 |             const digit* digits = ((PyLongObject*)x)->ob_digit;
          |                                                     ^~
    zmq/backend/cython/_device.c: In function ‘__Pyx_PyIndex_AsSsize_t’:
    zmq/backend/cython/_device.c:6086:45: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
     6086 |     const digit* digits = ((PyLongObject*)b)->ob_digit;
          |                                             ^~
    ************************************************
    ************************************************
    error: command '/usr/bin/gcc-11' failed with exit code 1
    error: subprocess-exited-with-error

log, https://github.com/Homebrew/homebrew-core/actions/runs/6450154450/job/17509259440?pr=150232
relates to Homebrew/homebrew-core#150232

Custom headers

I love the idea of this project, but I ran into issues sending custom headers. Which is required for my use case to use against AWS API.

Whenever I send a header which isn't "Content-Length" in Json format, all I get is errors, from zurl itself.

Here's the log message

[DEBUG] 2015-06-09 10:57:38.337 recv-init: { "body": "Action=DescribeCacheEngineVersions&Version=2015-02-02", "method": "POST", "uri": "http://elasticache.us-west-1.amazonaws.com/", "from": "cw2miner", "headers": [ [ <unknown>, <unknown> ] ], "id": "80b4902c-db42-4637-ab40-fed64271d625" }
[WARN] 2015-06-09 10:57:38.337 failed to parse zurl request
[INFO] 2015-06-09 10:57:38.337 OUT ERR id=80b4902c-db42-4637-ab40-fed64271d625 condition=bad-request
[DEBUG] 2015-06-09 10:57:38.338 send: { "condition": "bad-request", "seq": 0, "from": "{758d5129-3c5b-4901-8549-44e5526b222c}", "type": "error", "id": "80b4902c-db42-4637-ab40-fed64271d625" }

I would've hacked it on my own, but unfortunately I don't see an easy way to allow custom headers.
Thanks

BTW, the headers look roughly like(Scrambled a bit for confidentiality, but exact values aren't necessary to replicate)

x-amz-date: 20150609T151035Z
Authorization: AWS4-HMAC-SHA256 Credential=AKIBIUAXWJG2ZX736A3A/23170412/sa-east-1/elasticache/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=184663e3d0cac180bc28c8b4261701a9ed9083f069e3847094c375cb7d0fab84

Need help to fix zurl usage with curl 7.81.0

Hi. Homebrew maintainer here.

We are upgrading the curl package from 7.80.0 to 7.81.0.
We have a sanity check for zurl, which is now failing.

Here is the ruby code we run to test zurl (should be sort of readable):

test do
    conffile = testpath/"zurl.conf"
    ipcfile = testpath/"zurl-req"
    runfile = testpath/"test.py"

    resource("pyzmq").stage do
      system Formula["[email protected]"].opt_bin/"python3", *Language::Python.setup_install_args(testpath/"vendor")
    end

    conffile.write(<<~EOS,
      [General]
      in_req_spec=ipc://#{ipcfile}
      defpolicy=allow
      timeout=10
    EOS
                  )

    port = free_port
    runfile.write(<<~EOS,
      import json
      import threading
      from http.server import BaseHTTPRequestHandler, HTTPServer
      import zmq
      class TestHandler(BaseHTTPRequestHandler):
        def do_GET(self):
          self.send_response(200)
          self.end_headers()
          self.wfile.write(b'test response\\n')
      def server_worker(c):
        server = HTTPServer(('', #{port}), TestHandler)
        c.acquire()
        c.notify()
        c.release()
        try:
          server.serve_forever()
        except:
          server.server_close()
      c = threading.Condition()
      c.acquire()
      server_thread = threading.Thread(target=server_worker, args=(c,))
      server_thread.daemon = True
      server_thread.start()
      c.wait()
      c.release()
      ctx = zmq.Context()
      sock = ctx.socket(zmq.REQ)
      sock.connect('ipc://#{ipcfile}')
      req = {'id': '1', 'method': 'GET', 'uri': 'http://localhost:#{port}/test'}
      sock.send_string('J' + json.dumps(req))
      poller = zmq.Poller()
      poller.register(sock, zmq.POLLIN)
      socks = dict(poller.poll(15000))
      assert(socks.get(sock) == zmq.POLLIN)
      resp = json.loads(sock.recv()[1:])
      assert('type' not in resp)
      assert(resp['body'] == 'test response\\n')
    EOS
                 )

    pid = fork do
      exec "#{bin}/zurl", "--config=#{conffile}"
    end

    begin
      xy = Language::Python.major_minor_version Formula["[email protected]"].opt_bin/"python3"
      ENV["PYTHONPATH"] = testpath/"vendor/lib/python#{xy}/site-packages"
      system Formula["[email protected]"].opt_bin/"python3", runfile
    ensure
      Process.kill("TERM", pid)
      Process.wait(pid)
    end

Here is the test output with the old curl:

==> /home/linuxbrew/.linuxbrew/opt/[email protected]/bin/python3 -c import setuptools... --no-user-cfg install --prefix=/tmp/zurl-test-20220107-33874-19g1kvq/vendor --install-scripts=/tmp/zurl-test-20220107-33874-19g1kvq/vendor/bin --single-version-externall
==> /home/linuxbrew/.linuxbrew/opt/[email protected]/bin/python3 /tmp/zurl-test-20220107-33874-19g1kvq/test.py
[INFO] 2022-01-07 22:17:18.718 starting...
[INFO] 2022-01-07 22:17:18.731 started
[INFO] 2022-01-07 22:17:18.781 IN id=1, GET http://localhost:41917/test
[INFO] 2022-01-07 22:17:18.786 OUT id=1 code=200 14
[INFO] 2022-01-07 22:17:18.811 stopping...
[INFO] 2022-01-07 22:17:18.811 stopped

And the result with the newest curl:

==> /home/linuxbrew/.linuxbrew/opt/[email protected]/bin/python3 /tmp/zurl-test-20220105-1038056-fd89o3/test.py
[INFO] 2022-01-05 17:41:58.201 starting...
[INFO] 2022-01-05 17:41:58.205 started
[INFO] 2022-01-05 17:41:58.299 IN id=1, GET http://localhost:43513/test
Traceback (most recent call last):
  File "/tmp/zurl-test-20220105-1038056-fd89o3/test.py", line 34, in <module>
    assert(socks.get(sock) == zmq.POLLIN)
AssertionError
Error: zurl: failed
An exception occurred within a child process:
  BuildError: Failed executing: /home/linuxbrew/.linuxbrew/opt/[email protected]/bin/python3 /tmp/zurl-test-20220105-1038056-fd89o3/test.py

It looks like socks is an empty dict, so get returns None.

We are no zurl experts so any help is welcome.

For reference: Homebrew/homebrew-core#92549

QT 6 support

Hi

I am a maintainer of Homebrew, a package manager for macOS.

I was wondering if zurl could be made compatible with qt6? We need to slowly think about phasing QT 5 out.

==> ./configure --prefix=/usr/local/Cellar/zurl/1.11.1_1 --extraconf=QMAKE_MACOSX_DEPLOYMENT_TARGET=13
Configuring Zurl ...
Verifying Qt build environment ... fail

Reason: Unable to find the 'qmake' tool for Qt 4 or 5.

Be sure you have a proper Qt 4.0+ build environment set up.  This means not
just Qt, but also a C++ compiler, a make tool, and any other packages
necessary for compiling C++ programs.

Would it be possible to check if this works with qt 6 and make a new release? Thanks

Configure failure with Qt >= 5.10

The failure is caused by

zurl/configure

Line 195 in 5d738f1

?.?.?)

?.?.? fails to match a two-digit minor version. So any version > 5.9 will fail.

The error is

==> ./configure --prefix=/usr/local/Cellar/zurl/1.9.0_1 --extraconf=QMAKE_MACOSX_DEPLOYMENT_TARGET=10.11
Configuring Zurl ...
Verifying Qt build environment ... fail

Reason: Unable to find the 'qmake' tool for Qt 4 or 5.

Be sure you have a proper Qt 4.0+ build environment set up.  This means not
just Qt, but also a C++ compiler, a make tool, and any other packages
necessary for compiling C++ programs.

If you are certain everything is installed, then it could be that Qt is not
being recognized or that a different version of Qt is being detected by
mistake (for example, this could happen if $QTDIR is pointing to a Qt 3
installation).  At least one of the following conditions must be satisfied:

 1) --qtdir is set to the location of Qt
 2) $QTDIR is set to the location of Qt
 3) QtCore is in the pkg-config database
 4) qmake is in the $PATH

This script will use the first one it finds to be true, checked in the above
order.  #3 and #4 are the recommended options.  #1 and #2 are mainly for
overriding the system configuration.

I can work around it by string replacing "?.?.?" with "?.??.?"

posting data example

While reading the post example i have no idea how it works.

Why is this needed?
https://github.com/fanout/zurl/blob/master/tools/post.py#L39-L44
The only thing i see is the data being echoed back which is received
Same here: https://github.com/fanout/zurl/blob/master/tools/post.py#L48-L52
All i see is the seq counter is being increased while echoing the received data back, and in the case of the if 'body' in data and len(data['body']) > 0: line the body which just get send is also echoed back?

What are those credits?

Why isn't the first part of the file send in the initial push send here? https://github.com/fanout/zurl/blob/master/tools/post.py#L26

It's all a bit confusing.

requests keep going to existing persistent connections if IP address changes

If a domain's DNS record is updated to point somewhere else, and Zurl has an existing persistent connection, new requests will go to the original server over the existing persistent connection. If the original server never kills the connection, then requests may be sent to the original server indefinitely.

This could probably be considered an issue in libcurl. From what I can tell there is no straightforward solution.

A potential workaround may be to use multiple "multi" contexts, if this is possible. A multi context per domain would allow us to disconnect from specific domains, which maybe we could do after detecting DNS changes. If a multi context per domain is inefficient, we could have a two multi contexts (for current requests & old requests) and destroy the old context after a certain time period, with the drawback that long-lived connections end up getting periodically disconnected.

allow non-streaming requests over PUSH/SUB

When using the PUSH/SUB interface, handling HTTP requests that last longer than 60 seconds requires session management (remembering request IDs and sending keep alives to Zurl). It might be nice to optionally make the PUSH/SUB interface behave similar to REQ/REP, where there is only a single message for a request and a single message for a response, and no session management to worry about.

Sending data in body

Hello,

I'm trying to send some body data from PHP like this:

$ php zeromq-send.php
Sending
{"method":"POST","uri":"https://xxx.se/delay.php","body":"HELLO"}

But the body content HELLO won't show up in the POST-data on the server. What am I doing wrong?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.