jonathonmcmurray / req

HTTP requests library in kdb+/q

reQ

HTTP requests library in kdb+/q

kdb+ has built in functions for HTTP requests such as .Q.hg (GET) and .Q.hp(POST). However, these functions are somewhat limited by several factors. For example, using these functions you cannot supply custom HTTP headers within the requests (for example, authorization tokens that are required by many APIs, a user agent etc.). In addition, in case of an HTTP redirect response, .Q.hg/.Q.hp will fail.

reQ is a library designed to overcome these limitations for basic HTTP queries. The main features are:

• Custom headers
• Automatic following of HTTP redirects
• Automatic parsing of JSON responses to KDB+ object
• Support for lower versions of KDB+ (tested on 2.7 & 3.3)
• Cookie support

For more details on features & usage, please see the docs

Getting started

Standalone .q script

The simplest way for most people to get started with reQ will be via the standalone .q file available from the Releases tab of this repo. Download the .q script from the latest release & this can be loaded directly with \l when placed in project directory or QHOME etc.

q)\l req_0.1.3.q
q).req.g"https://httpbin.org/get"
args   | (`symbol$())!()
headers| `Accept`Host`User-Agent`X-Amzn-Trace-Id!("*/*";"httpbin.org";"kdb+/3..
origin | "90.249.33.209"
url    | "https://httpbin.org/get"

qutil package via Anaconda

Alternatively, reQ can be installed as a qutil via Anaconda/Miniconda

Assuming conda is installed, installation command is:

$ conda install -c jmcmurray req

And package will then be available in q via:

q).utl.require"req"

Standalone qutil package

If you want to use qutil package, but not via Anaconda, you can also download the package as a .tar.gz from the Releases tab & extract to your QPATH directory.

Updating documentation with qDoc

Using qDoc from Kx (included with Kx Developer)

$ # -norecurse so as not to go into ext/ dir
$ q $DEVELOPER_HOME/ws/qdoc.q_ -src req/ -out docs/api/ -norecurse
..
$ mkdocs serve # to preview locally
..
$ mkdocs gh-deploy # to deploy to github

Linting with qlint

Uses qlint.q_ from Kx (included with Kx Developer). Assumes DEVELOPER_HOME env var is set

$ q lint.q

Loads package & lints .req & related namespaces. Some errors/warnings excluded. Adding an additional arg on cmd line will halt process to inspect output table t.

Building releases

To build a release, build.sh is used. The script creates the unified, standalone .q script & the .tar.gz of qutil package.

Argument is version number e.g.

$ ./build.sh 0.1.3

req's Issues

Improve verbose output

In verbose mode, check if body ends with \n & add one if not

Add DELETE functions

Add a set of functions to issue DELETE requests, with projections for no body and no custom header scenarios

Make default behaviours more configurable

Make it possible to disable parsing with a flag e.g. .req.PARSE (similar to .req.VERBOSE flag).

Also .req.SIGNAL for controlling signalling if HTTP error response received.

Possibly also allow defining a "custom parse" function that receives headers & body and can do whatever necessary.

Also, for .req.VERBOSE allow defining as an int i.e. a handle to write to, so it's possible to configure this to write to stderr instead of stdout etc. - ensure backwards compatibility, cast booleans to int

Cookie tailmatching

Currently, the reQ cookie functionality will perform "tailmatching" for any cookie received with a Host attribute. For example, if a cookie is received with Host=example.com, it will (and should) match e.g. good.example.com subdomain. However, it will also match badexample.com, which is incorrect.

cURL suffered from a similar bug a few years back: https://daniel.haxx.se/blog/2013/04/15/tailmatching-them-cookies/

Read login credentials from .netrc file

Read ~/.netrc for login credentials if not passed in URL or cached in q session

https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html
https://httpie.org/doc#netrc

Should fall back to ~/.netrc in .auth.getcache if no value found in cache. Should also be a variable to disable e.g. .auth.usenetrc:1b by default

Add support for multipart formposts

Add support for creating & sending multipart formpost requests (required for, for example, uploading files to Slack: https://api.slack.com/methods/files.upload )

Relevant links:

https://tools.ietf.org/html/rfc7578
https://ec.haxx.se/http-multipart.html

Improve usability of .req.post

Currently it's a little cumbersome to pass e.g. a JSON body to a request. We should accept something like

.req.post[url;`json;`a`b`c!1 2 3]

to auto-convert to JSON & put correct Content-Type

Need to consider how it should work with custom headers required as well

Add support for DNS-over-HTTPS

DNS-over-HTTPS provides the functionality to perform DNS lookups over HTTPS rather than in plaintext, giving increased security.

APIs for this are provided by Google and CloudFlare, support should be added for these (disabled by default)

Google: https://developers.google.com/speed/public-dns/docs/dns-over-https#api_specification
CloudFlare: https://developers.cloudflare.com/1.1.1.1/dns-over-https/json-format/

CloudFlare also provide a version accepting & returning DNS wireformat rather than JSON, support for that could also be added: https://developers.cloudflare.com/1.1.1.1/dns-over-https/wireformat/

• Google JSON
• CloudFlare JSON
• CloudFlare wireformat
• TTL/cache invalidation
• Fallback to system DNS

RFCs

https://tools.ietf.org/html/rfc7858
https://tools.ietf.org/html/rfc8310
https://tools.ietf.org/html/rfc8484

Parse URLs without a forward slash before query

Currently, URL parsing splits hostname from path by searching for /

e.g. foo://example.com:8042/over/there?name=ferret

• here, the path (including query) is /over/there?name=ferret

In rare circumstances, it could be possible that a URL has no "path" but does have a query e.g.

foo://example.com:8042?name=ferret

In current implementation, a / is required before the ? to properly parse the path as /?name=ferret.

Should handle this automatically & insert leading / to path if necessary

Move decompression to before display for verbose mode

Currently, decompression (i.e. gzip decompression in kdb 4.0+) happens in .req.parseresp, meaning that verbose mode shows compressed response.

Should be done in .req.send as soon as response is received.

Should also make it simple to plug in additional handlers for different Content-Encoding types, probably via a dictionary of handlers that can be extended by users.

./

Should use .req.ty when checking type of response

https://github.com/jonathonmcmurray/reQ/blob/master/req.q#L179

:$[(`j in key`)&r[0][`$"Content-Type"]like .h.ty[`json],"*";.j.k;] r[1];

In this line we use .h.ty to check type of response, but we have .req.ty created by reQ which should be used instead (in case of other modules modifying .h.ty etc. e.g. serve from https://github.com/jonathonmcmurray/qutil_packages )

Restructure library into component parts

Separate out separate components (e.g. cookies) into separate q scripts

Create init.q for loading via qutil (.utl.pkg each component) & req.q for loading in vanilla q (\l each component)

• Separate scripts for each component
• init.q script
• req.q script
• build script - not everyone uses qutil, make an easy to load "monoscript" compiled from others

.req.enchd does not accept non-string values

Currently, .req.enchd is broken for non-string values or keys:

q).req.enchd`abc`def!("abc";"def")
'type
  [1]  /home/jmcmurray/git/reQ/req.q:113: .req.enchd:
  /* convert KDB dictionary to HTTP headers */
  :("\r\n" sv ": " sv/:flip (key;value)@\:d),"\r\n\r\n";                            //encode headers dict to HTTP headers
                     ^
 }
q.req))\
q).req.enchd("abc";"def")!"ab"
'type
  [1]  /home/jmcmurray/git/reQ/req.q:113: .req.enchd:
  /* convert KDB dictionary to HTTP headers */
  :("\r\n" sv ": " sv/:flip (key;value)@\:d),"\r\n\r\n";                            //encode headers dict to HTTP headers
                     ^
 }
q.req))\
q).req.enchd("abc";"def")!1 2
'type
  [1]  /home/jmcmurray/git/reQ/req.q:113: .req.enchd:
  /* convert KDB dictionary to HTTP headers */
  :("\r\n" sv ": " sv/:flip (key;value)@\:d),"\r\n\r\n";                            //encode headers dict to HTTP headers
                     ^
 }
q.req))

All of these should work, with keys/values being converted to strings if necessary

Currently keys will be converted in .req.headers if all keys are symbols (which should never happen given default headers are specified in .req.def with string keys)

Add caching support

HTTP clients should implement caching as per RFC 7234

The goal of caching in HTTP/1.1 is to significantly improve
performance by reusing a prior response message to satisfy a current
request.

Caching results makes our "requests" faster & reduces load on the server. Should handle Cache-Control & Expires response headers & cache if these are present.

Should also support If-Modified-Since & ETag on HTTP requests, handling the subsequent 304 HTTP status code for unchanged content

• Define cache in-memory object
• Cache responses with Cache-Control & Expires headers
• Detect requests with a cached response & use it
• If-Modified-Since & ETag support
• Handle 304 status code

Update Advent of Code example to use cookiejar

The current Advent of Code example manually adds the Cookie header to each request, this should be changed to add the cookie to .req.cookiejar and let it be sent automatically with requests

"Protocol not available" error over https

First off, thanks so much for the reQ package!

I tried to follow your blog article and get this error when trying https protocol. http works fine.

q).utl.require"req"
q).req.g"https://httpbin.org/get"
'conn. OS reports: Protocol not available
  [2]  /opt/miniconda3/envs/build/q/pkg/req/req.q:164: .req.send:
  d:headers[us;pr;hd;p];                                                            //get dictionary of HTTP headers for request
  r:hs d:buildquery[m;pr;nu;h;d;p];                                                 //build query and execute
    ^
  if[v;-1"-- REQUEST --\n",string[hs],"\n",d];                                      //if verbose, log request
q.req))
q).req.g"http://httpbin.org/get"
args   | (`symbol$())!()
headers| `Accept`Connection`Host`User-Agent!("*/*";"close";"httpbin.org";"kdb..
origin | "208.110.70.170"
url    | "http://httpbin.org/get"

I think I installed all dependencies:

(build) [dk@s110519 /home/dk/conda-pkgs]$ conda list
# packages in environment at /opt/miniconda3/envs/build:
#
# Name                    Version                   Build  Channel
ca-certificates           2018.03.07                    0  
json                      0.1.1                         0    file://opt/channel
q                         3.6                           0    file://opt/channel
qhttps                    0.0.1                         0    local
qlic                      2018.11.25                    0    file://opt/channel
qutil                     3.2.1                         0    file://opt/channel
req                       0.1.2                         0    file://opt/channel

Add qutil package

Create a qutil package for simple use of library

Add support for timeouts

Most HTTP clients implement timeout functionality for slow responding servers. In q, the -T cmd line arg or \T command can set a timeout for remotely executed commands, which we can use to timeout a local command by sending it to ourselves over handle 0 e.g.

q)\T 1
q)0(`.req.g;"http://httpbin.org/delay/10")
'stop
  [2]  /home/jmcmurray/miniconda3/q/packages/req/req.q:164: .req.send:
  d:headers[us;pr;hd;p];                                                            //get dictionary of HTTP headers for request
  r:hs d:buildquery[m;pr;nu;h;d;p];                                                 //build query and execute
    ^
  if[v;-1"-- REQUEST --\n",string[hs],"\n",d];                                      //if verbose, log request
q.req))

As an initial version, I'll implement a wrapper that

(i) sets timeout
(ii) performs request (with error trap) over handle 0
(iii) resets timeout
(iv) signals if timeout was hit

Future rewrites may allow a more sophisticated approach.

.req.hap fails if there is an @ in URL path

For example:

.req.hap"wss://stream.binance.com:9443/ws/btcusdt@depth"
"wss://"
"stream.binance.com:9443/ws/btcusdt"
"depth"
,"/"

host should be stream.binance.com:9443 and path /ws/btcusdt@depth

.req.auth broken on 3.3 and below

With 01ed1d6 backwards compatibility was added for older versions of q (.Q.k>=2.8 tested). However, .req.auth (i.e. base64 encode) was not properly tested and this is broken on .Q.k<=3.3 e.g. on 3.2

λ q req.q
KDB+ 3.2 2015.04.07 Copyright (C) 1993-2015 Kx Systems
w32/ 4()core 4095MB jonat laptop-o8a8co1o 169.254.46.92 NONEXPIRE

q).req.auth "user:pass"
k){x\:y}
'nyi
@
64\:
7697253
q.q))

.req.b64decode is also broken:

q).req.b64decode "aGVsbG8gd29ybGQ="
k){x\:y}
'nyi
@
256\:
6841708
q.q))

Add GitHub issue example

GitHub API allows addition of issues, should add this to the GitHub example script.

Record permanent redirects

When 301 status code is received, we need to log that this is a permanent URL change, as opposed to temporary with 302.

So we should keep a record when we receive 301s and automatically substitute replaced URLs in future HTTP requests

• Record permanent redirects
• Substitute URLs as necessary in future requests

Ignore case in HTTP response headers

In several places in code (e.g. setting cookies, redirections, content parsing) we rely on response headers having Title-Case - if headers are instead sent lower-case, code doesn't work correctly.

Simple fix is to convert all headers to lower case when received & reference them in this way throughout

Add .req.params function for adding parameters to URL

Add function to more easily construct a URL with parameters

e.g.

q).req.params[url;`param1`param2!(123;"abc")]
"{url}?param1=123&param2=abc"

Cache basic authentication credentials

As of 7750c99 reQ supports prompting for Basic auth credentials where server requests them.

In order to prevent re-prompting, reQ should cache the entered credentials for automatic use in future requests