anmonteiro / piaf

Web library for OCaml with support for HTTP/1.X / HTTP/2

piaf

Piaf is a client library for the HTTP/1.X and HTTP/2 protocols written entirely in OCaml.

Installation

Piaf is released to OPAM.

You can depend on it via esy or by running opam install piaf.

Note: make sure to mirror Piaf's own resolutions located in the opam file.

Usage & Examples

TODO, read the mli file for now.

Examples

open Piaf

let get_sync env ~sw url =
  print_endline "Sending request...";
  match Client.Oneshot.get ~sw env (Uri.of_string url) with
  | Ok response ->
    if Status.is_successful response.status
    then Body.to_string response.body
    else
      let message = Status.to_string response.status in
      Error (`Msg message)
  | Error e -> failwith (Error.to_string e)

let () =
  Eio_main.run (fun env ->
      Eio.Switch.run (fun sw ->
          match get_sync env ~sw "https://example.com" with
          | Ok body -> print_endline body
          | Error error ->
            let message = Error.to_string error in
            prerr_endline ("Error: " ^ message)))

There's a more substantive example of using Piaf's API in bin/carl.ml, an implementation of a subset of curl, in caml.

Development

There's two ways to get a development environemnt up and running. If you have (or don't mind getting) nix installed, the repository includes scripts to set up a sandbox. Otherwise you can use opam to install the necessary dependencies globally.

Option 1) Setting up the sandbox

Assuming nix has been installed and set up, run nix develop -c $SHELL in the repository root. Once it's done building, you should have the development environment set up!

Option 2) Setting up opam

For this approach you'll need to install opam and set it up with a switch using ocaml >= 4.08. Once that's done, run opam pin . --deps-only to install the dependencies.

Note that this installs the dependencies globally, and that the development environment is dependent on the switch used.

Building

Run dune build to build, dune build --watch to run a watcher daemon that will build incrementally.

Running examples

Run dune exec examples/docs/readme.exe to run the simple example above.

Run dune exec bin/carl.exe to run carl, the curl-like example.

License & Copyright

Copyright (c) 2019 António Nuno Monteiro

piaf is distributed under the 3-Clause BSD License, see LICENSE.

The vendor/multipart_form directory contains a fork of multipart_form which is licensed under the MIT License. multipart_form.LICENSE reproduces the original license text.

piaf's Issues

drain request body when switch is released

Flush headers immediately on the client if Expect: 100

After anmonteiro/httpun#104, we should flush headers immediately if headers contains Expect: 100-continue

Allow to pass cacert in client config as pem string

Right now to pass a certificate to you must provide a file path for the library to parse:

cacert : string option (** The path to a CA certificates file in PEM format *)

If I had the certificate already parsed I would need to write a file. It would be great if I could pass the string of the certificate as a config

Memory leak with Eio Piaf Server

Problem

I have memory leaks with a Piaf Server as a Proxy.
I already reproduced it on prod with a Linux x86 and locally with a macOS ARM64.
I suspect that this problem is related only to the Piaf Serve.

How to reproduce

With the new example in PR #146.

Execute to run the server:

dune exec examples/eio/eio_server_get.exe                                                                                                                           

Stress it with some client HTTP tool, for example hey.

hey -n 50000 http://localhost:8080/

Follow the memory with a tool like htop.
Before:

After:

The memory only increases after each request, it seems that there is no GC.

Cancelling forever-lived HTTP request

Currently Piaf doesn't expose a way to stop an HTTP request that lives forever after Body.to_string_stream is called.

Example

https://mainnet.api.tez.ie/monitor/valid_blocks

error when running multiple domains in tests

I'm writing a test that starts a server and uses the Oneshot client to make a request. This works fine when the server runs on a single domain, but when I call Piaf.Server.Command.shutdown with multiple domains, it fails with the error below:

[invalid] accept: file descriptor used after calling close!
          Raised at Eio_unix__Fd.use_exn.(fun) in file "lib_eio/unix/fd.ml", line 20, characters 40-61
          Called from Eio_posix__Err.run in file "lib_eio_posix/err.ml", line 27, characters 6-10
          Called from Eio_posix__Net.Listening_socket.accept in file "lib_eio_posix/net.ml", line 32, characters 30-65
          Called from Eio__Net.accept in file "lib_eio/net.ml" (inlined), line 261, characters 2-16
          Called from Eio__Net.accept_fork in file "lib_eio/net.ml" (inlined), line 265, characters 19-31
          Called from Piaf__Server.Command.accept_loop.(fun) in file "lib/server.ml", line 237, characters 10-661
          Called from Eio__core__Fiber.any.(fun).wrap in file "lib_eio/core/fiber.ml", line 97, characters 16-20
          Re-raised at Eio__core__Fiber.any in file "lib_eio/core/fiber.ml", line 138, characters 26-61
          Called from Eio__core__Fiber.first in file "lib_eio/core/fiber.ml" (inlined), line 145, characters 16-26
          Called from Piaf__Server.Command.accept_loop.(fun) in file "lib/server.ml", line 236, characters 8-707
          Called from Piaf__Server.Command.accept_loop.(fun) in file "lib/server.ml" (inlined), line 233, characters 19-825
          Called from Eio__core__Fiber.fork.(fun) in file "lib_eio/core/fiber.ml", line 20, characters 10-14
          Re-raised at Eio__core__Switch.maybe_raise_exs in file "lib_eio/core/switch.ml" (inlined), line 118, characters 21-56
          Called from Eio__core__Switch.run_internal in file "lib_eio/core/switch.ml" (inlined), line 139, characters 4-21
          Called from Eio__core__Switch.run.(fun) in file "lib_eio/core/switch.ml" (inlined), line 153, characters 35-62
          Called from Eio__core__Cancel.sub.(fun) in file "lib_eio/core/cancel.ml", line 171, characters 2-6
          Called from Eio__core__Cancel.sub.(fun) in file "lib_eio/core/cancel.ml" (inlined), line 170, characters 43-58
          Called from Eio__core__Cancel.with_cc in file "lib_eio/core/cancel.ml" (inlined), line 116, characters 8-12
          Called from Eio__core__Cancel.sub in file "lib_eio/core/cancel.ml" (inlined), line 170, characters 2-58
          Called from Eio__core__Switch.run in file "lib_eio/core/switch.ml" (inlined), line 153, characters 13-63
          Called from Piaf__Server.Command.listen.(fun).run_accept_loop in file "lib/server.ml" (inlined), line 280, characters 10-431
          Called from Eio__core__Cancel.sub in file "lib_eio/core/cancel.ml" (inlined), line 170, characters 2-58
          Called from Eio__core__Switch.run in file "lib_eio/core/switch.ml" (inlined), line 153, characters 13-63
          Called from Piaf__Server.Command.listen.(fun).run_accept_loop in file "lib/server.ml", line 280, characters 10-431
          Re-raised at Eio__core__Cancel.with_cc in file "lib_eio/core/cancel.ml" (inlined), line 118, characters 32-40
          Called from Eio__core__Cancel.sub in file "lib_eio/core/cancel.ml" (inlined), line 170, characters 2-58
          Called from Eio__core__Switch.run in file "lib_eio/core/switch.ml" (inlined), line 153, characters 13-63
          Called from Piaf__Server.Command.listen.(fun).run_accept_loop in file "lib/server.ml", line 280, characters 10-431
          Called from Eio__core__Fiber.any.(fun).wrap in file "lib_eio/core/fiber.ml", line 97, characters 16-20
          Re-raised at Eio__core__Fiber.any in file "lib_eio/core/fiber.ml", line 138, characters 26-61
          Called from Eio__Domain_manager.run.(fun) in file "lib_eio/domain_manager.ml", line 29, characters 4-231
          Re-raised at Eio__Domain_manager.run.(fun) in file "lib_eio/domain_manager.ml", line 43, characters 11-19
          Called from Eio_posix__Domain_mgr.wrap_backtrace in file "lib_eio_posix/domain_mgr.ml", line 80, characters 8-12
          Re-raised at Eio_posix__Domain_mgr.unwrap_backtrace in file "lib_eio_posix/domain_mgr.ml", line 88, characters 22-57
          Called from Piaf__Server.Command.listen.(fun) in file "lib/server.ml" (inlined), line 277, characters 25-736
          Called from Eio__core__Fiber.fork.(fun) in file "lib_eio/core/fiber.ml", line 20, characters 10-14
          Re-raised at Eio__core__Switch.maybe_raise_exs in file "lib_eio/core/switch.ml", line 118, characters 21-56
          Called from Eio__core__Switch.run_internal in file "lib_eio/core/switch.ml", line 139, characters 4-21
          Called from Eio__core__Cancel.with_cc in file "lib_eio/core/cancel.ml", line 116, characters 8-12
          Re-raised at Eio__core__Cancel.with_cc in file "lib_eio/core/cancel.ml", line 118, characters 32-40
          Called from Eio_posix__Sched.with_op in file "lib_eio_posix/sched.ml", line 312, characters 8-12
          Re-raised at Eio_posix__Sched.with_op in file "lib_eio_posix/sched.ml", line 318, characters 4-12
          Called from Eio_posix__Sched.run.(fun) in file "lib_eio_posix/sched.ml", line 370, characters 25-43
          Re-raised at Eio_posix__Sched.run.fork.(fun) in file "lib_eio_posix/sched.ml", line 333, characters 12-76
          Called from Stdlib__Fun.protect in file "fun.ml", line 33, characters 8-15
          Re-raised at Stdlib__Fun.protect in file "fun.ml", line 38, characters 6-52
          Called from Eio_posix__Sched.run in file "lib_eio_posix/sched.ml", line 366, characters 4-217
          Called from Eio_posix__Sched.with_sched in file "lib_eio_posix/sched.ml", line 252, characters 8-12
          Re-raised at Eio_posix__Sched.with_sched in file "lib_eio_posix/sched.ml", line 257, characters 4-39
          Called from Alcotest_engine__Core.Make.protect_test.(fun) in file "src/alcotest-engine/core.ml", line 181, characters 17-23
          Called from Alcotest_engine__Monad.Identity.catch in file "src/alcotest-engine/monad.ml", line 24, characters 31-35

Getting Unbound Module errors when building the binaries

Hi there,
how do I build the binaries? I cloned the repo and tried some dune build commands, but ended up having to opam install a ton of dependencies. I'm on Mac so I had to brew install pkg-config too. After that I'm still getting these errors:

$ dune build bin/carl.exe
File "lib/openssl.ml", line 72, characters 28-55:
72 |   let verify_error_string = Ssl.get_verify_error_string verify_result in
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Unbound value Ssl.get_verify_error_string
File "lib/piaf.mli", line 539, characters 7-36:
539 |     -> Httpaf_lwt_unix.Server.socket
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Unbound type constructor Httpaf_lwt_unix.Server.socket
File "lib/http1.ml", line 59, characters 21-38:
59 |     (Httpaf_client : Httpaf_lwt.Client)
                          ^^^^^^^^^^^^^^^^^
Error: Unbound module Httpaf_lwt

explore making the switch argument explicit in server request handlers

Client multipart support

It would be cool if we could send multipart uploads from piaf

client first class support to JSON

As discussed with @anmonteiro, it would be cool to have some way to integrate JSON into the Piaf client.

Antonio proposed the notion of a middleware on the frontend. Another option would be simply to provide some functor interface to instantiate.

Opam release

Hello,

I am reviving an old project of mine that depends on Piaf, from back when it was not published on Opam yet, and now that i check my dependencies I see that Piaf has been actively developed in the mean time, but that there has bee no release in a year. So, I was wondering is there is a plan for a release ?

Thanks for the great software!

Opam install issue

Not sure if I should post here or in opam repo but here goes.

I've been trying to install piaf through opam but just keep running into errors, every library so far works fine. I have tried with ocaml version 5.0.0~alpha0 and 1, 5.0.0+trunk and also 4.14.0, none of them seem to be able to compile the package correctly?

I also tried to install through the websocket branch but same issue, I also tried to simply git clone the repo and build it and got build errors too.

Im very new to ocaml, opam and dune so im probably doing something wrong. After cloning and running "opam install ." (opam verison 2.1) I got this:

error when using sending connection:close header in tests

I'm seeing odd behavior where the server works fine when run directly, but in tests, using the Oneshot client, I get an error like below. If I don't send the connection close parameter, the test passes. I only sent it as I saw it in some of the piaf client tests and assumed that it was convenient for more quickly cleaning up connections during tests. I can't tell if this is an error that is possible to happen normally in the server, of if it's an issue with the Oneshot client and will only happen in tests.

[ERROR] Error in connection handler: End_of_file
                         Raised by primitive operation at Unix in file "unix.ml" (inlined), line 602, characters 0-99
Called from Eio_posix__Low_level.shutdown.(fun) in file "lib_eio_posix/low_level.ml", line 80, characters 40-60
Called from Eio_unix__Rcfd.use in file "lib_eio/unix/rcfd.ml", line 161, characters 10-14
Re-raised at Eio_unix__Rcfd.use in file "lib_eio/unix/rcfd.ml", line 166, characters 6-41
Called from Eio_posix__Flow.Impl.shutdown in file "lib_eio_posix/flow.ml", line 55, characters 6-156
Re-raised at Eio__core__Switch.maybe_raise_exs in file "lib_eio/core/switch.ml" (inlined), line 118, characters 21-56
Called from Eio__core__Switch.run_internal in file "lib_eio/core/switch.ml" (inlined), line 139, characters 4-21
Called from Eio__core__Switch.run.(fun) in file "lib_eio/core/switch.ml" (inlined), line 153, characters 35-62
Called from Eio__core__Cancel.sub.(fun) in file "lib_eio/core/cancel.ml", line 171, characters 2-6
Called from Eio__core__Cancel.sub.(fun) in file "lib_eio/core/cancel.ml" (inlined), line 170, characters 43-58
Called from Eio__core__Cancel.with_cc in file "lib_eio/core/cancel.ml" (inlined), line 116, characters 8-12
Called from Eio__core__Cancel.sub in file "lib_eio/core/cancel.ml" (inlined), line 170, characters 2-58
Called from Eio__core__Switch.run in file "lib_eio/core/switch.ml" (inlined), line 153, characters 13-63
Called from Piaf__Server.Command.listen.accept_loop.accept.(fun) in file "lib/server.ml" (inlined), line 253, characters 15-417
Called from Eio__core__Cancel.sub in file "lib_eio/core/cancel.ml" (inlined), line 170, characters 2-58
Called from Eio__core__Switch.run in file "lib_eio/core/switch.ml" (inlined), line 153, characters 13-63
Called from Piaf__Server.Command.listen.accept_loop.accept.(fun) in file "lib/server.ml", line 253, characters 15-417
Re-raised at Eio__core__Cancel.with_cc in file "lib_eio/core/cancel.ml" (inlined), line 118, characters 32-40
Called from Eio__core__Cancel.sub in file "lib_eio/core/cancel.ml" (inlined), line 170, characters 2-58
Called from Eio__core__Switch.run in file "lib_eio/core/switch.ml" (inlined), line 153, characters 13-63
Called from Piaf__Server.Command.listen.accept_loop.accept.(fun) in file "lib/server.ml", line 253, characters 15-417
Called from Eio__Net.accept_fork.(fun) in file "lib_eio/net.ml", line 269, characters 40-83```

Make server error_handler signature take a Piaf.Error.t rather than Reqd.error

Don't re-issue POST requests on redirects

If the 301 status code is received in response to a request other
than GET or HEAD, the user agent MUST NOT automatically redirect the
request unless it can be confirmed by the user, since this might
change the conditions under which the request was issued.
Note: When automatically redirecting a POST request after
receiving a 301 status code, some existing HTTP/1.0 user agents
will erroneously change it into a GET request.

Type Errors when running examples

Hello, I'm getting these errors when running lwt_get and lwt_get_pipelined.

$ dune exec examples/lwt/lwt_get.exe                                                            
File "examples/lwt/lwt_get.ml", line 17, characters 13-14:
17 |     failwith e
                  ^
Error: This expression has type Piaf.Error.t
       but an expression was expected of type string

$ dune exec examples/lwt/lwt_get_pipelined.exe
File "examples/lwt/lwt_get_pipelined.ml", line 39, characters 6-43:
39 |       (Body.to_string_stream response.body)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: This expression has type
         string Lwt_stream.t * (unit, Error.t) Lwt_result.t
       but an expression was expected of type string Lwt_stream.t

In contrast, dune exec examples/lwt/echo_server.exe works without any errors.

websocket: is_fin bit needed to know when a message finished

The fin bit is needed to know when a websocket message finished, it is currently not exposed and is explicitly ignored at https://github.com/anmonteiro/piaf/blob/master/lib/ws.ml#L120

Compatibility with upstream Httpaf (Opium)

Hi there! I am currently working on a web server that serves an API which is itself a combination of multiple other APIs wrapped around a standardized interface. As such, my project needs both an http server library (to handle incoming requests) and an http client library (to forward those requests to the actual target APIs). I am using Opium (which depends on upstream Httpaf) as my main server library and I am trying to use Piaf (which depends on your fork of Httpaf) as my http client library. However, I am having the hardest time getting the project to compile because of conflicts between the two versions of Httpaf.

Any suggestions on how I can get this to work?

Here is the build error in question:

File "opium/src/app.ml", line 29, characters 31-32:
29 |     Rock.Server_connection.run f app
                                    ^
Error: This expression has type
         request_handler:(Httpaf.Reqd.t Gluten.reqd -> unit) ->
         error_handler:Httpaf.Server_connection.error_handler -> unit Lwt.t
       but an expression was expected of type
         request_handler:Httpaf.Server_connection.request_handler ->
         error_handler:Httpaf.Server_connection.error_handler -> 'a Lwt.t
       Type Httpaf.Reqd.t Gluten.reqd -> unit is not compatible with type
         Httpaf.Server_connection.request_handler = Httpaf.Reqd.t -> unit 
       Type Httpaf.Reqd.t Gluten.reqd = Httpaf.Reqd.t Gluten.Reqd.t
       is not compatible with type Httpaf.Reqd.t 

Here is part of my package.json (I am using esy):

"dependencies": {
  ...
  "@opam/httpaf-lwt-unix": "*",
  "@opam/httpaf-lwt": "*",
  "@opam/httpaf": "*",
  "@opam/piaf": "*",
  "@opam/opium": "~0.20.0"
},
"resolutions": {
  "@opam/httpaf-lwt-unix": "anmonteiro/httpaf:httpaf-lwt-unix.opam#0555dde898f0886fb9909005e00b7527cc1a3895",
  "@opam/httpaf-lwt": "anmonteiro/httpaf:httpaf-lwt.opam#0555dde898f0886fb9909005e00b7527cc1a3895",
  "@opam/httpaf": "anmonteiro/httpaf:httpaf.opam#0555dde898f0886fb9909005e00b7527cc1a3895"
}

Confusing Client API.

This is a fantastic library. The Oneshot API is great but the Client API is kinda strange. It's not obvious that the base Uri passed to the Client is transformed (that the params and query are cut off). If you allow arbitrary base Uri then we have another set of problems. For instance how to pass only the query parameters as target. Manual string concatenation seems a bit awful in this scenario.

Client.Oneshot.get never returns if I read a file before (upstram version with eio)

Hi,

This is a cross post from here.

It might not be an piaf issue, but I am hoping someone might be able to help me her.

I am using the git version of piaf, to be able to use it with eio.

Now this application:

open Piaf

let get ~sw env =
  let ( / ) = Eio.Path.( / ) in
  let path = Eio.Stdenv.cwd env / "key" in
  let key = Eio.Path.load path in
  print_endline key;
  let headers = [ ("Authorization", Format.sprintf "Bearer %s" key) ] in
  Client.Oneshot.get ~sw env (Uri.of_string "https://example.com") ~headers

let () =
  Eio_main.run @@ fun env ->
  Eio.Switch.run (fun sw ->
      let resp = get env ~sw in
      print_endline "done";
      let body =
        resp |> Result.get_ok
        |> (fun r -> Body.to_string r.body)
        |> Result.get_ok
      in
      print_endline body)

Never finishes. It does not event print "done", it only prints the contens from ./key (as it should).

If I remove the reading of the file:

open Piaf

let get ~sw env =
  let headers = [ ("Authorization", "Bearer nothing") ] in
  Client.Oneshot.get ~sw env (Uri.of_string "https://example.com") ~headers

let () =
  Eio_main.run @@ fun env ->
  Eio.Switch.run (fun sw ->
      let resp = get env ~sw in
      print_endline "done";
      let body =
        resp |> Result.get_ok
        |> (fun r -> Body.to_string r.body)
        |> Result.get_ok
      in
      print_endline body)

It works! I get the html from example.com printed.

any Idea why?

Thanks!