sneako / finch

Elixir HTTP client, focused on performance

An HTTP client with a focus on performance, built on top of Mint and NimblePool.

We attempt to achieve this goal by providing efficient connection pooling strategies and avoiding copying of memory wherever possible.

Most developers will most likely prefer to use the fabulous HTTP client Req which takes advantage of Finch's pooling and provides an extremely friendly and pleasant to use API.

Usage

In order to use Finch, you must start it and provide a :name. Often in your supervision tree:

children = [
  {Finch, name: MyFinch}
]

Or, in rare cases, dynamically:

Finch.start_link(name: MyFinch)

Once you have started your instance of Finch, you are ready to start making requests:

Finch.build(:get, "https://hex.pm") |> Finch.request(MyFinch)

When using HTTP/1, Finch will parse the passed in URL into a {scheme, host, port} tuple, and maintain one or more connection pools for each {scheme, host, port} you interact with.

You can also configure a pool size and count to be used for specific URLs that are known before starting Finch. The passed URLs will be parsed into {scheme, host, port}, and the corresponding pools will be started. See Finch.start_link/1 for configuration options.

children = [
  {Finch,
   name: MyConfiguredFinch,
   pools: %{
     :default => [size: 10],
     "https://hex.pm" => [size: 32, count: 8]
   }}
]

Pools will be started for each configured {scheme, host, port} when Finch is started. For any unconfigured {scheme, host, port}, the pool will be started the first time it is requested. Note pools are not automatically terminated by default, if you need to terminate them after some idle time, use the pool_max_idle_time option (available only for HTTP1 pools).

Telemetry

Finch uses Telemetry to provide instrumentation. See the Finch.Telemetry module for details on specific events.

Logging TLS Secrets

Finch supports logging TLS secrets to a file. These can be later used in a tool such as Wireshark to decrypt HTTPS sessions. To use this feature you must specify the file to which the secrets should be written. If you are using TLSv1.3 you must also add keep_secrets: true to your pool :transport_opts. For example:

{Finch,
 name: MyFinch,
 pools: %{
   default: [conn_opts: [transport_opts: [keep_secrets: true]]]
 }}

There are two different ways to specify this file:

1. The :ssl_key_log_file connection option in your pool configuration. For example:

{Finch,
 name: MyFinch,
 pools: %{
   default: [
     conn_opts: [
       ssl_key_log_file: "/writable/path/to/the/sslkey.log"
     ]
   ]
 }}

2. Alternatively, you could also set the SSLKEYLOGFILE environment variable.

Installation

The package can be installed by adding finch to your list of dependencies in mix.exs:

def deps do
  [
    {:finch, "~> 0.18"}
  ]
end

The docs can be found at https://hexdocs.pm/finch.

finch's Issues

Invalid response

I have this request: curl http://www.kakadu.de/podcast-kakadu.2730.de.podcast.xml.
The server returns a well formed HTTP response, however, Finch does not return an UTF-8 body, instead, it returns a binary.

{:ok,
 [
   {"vary", "Accept-Encoding"},
   {"cache-control", "public, max-age=900, pre-check=900, no-transform"},
   {"x-cache", "HIT"},
   {"content-type", "application/xml; charset=utf-8"},
   {"content-encoding", "gzip"},
   {"p3p", "CP=\"NOI NID ADMa OUR IND UNI COM NAV\""},
   {"date", "Tue, 06 Apr 2021 05:10:07 GMT"},
   {"x-server", "webr03"},
   {"expires", "Tue, 06 Apr 2021 05:25:08 GMT"},
   {"pragma", ""},
   {"accept-ranges", "bytes"},
   {"connection", "Keep-Alive"},
   {"last-modified", "Tue, 06 Apr 2021 05:10:09 GMT"},
   {"x-cache-info", "cached"},
   {"age", "930"},
   {"content-length", "32982"}
 ],
 <<31, 139, 8, 0, 0, 0, 0, 0, 0, 3, 237, 189, 91, 115, 219, 72, 182, 38, 250,
   222, 191, 2, 229, 29, 177, 199, 221, 35, 136, 184, 95, 84, 213, 229, 35, 149,
   175, 37, 203, 246, 182, 236, 242, 236, 154, 152, 96, ...>>}

I would have expected something along those lines (taken from curl):

< HTTP/1.1 200 OK
< Cache-Control: public, max-age=900, pre-check=900, no-transform
< X-Cache: HIT
< Content-Type: application/xml; charset=utf-8
< P3P: CP="NOI NID ADMa OUR IND UNI COM NAV"
< Date: Tue, 06 Apr 2021 05:33:44 GMT
< X-Server: webr02
< Expires: Tue, 06 Apr 2021 05:48:44 GMT
< Pragma:
< Transfer-Encoding: chunked
< Connection: Keep-Alive
< Last-modified: Tue, 06 Apr 2021 05:33:45 GMT
< X-Cache-Info: cached
< Age: 516
<
{ [945 bytes data]
<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet type="text/xsl" media="screen" href="http://www.kakadu.de/themes/dradio/podcast/podcast.xsl"?><rss xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
...

Feature Request - Decompress compressed responses

I wasn't able to find any reference to it in the code or docs so apologies if this is mentioned somewhere or already implemented.

I've run into an issue where an API is returning compressed responses (gzip in this instance) and it caused my client to fail. I was able to repair this with a stop gap measure on my end but it would be ideal if the Finch library was able to handle decompression seamlessly.

Thanks in advance for your consideration!

Add more fine grained timeout options: connect, recv

Right now, we pass the same value for receive_timeout, everywhere that will accept it, with no regard for any time that has already elapsed. We need update the receive_timeouts we pass by subtracting the elapsed time of any intermediate steps in order to properly respect the timeout.

Return response as a struct?

I was wondering if we should add a response struct. The main benefit behind this struct is that it would allow users of the library to understand what keys are available on the response they get back.

Telemetry - custom fields in metadata

Hey, I wonder if there is a way currently to add custom metadata fields (e.g client_id) to telemetry events emitted by Finch? My use case requires exporting metrics to Prometheus and it would be quite easy to label them with custom tags if those would be available in an event's metadata.

accept %URI{} as well as binary for url()

https://hexdocs.pm/finch/Finch.html#t:url/0 defines a url as a binary(). I am accustomed to passing around %URI{} structs instead.

Is there a reason you wouldn't also accept (or perhaps only expect) a %URI{} struct?

BTW - love the library, just thought I'd ask.

Finch times out when attempting to connect to a bad address

Currently when Finch attempts to connect to a server that doesn't exist/is down, it will time out instead of raising/throwing/returning an error stating that it can't connect.

I came across this when attempting to swap out HTTPoison with Finch in the Wallaby project.

I think other HTTP clients will usually use an error along the lines of :econnrefused or :nxdomain for the corresponding connection errors.

Not all errors returned by request/3 are Mint.Types.error()

This may be related to #54

We've noticed that some errors, like :disconnected, aren't "real" error types and can cause a FunctionClauseError when Exception.message is called on them.

See this branch for a demonstration:

https://github.com/lukebakken/finch/blob/some-errors-are-not-exceptions/test/finch/http2/pool_test.exs#L113-L155

See this issue for where we originally thought we should report it:

elixir-tesla/tesla#469

It seems like this type spec should be updated, or the "non-Exception" errors be wrapped:

https://github.com/keathley/finch/blob/main/lib/finch.ex#L226-L227

Thanks a lot!

Including helpers for multipart requests

Hi there - I'm need to build and make streaming multipart requests, and noticed that Finch doesn't have support for this out of the box. Is this something I could contribute, or is this considered beyond the scope of the library and a separate dependency would be more appropriate?

Thanks!

Expose connection options

We should allow the user to pass in options that will be passed to Mint.connect/4.

Everything is already wired up internally, we just need to accept the options in the pool config.

Add Credo?

I don't have too much experience with credo, but I definitely see it's value when collaborating with larger groups.

The credo file included in Norm looks like it could be a good start: https://github.com/keathley/norm/blob/master/.credo.exs

Check window sizes in HTTP/2

If you do not check the window sizes you can get errors if the client is sending faster than the server can receive:

Mint.HTTPError: the given data exceeds the connection window size, which is 65535.
The server will refill the window size of the connection when ready.
This will be handled transparently by stream/2.

Use the function Mint.HTTP2.get_window_size/2 https://hexdocs.pm/mint/Mint.HTTP2.html#get_window_size/2 to get the window sizes for the connection and request.

Support custom Mint transports

Hi!

Mint natively (and silently) supports custom transports by accepting a module implementing Mint.Core.Transport behaviour:

Mint.HTTP.connect(My.Custom.Module, "example.com", 80)

Although it's an undocumented (yet) feature, it works seamlessly. It would be great if Finch pools also accepted a tuple in the form of {module, host, port}. I saw the ongoing conversation about the UNIX socket (#139); this might be related.

Handling error with Finch

In HTTPPoison

headers=[{"Content-Type", "application/json"}]
case HTTPoison.post(url,P%{},headers, hackney: [pool: :first_pool]) do
      {:ok, %{status_code: status, body: data}} -> 
          ...........
      {:error, %HTTPoison.Error{reason: reason}} ->
          ............ 

how can we deal with timeout or how to handle error in Finch?

I tried the following to handle Error

case Finch.request(MyFinch, :get, "https://api.github.com/zen") do
      {:ok, %{status: status, body: data}} -> 
        IO.inspect status
        IO.inspect data
      {:error, %Finch.Error{reason: reason}} ->
        IO.inspect error
    end

but I get the following error :

Finch.Error.__struct__/0 is undefined, cannot expand struct Finch.Error. Make sure the struct name is correct. If the struct name exists and is correct but it still cannot be found, you likely have cyclic module usage in your code

Return helpful error on pool NimblePool.checkout timeouts

Right now the error returned from a checkout timeout in finch is very minimal and doesn't help a user solve the issue.

Capture and return a more helpful error to help the user know what options to tweak to resolve the issue.

Basic Auth is not working && Insecure option

I am trying to make a GET request to a y-cam, with basic authentication. Which is working perfectly alright in postman but not working in Finch.

The total stack trace is below with Basic auth credentials,

iex(2)> Finch.start_link(name: MyFinch)
{:ok, #PID<0.3648.0>}
iex(7)> Finch.build(:get, "http://93.107.118.128:8083/snapshot.jpg", [{"Authorization", "Basic ---="}]) |> Finch.request(MyFinch)                                                                   %Mint.HTTP1{       
  buffer: "",
  host: "93.107.118.128",
  mode: :passive,
  port: 8083,
  private: %{},
  request: nil,
  requests: {[], []},
  scheme_as_string: "http",
  socket: #Port<0.10>,
  state: :open,
  transport: Mint.Core.Transport.TCP
}
%Mint.HTTP1{
  buffer: "",
  host: "93.107.118.128",
  mode: :passive,
  port: 8083,
  private: %{},
  request: %{
    body: nil,
    connection: [],
    content_length: nil,
    data_buffer: [],
    headers_buffer: [],
    method: "GET",
    ref: #Reference<0.702117736.1220280322.1452>,
    state: :status,
    status: nil,
    transfer_encoding: [],
    version: nil
  },
  requests: {[], []},
  scheme_as_string: "http",
  socket: #Port<0.10>,
  state: :open,
  transport: Mint.Core.Transport.TCP
}
#Reference<0.702117736.1220280322.1452>
{:error, %Mint.TransportError{reason: :closed}}
iex(8)> 

where this works fine in postman

the response supposes to be an Image and postman do give it back. what could be wrong? I am at a dead end with this.

Also in past, while using HTTPoison, there was an option which we can pass in hackney, for :insecure while making a request.

*** insecure: to perform "insecure" SSL connections and transfers without checking the certificate**

is there any option available in Finch or mint?

Support SSLKEYLOGFILE environment variable.

In order to decrypt TLS sessions in tools like Wireshark, the per-session secrets can be logged to a file. This document specifies the file format - https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Key_Log_Format

Recent versions of Erlang/OTP (ssl version 10.2 or greater) have support for getting the keylog data for a TLS connection. If a user wishes to log these secrets to a file, they will export the SSLKEYLOGFILE environment variable prior to starting their application. Then, any time a TLS connection is established in this library and that variable is present, the secrets for that connection will be appended to the file.

See "Exporting the Secrets" here: https://erlang.org/doc/apps/ssl/using_ssl.html

I have the following PR to be accepted soon in mint that makes get_socket/1 a public API - elixir-mint/mint#310

Prior to establishing a connection, if the scheme is https and the SSLKEYLOGFILE env var is exported, the user must set the keep_secrets transport option. This is required for TLS 1.3 connections to preserve the one-time TLS secrets to be logged later (Erlang/OTP docs PR that came from figuring that out - erlang/otp#4743)

After a successful HTTP1 or HTTP2 connection (here, for instance) the new code will check that the scheme is https and that the env variable is exported. If both are true, the :ssl.connection_information function will be used to append the :keylog values to the file indicated in the SSLKEYLOGFILE environment variable.

One nice feature of :keylog is that the data is already correctly formatted to be written to the file.

I'll start work on this soon, thanks!

OS opening more connections than specified in pools config?

Hello,
I'm testing this library using the following configuration:

# MyApp.Application

{Finch, name: MyApp.Finch, pools: %{
    :default => [size: 3, count: 1],
    "https://host.sample-videos.com" => [size: 3, count: 1],
}}

And then doing a bunch of requests with:

Enum.map(1..20, fn n ->
  Task.start(fn ->
    Finch.build(:get, "https://www.sample-videos.com/video123/mp4/720/big_buck_bunny_720p_30mb.mp4")
    |> Finch.request(MyApp.Finch)
  end)
end) 

However the connections in iftop show 20 connections open:

 * :39881                    <=> host.sample-videos.com-:https  416b    479Kb   219Kb
 * :58309                    <=> host.sample-videos.com-:https  344Kb   479Kb   197Kb
 * :39449                    <=> host.sample-videos.com-:https  624b    477Kb   219Kb
 * :56327                    <=> host.sample-videos.com-:https  302Kb   470Kb   156Kb
 * :33007                    <=> host.sample-videos.com-:https  515Kb   440Kb   212Kb
 * :39955                    <=> host.sample-videos.com-:https  316Kb   439Kb   198Kb
 * :45601                    <=> host.sample-videos.com-:https  353Kb   437Kb   146Kb
 * :59039                    <=> host.sample-videos.com-:https  216Kb   399Kb   129Kb
 * :42899                    <=> host.sample-videos.com-:https  341Kb   372Kb   114Kb
 * :36477                    <=> host.sample-videos.com-:https  522Kb   368Kb   118Kb
 * :36269                    <=> host.sample-videos.com-:https  318Kb   341Kb   178Kb
 * :39667                    <=> host.sample-videos.com-:https  114Kb   341Kb   155Kb
 * :40727                    <=> host.sample-videos.com-:https  275Kb   314Kb   128Kb
 * :37057                    <=> host.sample-videos.com-:https  302Kb   305Kb   109Kb
 * :36993                    <=> host.sample-videos.com-:https  158Kb   215Kb  75,3Kb
 * :51285                    <=> host.sample-videos.com-:https  170Kb   213Kb  82,6Kb
 * :40895                    <=> host.sample-videos.com-:https  193Kb   192Kb  65,2Kb
 * :38113                    <=> host.sample-videos.com-:https  159Kb   189Kb  91,1Kb
 * :51565                    <=> host.sample-videos.com-:https    0b    162Kb   219Kb
 * :44515                    <=> host.sample-videos.com-:https    0b   1,26Kb   219Kb
 * :49556                    <=> lb-140-82-112-3-iad.git:https    0b    187b     47b
 ...
 ... etc

What am I missing? Shouldn't it open 3 connections at a time only?

HTTP/2

So far, we have focused on HTTP/1, but since HTTP/2 allows multiplexing, a different pooling strategy should perform even better. We probably want to just start a group of connections, register them in the Registry, and round-robin, LRU, or randomly choose one instead.

We need to allow users to explicitly choose which protocol (and therefore pooling strategy) to use, unless we can efficiently determine if the server supports H2 and then automatically choose the correct pooling strategy.

Allow custom pool assignment other than {scheme, host, port}

If pool is only determined by {scheme, host, port}, then SSL option will not be considered, which results in mixing up connection with different SSL option. (e.g. using SSL client side certificate for authentication)

Related

• https://elixirforum.com/t/strange-httpc-hackeny-behavior-on-ssl-verify/22695/3
• benoitc/hackney#570

Package `finch` in hex.pm is already registered

Hello, here is the link to finch package of hex.pm, we may need to face this problem one day in the future, WDYT?

Telemetry event [:finch, :response, :stop] doesn't include the response status

It would be nice to have the response HTTP status code in the [:finch, :response, :stop] event.

Mint.TransportError: :closed

I am using Finch in a service that relies on a remote service for data, but I am doing a fair amount of caching and other work at my service to avoid having to make this remote call too often for performance reasons.

As a result, we are periodically seeing this socket close error. A related issue on Mint points to different defaults being necessary for :conn_opts as part of the initial pool configuration. I don't know if it's actually the fix, but any possibility that this socket being closed can be detected behind the scenes in Finch? The ideal would be that if a connection gets closed due to inactivity that it wouldn't be surfaced when making a later request.

Version Information:
Finch: 0.2.0
Mint: 1.1.0

Add HTTP1 integration tests

I'd like to add some of these. Is there a reason there are none other than no one implemented yet?

Finch defaults to :http1 for dynamic requests

The following code uses http1 to make a request even though the hex.pm server supports http2.

Finch.start_link(name: MyFinch)
Finch.build(:get, "https://hex.pm") |> Finch.request(MyFinch)

This happens because the default type here evaluates to :http1 https://github.com/keathley/finch/blob/main/lib/finch/pool_manager.ex#L61 Ideally, the type of pool should be decided by whether the server supports http2.

Exponential backoff test is flaky

On master (rarely):

  1) test request/5 will exponentially backoff until connection succeeds (FinchTest)
     test/finch_test.exs:205
     Assertion with >= failed
     code:  assert interval >= initial_backoff
     left:  35
     right: 37
     stacktrace:
       test/finch_test.exs:265: anonymous fn/3 in FinchTest."test request/5 will exponentially backoff until connection succeeds"/1
       (elixir 1.10.1) lib/enum.ex:783: Enum."-each/2-lists^foreach/1-0-"/2
       (elixir 1.10.1) lib/enum.ex:783: Enum.each/2
       test/finch_test.exs:262: (test)

HTTP method api?

I was wondering if we should support the standard "HTTP method" API in Finch. I'm content to use the single request function and specify the method as an atom. But I figured I would see what others thought.

Allow pool configuration urls to be provided as a binary?

Right now we force the user to mentally parse the url in to the {scheme, host, port} (not that it's very difficult), but maybe some users would prefer to configure their pools like:

pools: %{
  "https://hex.pm" => %{size: 10, count: 2}
}

instead of the current:

pools: %{
  {:https, "hex.pm", 443} => %{size: 10, count: 2}
}

Error struct

We should have an exception that we can use to capture different errors. I'm not sure if we should wrap over the Mint errors or not. But it seems like we might want to for consistency.

HTTP/2 does not emit :queue Telemetry event

Would this event start before the :gen_statem.call/2 and end when we get a response (or error/exception)?

Also, this raises the question of whether we should add support for the :pool_timeout option, which would then be passed to :gen_statem.call/3. We would probably want to use a :dirty timeout to avoid copying and sending the request via a spawned proxy process.

How to use ExVCR with Finch?

My project uses Finch to make parallel HTTP requests.

I tried to add VCR to my tests, but the HTTP requests are still happening.

Here is my test:

defmodule MyClientTest do
  use ExUnit.Case, async: true
  use ExVCR.Mock #, adapter: ExVCR.Adapter.Httpc

  setup do
    ExVCR.Config.cassette_library_dir("fixture/vcr_cassettes")
    :ok
  end

  test "list_apps" do
    use_cassette "list_apps" do
      list_apps = MyClient.list_apps
      assert MyClient.list_apps == ["app1", "app2"]
    end
  end

end

I see a list_apps.json file in the fixture/vcr_cassettes, but its content is only []

Here is my MyClient module:

defmodule MyClient do
  alias Finch.Response

  def child_spec do
    {Finch,
     name: __MODULE__,
     pools: %{
       "https://myapp.com" => [size: 100]
     }}
  end

  def applications_response do
    :get
    |> Finch.build("https://myapp.com/v2/apps.json")
    |> Finch.request(__MODULE__)
  end

  def handle_applications_response({:ok, %Response{body: body}}) do
    body
    |> Jason.decode!()
    end
  end

  def list_apps do
    handle_applications_response(applications_response())
  end

end

How to apply VCR to this example?

Emit telemetry events

Finch should support telemetry events around requests. It probably makes sense to follow the new span conventions.

Here are the metrics that we tend to use when diagnosing production issues:

• queue duration
• new connection duration
• request duration
• response duration
• rate of new connections vs rate of reused connection

I think all of the durations we can get "for free" using spans.

Converting master to main

@sneako Just a heads up that I'm going to migrate the "master" branch to "main". Shouldn't cause any issues but wanted you to be aware 👍.

Support unix sockets

Hello,

Mint natively supports making requests on unix sockets by setting the address to {:local, "/path/to/socket.sock"} and the port number to 0, for example:

{:ok, conn} = Mint.HTTP.connect(:http, {:local, "/var/run/docker.sock"}, 0, hostname: "localhost")

For more context, tests for this functionality in Mint can be found here.

However, to my knowledge Finch does not currently allow this.
In case this feature may be desirable, a possible approach could be to do as Hackney does, ie. supporting url schemas in the form of eg. "http+unix".

Streaming response continuously?

Imagine connecting to a forex/crypto trader that streams the response continuously with real-time price changes.

I can achieve that with Mint easily through their API (see sample code below); we can easily stream and read from the response forever. But it seems that it is not possible with Flinch?

I stumbled upon the same demand for Tesla:
elixir-tesla/tesla#271
but open since 2018

Problem with mint is that I don't have a connection pooling (there's mint_pool in development but stopped for now) and using something higher-level such as flinch would be easier to work with.

defmodule Forex do
  defp receive_next_and_stream(conn) do
    receive do
      message ->
        result = Mint.HTTP.stream(conn, message)
        {:ok, conn, responses} = result
        receive_next_and_stream(conn)
    end
  end

  def stream_prices() do
    {:ok, conn} =
      Mint.HTTP.connect(:https, "stream-fxpractice.oanda.com", 443, timeout: :infinity)

    uri = "/v3/accounts/" <> account_id <> "/pricing/stream?instruments=EUR_GBP"

    {:ok, conn, request_ref} = HTTP.request(conn, "GET", uri, [{"Authorization", token}])

    receive_next_and_stream(conn)
  end
end

Memory pressure caused by connections removed from a pool

When a checkout fails because it exceeds max_idle_time Finch removes the connection from the pool. However it doesn't look to be closing the connection.

https://github.com/keathley/finch/blob/main/lib/finch/http1/pool.ex#L88-L98

The socket would be kept open until it's closed by the server. This can cause a big memory pressure under the high load.

I am happy to open a PR that fixes the issue but I thought I would check if that was intentional. Are there any reasons Finch doesn't close these connections?

When HTTP2 connection turns to read_only, it's not recovered

When my HTTP2 connection turns to read_only, it keeps return read_only error until I restart my app.

{:error, %{reason: :read_only}}

Setting:

defmodule MyApp.Application do
  use Application

  def start(_type, _args) do
    children = [
      {Finch,
       name: MyApp.FintchPool,
       pools: %{
         :default => [protocol: :http2]
       }}
    ]

    Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
  end
end

First Use Time & Idle Time Tracking

This will help in configuring pool size & count.

We can record System.monotonic_time on init and handle_checkin and subtract it from System.monotonic_time on handle_checkout.

Since init doesn't connect, maybe "first_use_time" makes sense as the name for that measurement.

http/1.0 is not being handled correctly.

I am doing this in mint.

{:ok, conn} = Mint.HTTP.connect(:http, "93.107.118.128", 8083)
{:ok, conn, request_ref} = Mint.HTTP.request(conn, "GET", "/snapshot.jpg", [{"Authorization", "Basic junaid="}], "")

as a result of first, we get this

{:ok,
 %Mint.HTTP1{
   buffer: "",
   host: "93.107.118.128",
   mode: :active,
   port: 8083,
   private: %{},
   request: nil,
   requests: {[], []},
   scheme_as_string: "http",
   socket: #Port<0.7>,
   state: :open,
   transport: Mint.Core.Transport.TCP
 }}

and on 2nd we get this.

{:ok,
 %Mint.HTTP1{
   buffer: "",
   host: "93.107.118.128",
   mode: :active,
   port: 8083,
   private: %{},
   request: %{
     body: nil,
     connection: [],
     content_length: nil,
     data_buffer: [],
     headers_buffer: [],
     method: "GET",
     ref: #Reference<0.2499171616.2477260803.176902>,
     state: :status,
     status: nil,
     transfer_encoding: [],
     version: nil
   },
   requests: {[], []},
   scheme_as_string: "http",
   socket: #Port<0.7>,
   state: :open,
   transport: Mint.Core.Transport.TCP
 }, #Reference<0.2499171616.2477260803.176902>}

when I do this.

{:ok, conn, responses} = receive do msg -> Mint.HTTP.stream(conn, msg) end

I get this

{:ok,
 %Mint.HTTP1{
   buffer: "",
   host: "93.107.118.128",
   mode: :active,
   port: 8083,
   private: %{},
   request: %{
     body: :until_closed,
     connection: [],
     content_length: nil,
     data_buffer: [[] | ""],
     headers_buffer: [],
     method: "GET",
     ref: #Reference<0.2499171616.2477260803.176902>,
     state: :body,
     status: 200,
     transfer_encoding: [],
     version: {1, 0}
   },
   requests: {[], []},
   scheme_as_string: "http",
   socket: #Port<0.7>,
   state: :open,
   transport: Mint.Core.Transport.TCP
 },
 [
   {:status, #Reference<0.2499171616.2477260803.176902>, 200},
   {:headers, #Reference<0.2499171616.2477260803.176902>,
    [
      {"server", "HTTPD"},
      {"pragma", "no-cache"},
      {"cache-control", "no-cache"},
      {"content-type", "image/jpeg"}
    ]}
 ]}

on-again doing this

 {:ok, conn, responses} = receive do msg -> Mint.HTTP.stream(conn, msg) end

I get this as response

{:ok,
 %Mint.HTTP1{
   buffer: "",
   host: "93.107.118.128",
   mode: :active,
   port: 8083,
   private: %{},
   request: %{
     body: :until_closed,
     connection: [],
     content_length: nil,
     data_buffer: [],
     headers_buffer: [],
     method: "GET",
     ref: #Reference<0.2499171616.2477260803.176902>,
     state: :body,
     status: 200,
     transfer_encoding: [],
     version: {1, 0}
   },
   requests: {[], []},
   scheme_as_string: "http",
   socket: #Port<0.7>,
   state: :open,
   transport: Mint.Core.Transport.TCP
 },
 [
   {:data, #Reference<0.2499171616.2477260803.176902>,
    <<255, 216, 255, 224, 0, 16, 74, 70, 73, 70, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0,
      255, 219, 0, 67, 0, 13, 9, 10, 11, 10, 8, 13, 11, 10, 11, 14, 14, 13, 15,
      19, 32, 21, 19, ...>>}
 ]}

but when I do the same request in Finch

Finch.build(:get, "http://93.107.118.128:8083/snapshot.jpg", [{"Authorization", "Basic junaid="}]) |> Finch.request(MyFinch)

I just get this.

{:error, %Mint.TransportError{reason: :closed}}

proxy

Hello,
Need help, can't get the proxy working. Tried setting conn_opts, that is supposed to pass to Mint.HTTP.connect/4.
conn_opts: [proxy: [:http, proxyurl, proxyport, []]]

Spec & document Finch.request/6

Non-string values in headers cause a Mint exception

I noticed this when trying to use Finch as an :ex_aws client adapter where the {"content-length", 0} headers (and like) cause a crash.

The exact same issue with Mojito: appcues/mojito#46

The issue was opened in Mint too: elixir-mint/mint#235

Mint decided to leave it to the wrapping implementations has they clearly expect a string. Mojito decided to serialise their header values recursively.

For now I am doing exactly what Mojito is doing but in my own :ex_aws client adapter code.

Need to URI.encode the url - is it expected?

First thanks for the great work on Finch, which we use as an easy streamer to compute checksums on remote resources (https://github.com/etalab/transport-site/blob/master/apps/shared/lib/http_stream_v2.ex), and more use will likely come. So, thanks!

While doing integration work, we noticed that doing a query against an url with | in it will fail:

url = "http://tsvc2.pilote3.cityway.fr/api/Export/v1/GetExportedDataFile?ExportFormat=Gtfs&OperatorCode=AIXENBUS|CG13|CPA|ACCM|FRIOUL|AGGLOBUS|ETANG|COTEBLEUE|MARCOULINE|CIGALES|BUSDESCOLLINES|LIBEBUS|MILSAB|RTMNAV|RTM|SANPROVENCE"

{:error,
 %Mint.HTTPError{
   module: Mint.HTTP1,
   reason: {:invalid_request_target, ...}

(as I discovered by digging into the code, the error is coming from Mint, which does validation etc)

If we modify the url by encoding it from our side, the same query works:

Finch.build(:head, URI.encode(url)) |> Finch.request(MyFinch)
{:ok, ...

What caught us a bit by surprise is that coming from HTTPoison, this is handled by default.

I'm not saying it is a good idea to support this by default of course (I much prefer letting the caller keep the control), but I could not find a mention of that caveat (if confirmed) in the documentation.

Do you think it would be worth adding a documentation note on that?

Thanks!

Implement alternative pool selection strategies

Currently we randomly select a pool, which works already works quite well, but perhaps round robin or least loaded algorithms could perform better under different types of workloads.

See: https://github.com/keathley/finch/blob/1486a224c2856ba5788d1c84d185773ec7f2d5e7/lib/finch/pool_manager.ex#L39

Proxying a HTTP/1 connection

I tried to configure proxying but could not get it to work. While I was digging into the code, I noticed that both pool implementations use the connect/4 function specific to the HTTP protocol version. According to the docs Mint.HTTP1.connect/4 "[…] doesn't support proxying" though. The :proxy configuration seems to be supported by Mint.HTTP.connect/4 only.

Am I missing something? Has anyone successfully set up proxying?

Example code:

pools: %{
  :default => [size: 5, conn_opts: [proxy: {:http, proxy_url, proxy_port, []}]]
}

Compilation error in file lib/finch/http2/pool.ex

I just added finch to my project and I get this compilation error:

==> mint
Compiling 1 file (.erl)
Compiling 1 file (.ex)
Generated mint app
==> finch
Compiling 9 files (.ex)

== Compilation error in file lib/finch/http2/pool.ex ==
** (CompileError) lib/finch/http2/pool.ex:227: Mint.HTTPError.__struct__/0 is undefined, cannot expand struct Mint.HTTPError. Make sure the struct name is correct. If the struct name exists and is correct but it still cannot be found, you likely have cyclic module usage in your code
    (stdlib 3.13) lists.erl:1354: :lists.mapfoldl/3
    (stdlib 3.13) lists.erl:1355: :lists.mapfoldl/3

FWIW I don't use mint or any packages that finch depends on.

Any ideas how to resolve this?

Versions:

Versions:
elixir 1.11.2
erlang 23.0.3
finch 0.5.2

Reconnection strategy

Right now we just retry immediately, over and over, until the pool checkout timeouts.

An exponential backoff probably makes sense here.

See: https://github.com/keathley/finch/blob/2796b85bce60c0f5561da52e04c10c3a3cb12fdc/lib/finch/conn.ex#L20

Pool Size default (for HTTP/1.1) is too low

I've been comparing different adapters for use in Tesla and I believe the default pool size setting for Finch is too low to be comparable to other adapter defaults.

Request per second:

• Finch default pool size of 10: RPS: 198.13284094832753
• Hackney default: RPS: 932.1966376616629
• HTTPC default: RPS: 3225.471319940388
• Finch pool size of 50: RPS: 956.4245735627867

These were all tested the exact same way, the only difference is the Tesla adapter used for the requests.

I'm happy to make an PR to adjust the default pool size to be more in line with other libraries.

Abort streamed request

Say I'm using Finch to proxy some request to an upstream http server I might want to stop the stream if the caller closed the connection:

Finch.stream(request, Pour.HTTP, conn, fn
  {:status, status}, conn -> 
    send_chunked(conn, status)
    
  {:headers, _}, conn -> 
    conn
    
  {:data, _}, conn -> 
    case Plug.Conn.chunk(conn, chunk) do
      {:ok, conn} -> conn
      {:error, :closed} -> # how to abort here
    end
end)

Support arbitrary string methods

Right now, we currently have guard clauses that are perhaps a bit too restrictive and therefore do not allow the user to use arbitrary HTTP methods.

Are there any good reasons to keep this limitation, or should it be removed?