Large resources unusable with 2.0.0 about k8s

[elliottneilclark]

I've tried with K8s.Client.async as well and it also has the same limitation.

[elliottneilclark]

I am seeing a K8s.Client.Mint.HTTPAdapter with a Mint.HTTP2 connection. Interestingly enough if I put some debug logging into K8s.Client.Mint.HTTPAdapter specifically https://github.com/coryodaniel/k8s/blob/develop/lib/k8s/client/mint/http_adapter.ex#L167

There allows for pringint all the requests that are errors while being submitted. From there I do indeed see a large request:

String.length(body) #=> 421097 

Printing the connection gives the following:

state.conn #=> %Mint.HTTP2{                                                                        
  transport: Mint.Core.Transport.SSL,                                                              
  socket: {:sslsocket, {:gen_tcp, #Port<0.43>, :tls_connection, :undefined},                       
   [#PID<0.1997.0>, #PID<0.1996.0>]},                                                              
  mode: :active,                                                                                   
  hostname: "127.0.0.1",                                                                           
  port: 46697,                                                                                     
  scheme: "https",                                                                                 
  state: :open,                                                                                    
  buffer: "",                                                                                      
  window_size: 65536000, 

The mint connection has a working decode table. It's state is open. And very suprisingly the window size is ~62MB. So this body should be well within the expected window size.

That results in the following error:

%Mint.HTTPError{                                                                         
  reason: {:exceeds_window_size, :request, 262144},                                                
  module: Mint.HTTP2                                                                               
}

According to existing git issues around this the window size should be re-calculated every time someone calls stream on the Mint connection. So that should be fine. However, according to https://hexdocs.pm/mint/Mint.HTTP2.html#get_window_size/2 there's also per request size.

So a couple of questions:

• I see a bunch of streams attached to the connection in state: :half_closed_local. Is there a leak somewhere in the streams runners that's not correctly releasing window?
• Do we just need to put some code in K8s.Client.Mint.HTTPAdapter that checks the window size for the request and streams it if larger.

[mruoss]

Hi @elliottneilclark

First of all (sorry, I only just realised now it's you): A big thanks for the sponsoring! I really appreciate it!

That being said, I'm really disappointed you're working witch such big resources. That's really not okay!!! ;)
Just kidding, thanks for bringing it up and investigating the problem. I was not aware of this limit and it seems like a non-trivial problem to solve but I'll start by creating a failing regression test and take it from there.

But I'm with you on the two assumptions:

• There is indeed a leak somewhere
• The body is just too big so we have to use Mint.HTTP.stream_request_body/3. They suggest to use Mint.HTTP2.get_window_size/2 to check whether it's necessary or not.

[elliottneilclark]

A big thanks for the sponsoring! I really appreciate it!

Thank you for all your work!

The body is just too big so we have to use Mint.HTTP.stream_request_body/3.

I poked around at doing that super simply and I didn't see success. So I was going to follow the lead of Tesla http: https://github.com/elixir-tesla/tesla/blob/master/lib/tesla/adapter/mint.ex#L164

I'll start by creating a failing regression test and take it from there.

Thank you again. Let me know if I can help in any way.

[mruoss]

I'm a bit confused about the discrepancy in the numbers. Locally and in the failing tests I see {:exceeds_window_size, :request, 1_048_576} while you're reporting {:exceeds_window_size, :request, 262_144}. So I'm wondering whether there's a leak nontheless...

[mruoss]

I poked around at doing that super simply and I didn't see success. So I was going to follow the lead of Tesla http: https://github.com/elixir-tesla/tesla/blob/master/lib/tesla/adapter/mint.ex#L164

Yeah, Tesla leaves checking body size to the user: https://github.com/elixir-tesla/tesla/blob/dfc4ea56358fb1be593ae517030648602e38b731/lib/tesla/adapter/mint.ex#L37

Processing of the chunks and checking body size must be done by yourself.