Understanding slow http_client output about benthos

[Jeffail]

Hey @ZachTB123, thanks for mentioning the jsonschema issue, it should be fixed with 8a48856.

Have you tried replacing the kafka_franz input with generate to get some test messages through? Once the request is successful the next thing is acknowledging the message and finalising the transaction, which includes registering the message checkpoint for commit. This shouldn't be blocked on anything that the kafka input is doing but it'd be good to rule it out.

[Jeffail]

Oh hang on I think I know what's happening, the latency is a bit skewed because it covers the entire batch of messages, so in your case you're probably looking at a trace of a message thats one of a batch of 100 or so and its output trace isn't finished until the entire batch is delivered.

You aren't actually benefitting from batching right now because https://www.benthos.dev/docs/components/outputs/http_client#batch_as_multipart isn't enabled and you aren't archiving the data to reduce the number of requests. I'd say either eliminate the batches by using a split processor to break those messages out, don't do any batching at all, and then maybe try increasing https://www.benthos.dev/docs/components/outputs/http_client#max_in_flight.