http over amqp_0_9 proxy about benthos

[mihaitodor]

Hey @jandunk 👋 Please find some answers inline:

This should then somehow be picked up by the Benthos config which is listening as a webserver and should sync_response the webserver's response from the amqp_0_9 queue which contains the response matching the id added at the input.

I'm not sure I follow. Is this a 3rd Benthos instance? The one you listed above as "inner config" uses amqp_0_9 as the output (and stdout) and there's no sync_response output there.

Could someone help me to implement the logic for injecting and matching the id and syncing the response from the test_response queue in the inner config?

I don't think there's a way right now to a) send a message to an amqp_0_9 output and then b) wait for a response on another amqp_0_9 topic before c) doing a sync_response to the http_server input. I suppose one could hack something together using Benthos in Streams Mode where one stream has the http_server input and reads from a shared cache in a while loop after writing a message over amqp, but I wouldn't want to implement something like that.

The best thing to do is to develop a custom processor plugin based on the existing amqp_0_9 output with some additions such that it can a) write messages to AMQP just like the current output and, also b) block until it receives the expected response on another topic. Then you can use the sync_response output as expected. Since the APIs needed to replicate the sync_response output behaviour are internal, implementing it as a processor is the only reasonable approach currently. Not sure what edge cases you might encounter when considering parallel processing and retries...

LE: Moving this to a discussion as per #2026.