Add "metadata" field to the Message struct about broadway

[msaraiva]

Hi folks!

Here's the suggestion to provide message attributes using the metadata field:

The new configuration will accept:

  producers: [
    default: [
      module: {BroadwaySQS.Producer,
        queue_name: "my_queue",
        metadata: [:message_id, :receipt_handle]
      }
    ]
  ]

Where metadata is a list of attributes to be retrieved. Then, you can access the attributes via message.metadata:

 def handle_message(_, message, _) do
    IO.inspect(message.metadata.message_id, label: "Msg id")
    message
  end

The list of available attributes should be defined and documented by the client. On SQS, we'll start with: :message_id, :receipt_handle, :attributes and :message_attributes. We'll also take a look at RabbitMQ to decide which group of attributes we'll make available and how. @narrowtux, in case you already have suggestions, please let us know.

@dokie, with this change, it's most likely that will revert this commit since both attributes from the receipt will already be available in our new metadata field.

I'll start to work on this sometime this week so any feedback/suggestion is more than welcome.

-marlus

[msaraiva]

@gvaughn that's exactly suggestion 1. We expose :message_attribute_names and :attribute_names in the :client_opts so it can be forward to the server. Then we treat the response and extract those values and add them to the new metadata field, along with message_id, receipt_handle and md5_of_body. The data field of the Message struct will remain unchanged and represents the body, the metadata will hold all other values. The only difference from the original proposal is that we're not adding a :metadata option anymore, we're adding only the metadata field to the Message struct.

[narrowtux]

Same thing would be interesting for the RabbitMQ producer as well

[gvaughn]

I recommend you also include message_attributes SQS considers them different.

Edit: I just noticed which Repo I clicked to. This might be confusing with the SQS terminology. Perhaps metadata is what you'd want to call it in the Message struct level?

[msaraiva]

Yes, definitely. We've discussed this earlier today but I forgot to update the title :)

[narrowtux]

for RabbitMQ I only ever needed the routing key - I'll have a look at the other metadata you can receive there

[msaraiva]

In order to retrieve any of the attributes and/or message attributes, I'll need to pass :attribute_names and/or :message_attribute_names to ExAws.SQS.receive_message/2. Consequently, in order to keep the API as suggested before, I would need to pass both of those options as :all, which is definitely undesirable since it will increase data traffic unnecessarily for most cases.

Suggestion 1

Instead of creating a new option :metadata, we could expose :attribute_names and :message_attribute_names to the user. The values, however, could still be set on the metadata field of the %Message{} as originally planned. I would also always add message_id, receipt_handle and md5_of_body to the metadata since they are always returned by the SQS service.

Suggestion 2

We don't set the :data field with the body but with the whole message coming from the server. This would be a breaking change and it will make the metadata field useless when working with the SQS client. The whole message would look like this:

%{
  body: "Mensagem 9",
  message_id: "...",
  receipt_handle: "..."
  md5_of_body: "...",
  attributes: %{
    "approximate_first_receive_timestamp" => "...",
    "approximate_receive_count" => 1,
    "sender_id" => "...",
    "sent_timestamp" => "..."
  },
  message_attributes: %{
    "TestStringAttribute" => %{
      binary_value: "",
      data_type: "String",
      name: "TestStringAttribute",
      string_value: "A test",
      value: "A test"
    }
  }
}

Personally, I would go with suggestion 1. Thoughts?

[gvaughn]

@msaraiva I had expected to see :message_attribute_names and :attribute_names inside the :client_opts given to the SQSClient. As long as those are validated and handled properly, SQS will include the data in the results. The main issue in the Broadway project is to have a place in the Message to put them when the SQSClient does wrap_received_messages