http-log create incorrect elasticsearch mapping and trigger parse error about kong

[samugi]

hi @gzOne2Free, thanks for opening this issue. It looks like this may be related with your Elasticsearch mapping, I found this stackoverflow which seems related.

In short:

instead of declaring the whole array as an object, speficy the array entries' data type (text) directly

"consumer": {
(unchanged)
  "properties": {
    "tags": {
      "type": "text"
    }
(unchanged)

Also, you may consider using keyword instead of text.

Could you give it a try and let me know if it works for you?

[gzOne2Free]

@samugi
When I create a new index with desired mapping and update http-log endpoint, new kong access logs can send to ES correctly. Thanks~

But how can I get/prepare the full http-log fields definition when initial a new http-log stream to ES? As I seen in access log through file-log plugin, every access log record has vary/different section(such as service, route, cosumer etc.) by each other. So I don't know how to get/generate a FULL list of access log fields. Any idea?

[samugi]

Hi @gzOne2Free, I'm really glad that helped!

The http-log plugin uses the Kong PDK log serializer to produce access log records. More information on the available fields can be found in our docs, also the relevant source code is here.

For more information about how to generate the Elasticsearch mappings my advice would be to refer to the Elasticsearch official docs. I believe mappings do not define fields as required so in this case it should be enough to include all the fields that you may expect coming from the http-log plugin output as you say.

I'm closing this issue as resolved since your original problem was addressed. Please feel free to open up a new issue or discussion if you have any questions regarding Kong or its plugins. Thank you!