Support additional json encoders/formatters about quarkus-logging-json

[nadworny]

It works as suggested, see the updated serializer class.

[dfa1]

hello,

I had same problem... my solution was simply to disable all all other fields:

#105 (comment)

Hope it helps!

[dfa1]

yes, I had exactly same constraint!

You can disable everything and you just get a starting {

[dfa1]

well, you have several options:

1. parse and copy all keys to the generator. Pseudo-code:

Json json = Json.parse(ecsFormatter.format(event));
writeToGenerator(generator, key, json)

def writeToGenerator(generator, key, value) {
   if (value instanceof JsonNumber) {
      generator.writeNumber(key, value);
   } else if (value instanceof JsonString) {
     generator.writeString(key, value);
   } else if (value instanceof JsonObject) {
      for (key in keys(value)) {
         writeToGenerator(generator, key, value.get(key))
      }
   } // TODO: handle arrays too

it could be a bit tricky since it is a recursive algorithm. And it is not very efficient too (since you have to parse JSON and recreate it with another shape).

2. reimplement the ECS format directly using the generator. You could use EcsJsonFormatter as reference:

https://github.com/elastic/ecs-logging-java/blob/fb287cc003ca48cbb9440243b50d79031e2dfa4f/ecs-logging-core/src/main/java/co/elastic/logging/EcsJsonSerializer.java

in both cases you have to disable all providers (like per my original comment).

3. Another option would be to write a custom quarkus extension that uses directly the EcsFormatter to format the log. Basically providing another implementation of ExtFormatter, like this library:

https://github.com/quarkiverse/quarkus-logging-json/blob/master/runtime/src/main/java/io/quarkiverse/loggingjson/JsonFormatter.java

Hope it helps :-)

PS
In my company we started with option 3. and then switched to option 2. few months back and we are quite happy now as we don't have to maintain a custom quarkus extension!

[dfa1]

@nadworny you're welcome fellow Zuricher ;)

According to the stacktrace:

       java.lang.RuntimeException: com.fasterxml.jackson.core.JsonGenerationException: Can not start an object, expecting field name (context: Object)
       at io.quarkiverse.loggingjson.JsonFormatter.format(JsonFormatter.java:37)

there is an unexpected JSON object. What is on line 37? Maybe this line?

    generator.writeObject(json);

if so, just use:

   generator.writeObjectFieldStart("log"); // will write "log": {
   generator.writeObjectFieldStart("origin"); // will write "origin": {
   generator.writeObjectFieldStart("file"); // will write "file": {

NB
with the generator you can skip "quoting as string" as the generator is safe

[dfa1]

here: https://github.com/elastic/ecs-logging-java

[SlyngDK]

Hi @nadworny @dfa1

I have released version 1.1.0, with ecs logging format.

quarkus.log.console.json.log-format=ecs

You are free to test that and give some feedback.

[SlyngDK]

I have added ecs.version in 1.1.1.
I am not sure about the log.origin part.

[nadworny]

Thanks but what I need is to replace the whole log statement with the ECS formatter. At the moment I'm only able to add the ECS formatted string as a new field which isn't going to be properly ingested by Elastic.

[nadworny]

So if I understood you correctly, when I disable all the fields and write a new field using generator, it should wrap it as a main json? I did the following but it still writes it as a new field and on top of that is also escaping the " in the strings. Here is how it looks:

{"":"{\"@timestamp\":\"2021-09-27T07:43:08.945Z\", \"log.level\": \"INFO\", \"message\":\"Live reload total time: 0.636s \", \"ecs.version\": \"1.2.0\",\"process.thread.name\":\"Timer-0\",\"log.logger\":\"io.quarkus.deployment.dev.RuntimeUpdatesProcessor\"}\n"}

Code:

    @Override
    public void writeTo(JsonGenerator generator, ExtLogRecord event) throws IOException {
        var ecsFormatter = new EcsFormatter();
        generator.writeStringField("", ecsFormatter.format(event));
    }

Could you please provide a code snippet of how you set it up?

[dfa1]

I cannot share my code (it is a custom formatter for another "unfied log format" we have internally in SIX).

But of course you can't do that:

    generator.writeStringField("", ecsFormatter.format(event));

ecsFormatter is giving you a JSON as a string... so you should first parse it back and then "copy" all the content in the generator.

Otherwise, you could propose a PR in the ECS repository to extend quarkus logging directly.

[nadworny]

I think I got you. But how do I "copy" all the content in the generator? Which method do you use? You don't have to paste here the whole code, just the one that is invoking the generator interface.

[nadworny]

Thanks a lot fellow Zuricher @dfa1 ;)

This is what I came up with:

• Serializer: https://gist.github.com/nadworny/5fc96555c5f5d24c919fcacc482e820c
• Formatter: https://gist.github.com/nadworny/a80d405b4b4fc50f6e078ca14f451ab4

Everything seems to be working properly except of the following:

    public static void serializeOrigin(JsonGenerator generator, String fileName, String methodName, int lineNumber) throws IOException {
        StringBuilder builderFileName = new StringBuilder();
        StringBuilder builderFunction = new StringBuilder();
        JsonUtils.quoteAsString(fileName, builderFileName);
        JsonUtils.quoteAsString(methodName, builderFunction);
        generator.writeStartObject();
        var json = Json.createObjectBuilder()
            .add("log", Json.createObjectBuilder()
                .add("origin", Json.createObjectBuilder()
                    .add("file", Json.createObjectBuilder()
                        .add("name", builderFileName.toString())
                        .add("line", lineNumber))
                    .add("function", builderFunction.toString()))).build();
        generator.writeObject(json);
        generator.writeEndObject();
    }

I'm getting this error: https://gist.github.com/nadworny/3fd4a8a79fd0c7e8112cfcc1aabaf786

Any idea what am I doing wrong?

[nadworny]

Thanks, I'll try it tomorrow.

You mentioned once:

Otherwise, you could propose a PR in the ECS repository to extend quarkus logging directly.

Which repository did you mean? Which part of my code do you think I could propose?

[nadworny]

Awesome, I'll test it this week

[nadworny]

hey @SlyngDK, it works, thanks a lot!

The only thing I'm missing but it's not a must atm are the log.origin and ecs.version fields.

[nadworny]

Take a look at the serializeOrigin https://gist.github.com/nadworny/5fc96555c5f5d24c919fcacc482e820c#file-ecsjsonquarkusserializer-java-L104 but I'm not sure if it's also part of 1.1.1 version? We are using 1.2.0...