quarkiverse / quarkus-logging-splunk

Quarkus extension to be able to send logs to a Splunk HTTP Event Collector

Quarkus Splunk extension

A Quarkus extension to send logs to a Splunk HTTP Event Collector (HEC).

To get started, add the dependency:

<dependency>
    <groupId>io.quarkiverse.logging.splunk</groupId>
    <artifactId>quarkus-logging-splunk</artifactId>
</dependency>

For more details, check the complete documentation.

This extension is based on the official Splunk's HEC client. But, it defines its own Java Logging Handler rather than using the official one, since it was not compatible with JBoss logger implementation used in Quarkus and, as such, prevents to record steps at build time.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

vietk 💻 🚧

rquinio1A 💻 🚧

Lindsey Burnett 💻 ⚠️

This project follows the all-contributors specification. Contributions of any kind welcome!

quarkus-logging-splunk's Issues

Add guide to extension metadata

When in Dev mode, the extension panel should show a link to the guide.

According to https://quarkus.io/guides/extension-metadata there is a guide metadata that likely trigger this.

Create maintainance branch for Quarkus 3.2 LTS

Now the new LTS 3.8 of Quarkus is out, we need to create a new long term maintenance branch for this extension.

The HttpEventCollectorSender does not read the encrypted configuration from the quarkus profile

The configurations for splunk (url, token) are encrypted in the yaml file in quarkus application using jasypt encrpytion (See https://docs.quarkiverse.io/quarkus-config-extensions/dev/jasypt.html). However, the encrypted values are not decrypted in HttpEventCollectorSender . It takes the part ENC(...) directly.

However, the encrypted value could be read normally using @ConfigProperty() or using ConfigProvider.getConfig().getConfigValue().

Async handler

We've noticed some contention in splunk library due to https://github.com/splunk/splunk-library-javalogging/blob/82c4a568341449828699a6877e9221d9cb772bd0/src/main/java/com/splunk/logging/HttpEventCollectorSender.java#L424-L428 , cf splunk/splunk-library-javalogging#257
Since that happens before the HTTP enqueuing, the parallel mode doesn't help.

Quarkus has an async mode for the built-in handlers (console, file, ...) but it's not wrapped automatically for other named handlers.

Potential memory issues when slow HEC responses

In another project using splunk-library-javalogging, it has been reported that if HEC endpoint is slow to answer because it's in bad shape, then this can lead to OutOfMemory on the client, because log event objects are not garbage collected on the client.

In sync mode, slow logging may slow some application threads, but there is no max "in-flight" log event batches
In async mode, using quarkus.log.handler.splunk.async and quarkus.log.handler.splunk.async.overflow=discard could mitigate the issue, if Quarkus starts droping logs after the queue limit has been reached. However the dequeing is probably not limited by anything today, so it could be the same problem.

If the number of "in-flight" batches is over a certain limit, we could drop log events/stop dequeing, rather than adding them into in a new batch.
This would require to have counters over in-flight vs completed log events, cf #57
cf splunk/splunk-library-javalogging#265
cf splunk/splunk-library-javalogging#98

GraalVM support

Hi,

I have a maven project which uses a lot of Quarkus libraries. I also need your fantastic Splunk integration which works fine. However, when I want to use GraalVM it gives me the following error:

Error: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved field during parsing: io.quarkus.bootstrap.logging.InitialConfigurator.DELAYED_HANDLER. To diagnose the issue you can use the --allow-incomplete-classpath option. The missing field is then reported at run time when it is accessed the first time.
Detailed message:
Trace:
at parsing io.quarkiverse.logging.splunk.SplunkErrorCallback.isConsoleHandlerEnabled(SplunkErrorCallback.java:68)
Call path from entry point to io.quarkiverse.logging.splunk.SplunkErrorCallback.isConsoleHandlerEnabled():
at io.quarkiverse.logging.splunk.SplunkErrorCallback.isConsoleHandlerEnabled(SplunkErrorCallback.java:67)
at io.quarkiverse.logging.splunk.SplunkErrorCallback.error(SplunkErrorCallback.java:55)
at com.splunk.logging.HttpEventCollectorErrorHandler.error(HttpEventCollectorErrorHandler.java:117)
at com.splunk.logging.HttpEventCollectorSender$3.completed(HttpEventCollectorSender.java:308)
at com.splunk.logging.HttpEventCollectorSender$4.onResponse(HttpEventCollectorSender.java:356)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:174)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.lang.Thread.run(Thread.java:829)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:553)
at com.oracle.svm.core.windows.WindowsJavaThreads.osThreadStartRoutine(WindowsJavaThreads.java:138)
at com.oracle.svm.core.code.IsolateEnterStub.WindowsJavaThreads_osThreadStartRoutine_4bc03aa26f8cdfc97ebd54050e8ae4bce1023851(generated:0)

Is there a solution for this?

Add quarkus.log.handler.splunk.filter

Add a config entry for filtering log events in https://github.com/quarkiverse/quarkus-logging-splunk/blob/main/runtime/src/main/java/io/quarkiverse/logging/splunk/SplunkLogHandler.java#L17 and apply it to the ExtLogHandler.

Note: when in async mode, Quarkus logging extensions register filters on the async thread rather than on the AsyncHandler, meaning filters cannot access RequestScope beans (quarkusio/quarkus#37334)

Filtering log entries

Does this extension support filtering log entries based on some criteria such that I can forward only a subset of my logs to Splunk? What I'm trying to do is log some entries to stdout and some to Splunk without having to build a Splunk client myself.

Generic mechanism for metadata name override/exclusion

Today we have few config property for enabling optional metatdata, but this is not generic:

quarkus.log.handler.splunk.include-exception=true
quarkus.log.handler.splunk.include-logger-name=true
quarkus.log.handler.splunk.include-thread-name=true

For MDC keys, it's either all included or none.

Also we may need to customize the field name, as there's no standard format in splunk (i.e. severity vs level vs severityText, logger vs category, etc. ).
This could also be used for MDC keys remapping (traceId vs trace_id, etc.)

Other structured handlers have implemented this: quarkusio/quarkus#23977

OTel splunk exporter has similar remapping for OTel metadata names to splunk field name, but it's not generic:
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/splunkhecexporter

Add contributors

associate Splunk log handler to a category

Hi,
is it possible to associate a category (i.e. "org.hibernate") with the Splunk log handler so to send only specific log events to Splunk? It's not clear to me from the docs.

Handler shutdown is not graceful: it does not wait all log to be sent

When the log handler is being closed, the underlying HttpEventCollectorSender is closed as well.
However by default, the sender is only shutting down the executor service. It has two impacts :

1. Queued logs will be not sent
2. Some error log may arise when queued logs like this one Error while sending events to Splunk HEC: executor rejected java.io.InterruptedIOException: executor rejected

It's a valid use case that an application could try to send its logs during a certain period of time to ensure persistance in Splunk

Implementation idea :

• We could configure and pass the to TimeoutSettings structure to the HttpEventCollectorSender constructor with a non-0 default termination timemout
• Pay attention the same structure is used in flushHttpClient that is called on every flush (i.e the flush become synchronous until timeout is reached)

Readiness issue in Native Test

We have a native test that does two things:

1. Spunk extension is working natively
2. Check the integration of the extension with splunk running in a TestContainer docker container

The integration test with Splunk is flaky because despite efforts to be sure the container is ready, it seems it's not enough.

To solve this we have to find a better way to test the readiness of Splunk: there's a Splunk API to get the health status

Invalid units for batch interval and default batch size

Regarding batch-interval, splunk HttpEventCollectorSender expects milliseconds and here, we provide seconds.

Regarding batch-size, the doc says 10KB, but the value is 10B (missing a 1024 factor).

See for example the constants used on splunk library side
https://github.com/splunk/splunk-library-javalogging/blob/cf59a50d63ef495398f30a8f59b96b5bb62962d5/src/main/java/com/splunk/logging/HttpEventCollectorSender.java#L68-L74

Support named handlers

Check if something needs to be done tu support "named handlers" (cf https://quarkus.io/guides/logging#category-named-handlers-example) and document it.

This would allow to have 2 Splunk handlers configured differently ( different indexes, different serialization format, ...)

An example of usage would be the default handler for regular logs and a special handler for functional monitoring

quarkus.log.handler.splunk."MONITORING".enable=true
quarkus.log.handler.splunk."MONITORING".index=otherindex
quarkus.log.handler.splunk."MONITORING".format=%s%e
quarkus.log.handler.splunk."MONITORING".metadata-source-type=monitoring
quarkus.log.handler.splunk."MONITORING".serialization=nested
quarkus.log.category."com.company.monitoring".handlers=MONITORING

Split dev services into a separate module

Hello,

The dev services PR added a lot of dependencies that we do not want to include in a production application (TestContainers, junit, docker libs etc.) for various reasons including security
#242

I am using this extension in keycloak and the application fails to start if those dependencies are excluded due to the presence of io.quarkiverse.logging.splunk.DevServicesLoggingSplunkProcessor in the classpath (see error below)

All the dev services related code + dependencies should moved into a separate module that can be easily excluded at runtime, as is done for the core quarkus extensions
e.g. https://github.com/quarkusio/quarkus/blob/main/extensions/jdbc/jdbc-postgresql/deployment/pom.xml#L26
https://github.com/quarkusio/quarkus/blob/main/extensions/devservices/postgresql/pom.xml

ERROR: Failed to run 'build' command.
Error details:
java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doReaugment(QuarkusEntryPoint.java:90)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:49)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:33)
	at org.keycloak.quarkus.runtime.cli.command.Build.run(Build.java:85)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2026)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
	at picocli.CommandLine.execute(CommandLine.java:2170)
	at org.keycloak.quarkus.runtime.cli.Picocli.parseAndRun(Picocli.java:125)
	at org.keycloak.quarkus.runtime.KeycloakMain.main(KeycloakMain.java:98)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:62)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:33)
Caused by: java.lang.RuntimeException: Failed to load steps from class io.quarkiverse.logging.splunk.DevServicesLoggingSplunkProcessor
	at io.quarkus.deployment.ExtensionLoader.loadStepsFrom(ExtensionLoader.java:164)
	at io.quarkus.deployment.QuarkusAugmentor.run(QuarkusAugmentor.java:107)
	at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:330)
	at io.quarkus.runner.bootstrap.AugmentActionImpl.createProductionApplication(AugmentActionImpl.java:175)
	at io.quarkus.deployment.mutability.ReaugmentTask.main(ReaugmentTask.java:69)
	... 24 more
Caused by: java.lang.NoClassDefFoundError: org/testcontainers/containers/GenericContainer
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:508)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:468)
	at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
	at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3402)
	at java.base/java.lang.Class.getDeclaredMethods(Class.java:2504)
	at io.quarkus.deployment.ExtensionLoader.getMethods(ExtensionLoader.java:922)
	at io.quarkus.deployment.ExtensionLoader.loadStepsFromClass(ExtensionLoader.java:432)
	at io.quarkus.deployment.ExtensionLoader.loadStepsFrom(ExtensionLoader.java:162)
	... 28 more
Caused by: java.lang.ClassNotFoundException: org.testcontainers.containers.GenericContainer
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:520)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:468)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:518)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.loadClass(QuarkusClassLoader.java:468)
	... 38 more

DevService Ideas for Splunk

Have you thought about adding a DevService using Splunk Free?

Or I would love some docs on how to configure it for local Splunk for testing. For example this Docker Compose file:

version: "3.6"

services:
  splunk:
    image: splunk/splunk:latest
    container_name: splunk
    environment:
      - SPLUNK_START_ARGS=--accept-license
      - SPLUNK_LICENSE_URI=Free
      - SPLUNK_PASSWORD=splunkadmin
    ports:
      - "8000:8000"  # Splunk Web Interface
      - "8088:8088"  # HEC (HTTP Event Collector)
    volumes:
      - splunk_data:/opt/splunk/var/lib/splunk
      
volumes:
  splunk_data:

What is the Quarkus Logging Splunk configuration to hit this localhost:8088 docker compose stack?

Splunk config extension documentation is missing

Apparently we have no more access to the config extension, users are complaining they can't really see what is configurable

Support for JSON formatter mapping to the Splunk

Can we please share some insights on how to add JSON formatter to the log messages using this library?

Support for raw events

Hi,

any plans to support raw events? If you're OK with that, I can try to put a PR together.

Thanks,

lorenzo

Which version of opentelemetry does this work with?

Noticing that the Quarkus opentelemtry libraries are at version 1.6 and the latest library is 1.9.1

Any ideas on how to set up gradle to use the correct opentelemtry library.

• assming that's required.

Support flat event format

Today the extension serializes structured logs via nested JSON:

{
  "time": "timestamp",
  "host": "hostname",
  "source": "mysource",
  "sourcetype": "mysourcetype",
  "index": "myindex",
  "event": {
    "message": "2023-01-05 ERROR The log message",
    "logger": "com.acme.MyClass",
    "severity": "ERROR",
    "properties": { // Dynamic metadata
      "property-key": "property-value"
    }
  },
  "fields": { // Static metadata
  }
}

This is the default behavior of splunk-java-logging library.

However there's another format which appears more "standard" and makes sense to support

{
  "time": "timestamp",
  "host": "hostname",
  "source": "mysource",
  "sourcetype": "mysourcetype",
  "index": "myindex",
  "event": "2023-01-05 ERROR The log message",
  "fields": { // Static and dynamic metadata
    "level": "ERROR",
    "property-key": "property-value"
  }
}

It used in particular by Otel splunk exporter (https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/splunkhecexporter / https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#splunk-hec )
Splunk itself doesn't define any standard metadata name for error level/severity so these make sense to be configurable.

Documentation for high throughput logging

With the #115 merged there's two asynchronous system for logging quarkus + splunk-library-logging, we should write a documentation about how and what to configure to achieve logging with high throughput

HEC client metrics

Some metrics linked to the HEC client (HTTP connection pool, number of events batch ok/failed, etc.) would be interesting for monitoring purposes.