logstash-plugins / logstash-codec-java_codec_example

EXPERIMENTAL: Example Java codec plugin for Logstash to bootstrap your effort to write your own Java codec plugin!

Logstash Java Plugin

This is a Java plugin for Logstash.

It is fully free and fully open source. The license is Apache 2.0, meaning you are free to use it however you want.

The documentation for Logstash Java plugins is available here.

logstash-codec-java_codec_example's Issues

[Question] > Could not read script ... rubyUtils.gradle as it does not exist.

I just got the err when i run

gradlew.bat assemble

And https://www.elastic.co/guide/en/logstash/6.6/java-filter-plugin.html#_set_up_your_environment_2 No declaration of rubyUtils.gradle ,

so, what can i do for it?

thank any help!!

Test failed

According to the steps of https://www.elastic.co/guide/en/logstash/current/java-codec-plugin.html#_feedback_2, Run Logstash with the Java codec plugin, failed。
Using windows system, the command is echo "foo,bar" | logstash.bat -e "input {java_stdin {codec => java_codec_example} }"
ERROE:
C:\develop\logstash-7.16.2\bin>echo "foo,bar" | logstash.bat -e "input { java_stdin { codec => java_codec_example } }"
Using LS_JAVA_HOME defined java: C:\develop\IdeaProjects\logstash-7.16.2\jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to C:/develop/logstash-7.16.2/logs which is now configured via log4j2.properties
[2021-12-28T14:19:45,232][INFO ][logstash.runner ] Log4j configuration path used is: C:\develop\logstash-7.16.2\config\log4j2.properties
[2021-12-28T14:19:45,247][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.16.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [mswin32-x86_64]"}
[2021-12-28T14:19:45,404][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-12-28T14:19:46,951][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2021-12-28T14:19:47,493][INFO ][org.reflections.Reflections] Reflections took 86 ms to scan 1 urls, producing 119 keys and 417 values
[2021-12-28T14:19:48,038][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: java.lang.IllegalStateException: java.lang.NoSuchMethodException: org.logstashplugins.JavaCodecExample.(java.lang.String, co.elastic.logstash.api.Configuration, co.elastic.logstash.api.Context)", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "C_3a_.develop.logstash_minus_7_dot_16_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(C:/develop/logstash-7.16.2/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.develop.logstash_minus_7_dot_16_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(C:/develop/logstash-7.16.2/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "C_3a_.develop.logstash_minus_7_dot_16_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(C:/develop/logstash-7.16.2/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.develop.logstash_minus_7_dot_16_dot_2.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(C:/develop/logstash-7.16.2/logstash-core/lib/logstash/agent.rb:383)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
warning: thread "Converge PipelineAction::Create

" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>
create at org/logstash/execution/ConvergeResultExt.java:135
add at org/logstash/execution/ConvergeResultExt.java:60
converge_state at C:/develop/logstash-7.16.2/logstash-core/lib/logstash/agent.rb:396
[2021-12-28T14:19:48,047][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>"}
[2021-12-28T14:19:48,058][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in create'", "org/logstash/execution/ConvergeResultExt.java:60:in add'", "C:/develop/logstash-7.16.2/logstash-core/lib/logstash/agent.rb:396:in `block in converge_state'"]}
[2021-12-28T14:19:48,065][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at C_3a_.develop.logstash_minus_7_dot_16_dot_2.lib.bootstrap.environment.(C:\develop\logstash-7.16.2\lib\bootstrap\environment.rb:94) ~[?:?]

How to define 'LOGSTASH_CORE_PATH'?

Could not get unknown property 'LOGSTASH_CORE_PATH' for object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.
Open File

compile fileTree(dir: LOGSTASH_CORE_PATH, include: "**/logstash-core-?.?.?.jar")

Try on java codec plugin

Build, package gem, then installed the Java plugin in Logstash
Now, Run Logstash with the Java codec plugin to test:

`echo "foo,bar" | $LS_HOME/bin/logstash -e 'input { java_stdin { codec => java_codec_example } }'

{
"@timestamp" => 2019-11-11T09:08:30.159Z,
"hostname" => "fr-133",
"message" => "foo",
"@Version" => "1"
}
{
"@timestamp" => 2019-11-11T09:08:30.159Z,
"hostname" => "fr-133",
"message" => "bar\n",
"@Version" => "1"
}
`

Test a config as below:

test_plugin.conf

`
input {
stdin {}
}

output {
stdout {
codec => java_codec_example
}
}`

$LS_HOME/bin/logstash -f test_plugin.conf

Error:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-11-11 09:10:41.173 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-11-11 09:10:41.191 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.4.2"}
[INFO ] 2019-11-11 09:10:43.820 [Converge PipelineAction::Create

] Reflections - Reflections took 50 ms to scan 1 urls, producing 20 keys and 40 values
[ERROR] 2019-11-11 09:10:44.312 [Converge PipelineAction::Create] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (NoMethodError) undefined method new' for class org.logstashplugins.JavaCodecExample:Java::JavaClass", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:100)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:60)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1156)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1143)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:26)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.Block.call(Block.java:129)", "org.jruby.RubyProc.call(RubyProc.java:295)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.RubyProc.call(RubyProc.java:270)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]} warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true): LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateExceptionforPipelineAction::Createcreate at org/logstash/execution/ConvergeResultExt.java:109 add at org/logstash/execution/ConvergeResultExt.java:37 converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:339 [ERROR] 2019-11-11 09:10:44.322 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handleJava::JavaLang::IllegalStateExceptionforPipelineAction::Create", :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in create'", "org/logstash/execution/ConvergeResultExt.java:37:in add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:339:in block in converge_state'"]}
[FATAL] 2019-11-11 09:10:44.374 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:109:in create'", "org/logstash/execution/ConvergeResultExt.java:37:in add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:339:in `block in converge_state'"]}
[ERROR] 2019-11-11 09:10:44.451 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit