jordansissel / grok Goto Github PK

Does not compile with clang from Xcode 12 or later on macOS:

conf.tab.c:1429:16: error: implicit declaration of function 'yylex' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
      yychar = YYLEX;
               ^
conf.tab.c:745:16: note: expanded from macro 'YYLEX'
# define YYLEX yylex (&yylval, &yylloc)
               ^

I downloaded the code from the v0.9.2 tag and after fixing a few issues reported elsewhere in this issue tracker, I got something that built. But it reported an unexpected version:

$ grok --version
grok 1.20220114.1

2022-01-14 is today's date. I expect if I build it tomorrow, it will report tomorrow's date. That makes the build non-reproducible.

I had expected to see either "grok 0.9.2" or perhaps "grok 1.20111028.1" (2011-10-28 being the date when the v0.9.2 tag was created).

Let's say that I have this pattern: "<%{NUMBER}>%{SYSLOGTIMESTAMP:timestamp} %{DATA}: \\|%{NUMBER}\\|%{DATA:qtype}\\|

And when I grok this text "<116>Dec 11 15:29:51 SyslogAlertForwarder: |6583514977575335971|Signature"

How can I get all the fields by subname like timestamp=Dec 11 15:29:51, qtype=Signature the fastest way in code? (Best performance)

I call grok_exec to check whether the text match the pattern. But I don't know how to get the fields' value in the pattern.

    len = grok_exec(&grok, text, &output);
        if (len == 0) {
          match = true;
        }

Thanks!

pygrok, the python bindings to grok C++ are long gone but never forgotten. Given python's omnipresence in the server world, would it be possible to ressurect it?

From logs:

so idk enough about bash to make the pr myself
but I have a small change to the grok makefile to make it work on os x
or at least with homebrew

    if command -v brew >/dev/null; then
    PREFIX?=$(brew --prefix)
    end

right around here: https://github.com/jordansissel/grok/blob/master/Makefile
That will make grok compile correctly under homebrew, assuming you have libevent and tokyo cabinet installed. I tried searching for directions yesterday, but failed to find how to make multiple if statements

Running grok on RHEL5

Internal compiler error: unrecognized character after (?<
Regexp: (?!<\\)%\{(?<name>(?<pattern>[A-z0-9]+)(?::(?<subname>[A-z0-9_:]+))?)(?:=(?<definition>(?:(?P<curly2>\{(?:(?>[^{}]+|(?>\\[{}])+)|(?P>curly2))*\})+|(?:[^{}]+|\\[{}])+)+))?\s*(?<predicate>(?:(?P<curly>\{(?:(?>[^{}]+|(?>\\[{}])+)|(?P>curly))*\})|(?:[^{}]+|\\[{}])+)+)?\}

Clean Ubuntu Server 12.04 LTS following https://github.com/jordansissel/grok/blob/master/INSTALL

apt-get install build-essential unzip -y
apt-get install bison ctags flex gperf libevent-dev libpcre3-dev libtokyocabinet-dev -y
wget https://github.com/jordansissel/grok/archive/master.zip -O grok.zip
unzip grok.zip
cd grok-master && make grok

Outputs:

sh ./version.sh --header > grok_version.h
[ -f grok_capture_xdr.h ] && rm grok_capture_xdr.h || true
rpcgen -h grok_capture.x -o grok_capture_xdr.h
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  grok.c -o gr                                                                                                                     ok.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  grokre.c -o                                                                                                                      grokre.o
grokre.c: In function âgrok_pattern_expandâ:
grokre.c:312:7: warning: passing argument 1 of âfreeâ discards âconstâ qualifier                                                                                                                      from pointer target type [enabled by default]
/usr/include/stdlib.h:488:13: note: expected âvoid *â but argument is of type âc                                                                                                                     onst char *â
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  grok_capture                                                                                                                     .c -o grok_capture.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  grok_pattern                                                                                                                     .c -o grok_pattern.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  stringhelper                                                                                                                     .c -o stringhelper.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  predicates.c                                                                                                                      -o predicates.o
[ -f grok_capture_xdr.c ] && rm grok_capture_xdr.c || true
rpcgen -c grok_capture.x -o grok_capture_xdr.c
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  grok_capture                                                                                                                     _xdr.c -o grok_capture_xdr.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  grok_match.c                                                                                                                      -o grok_match.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  grok_logging                                                                                                                     .c -o grok_logging.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  grok_program                                                                                                                     .c -o grok_program.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  grok_input.c                                                                                                                      -o grok_input.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  grok_matchco                                                                                                                     nf.c -o grok_matchconf.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  libc_helper.                                                                                                                     c -o libc_helper.o
/usr/bin/gperf grok_matchconf_macro.gperf > grok_matchconf_macro.c
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  grok_matchco                                                                                                                     nf_macro.c -o grok_matchconf_macro.o
/usr/bin/gperf filters.gperf > filters.c
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  filters.c -o                                                                                                                      filters.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  grok_discove                                                                                                                     r.c -o grok_discover.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  conf.tab.c -                                                                                                                     o conf.tab.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  conf.yy.c -o                                                                                                                      conf.yy.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  main.c -o ma                                                                                                                     in.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux  grok_config.                                                                                                                     c -o grok_config.o
cc -ldl -lpcre -levent -rdynamic -ltokyocabinet -L/usr/local/lib  -levent grok.o                                                                                                                      grokre.o grok_capture.o grok_pattern.o stringhelper.o predicates.o grok_capture                                                                                                                     _xdr.o grok_match.o grok_logging.o grok_program.o grok_input.o grok_matchconf.o                                                                                                                      libc_helper.o grok_matchconf_macro.o filters.o grok_discover.o conf.tab.o conf.y                                                                                                                     y.o main.o grok_config.o -o grok
grok.o: In function `grok_pcre_callout':
grok.c:(.text+0x66): undefined reference to `dlopen'
grok.c:(.text+0x72): undefined reference to `dlsym'
grok.o: In function `grok_init':
grok.c:(.text+0x1b5): undefined reference to `pcre_callout'
grok.c:(.text+0x213): undefined reference to `tctreenew'
grok.c:(.text+0x21c): undefined reference to `tctreenew'
grok.c:(.text+0x225): undefined reference to `tctreenew'
grok.c:(.text+0x22e): undefined reference to `tctreenew'
grok.c:(.text+0x237): undefined reference to `tctreenew'
grok.c:(.text+0x273): undefined reference to `pcre_compile'
grok.c:(.text+0x2a0): undefined reference to `pcre_fullinfo'
grok.c:(.text+0x2b3): undefined reference to `pcre_get_stringnumber'
grok.c:(.text+0x2cc): undefined reference to `pcre_get_stringnumber'
grok.c:(.text+0x2e5): undefined reference to `pcre_get_stringnumber'
grok.c:(.text+0x2fe): undefined reference to `pcre_get_stringnumber'
grok.c:(.text+0x317): undefined reference to `pcre_get_stringnumber'
grokre.o: In function `grok_free_clone':
grokre.c:(.text+0x10): undefined reference to `pcre_free'
grokre.c:(.text+0x3c): undefined reference to `tctreedel'
grokre.c:(.text+0x4a): undefined reference to `tctreedel'
grokre.c:(.text+0x58): undefined reference to `tctreedel'
grokre.o: In function `grok_compilen':
grokre.c:(.text+0xef): undefined reference to `tctreeclear'
grokre.c:(.text+0xf8): undefined reference to `tctreeclear'
grokre.c:(.text+0x101): undefined reference to `tctreeclear'
grokre.c:(.text+0x10a): undefined reference to `tctreeclear'
grokre.c:(.text+0x1e9): undefined reference to `pcre_exec'
grokre.c:(.text+0x242): undefined reference to `pcre_get_substring'
grokre.c:(.text+0x2e6): undefined reference to `pcre_get_substring'
grokre.c:(.text+0x304): undefined reference to `pcre_get_substring'
grokre.c:(.text+0x568): undefined reference to `pcre_free_substring'
grokre.c:(.text+0xa86): undefined reference to `pcre_compile'
grokre.c:(.text+0xaa6): undefined reference to `pcre_fullinfo'
grokre.c:(.text+0xadb): undefined reference to `pcre_fullinfo'
grokre.c:(.text+0xaf3): undefined reference to `pcre_fullinfo'
grokre.c:(.text+0xb08): undefined reference to `pcre_fullinfo'
grokre.o: In function `grok_execn':
grokre.c:(.text+0xe5a): undefined reference to `pcre_exec'
grokre.o: In function `grok_free_clone':
grokre.c:(.text+0x67): undefined reference to `tctreedel'
grokre.o: In function `grok_free':
grokre.c:(.text+0x94): undefined reference to `tctreedel'
grok_capture.o: In function `grok_capture_add':
grok_capture.c:(.text+0x93): undefined reference to `tctreeput'
grok_capture.c:(.text+0xae): undefined reference to `tctreeput'
grok_capture.c:(.text+0xc2): undefined reference to `tctreeget'
grok_capture.c:(.text+0xd6): undefined reference to `tclistnum'
grok_capture.c:(.text+0x105): undefined reference to `tclistval'
grok_capture.c:(.text+0x11d): undefined reference to `tclistremove'
grok_capture.c:(.text+0x12d): undefined reference to `tclistpush'
grok_capture.c:(.text+0x145): undefined reference to `tctreeput'
grok_capture.c:(.text+0x15a): undefined reference to `tctreeget'
grok_capture.c:(.text+0x16e): undefined reference to `tclistnum'
grok_capture.c:(.text+0x195): undefined reference to `tclistval'
grok_capture.c:(.text+0x1ad): undefined reference to `tclistremove'
grok_capture.c:(.text+0x1bd): undefined reference to `tclistpush'
grok_capture.c:(.text+0x1d6): undefined reference to `tctreeput'
grok_capture.c:(.text+0x22a): undefined reference to `tclistnew'
grok_capture.c:(.text+0x237): undefined reference to `tclistnew'
grok_capture.o: In function `grok_capture_get_by_id':
grok_capture.c:(.text+0x26c): undefined reference to `tctreeget'
grok_capture.o: In function `grok_capture_get_by_name':
grok_capture.c:(.text+0x2a3): undefined reference to `tctreeget'
grok_capture.c:(.text+0x2b9): undefined reference to `tclistval'
grok_capture.o: In function `grok_capture_get_by_subname':
grok_capture.c:(.text+0x2f3): undefined reference to `tctreeget'
grok_capture.c:(.text+0x309): undefined reference to `tclistval'
grok_capture.o: In function `grok_capture_get_by_capture_number':
grok_capture.c:(.text+0x33c): undefined reference to `tctreeget'
grok_capture.o: In function `grok_capture_walk_next':
grok_capture.c:(.text+0x6e3): undefined reference to `tctreeiternext'
grok_capture.c:(.text+0x706): undefined reference to `tctreeget'
grok_capture.o: In function `grok_capture_walk_init':
grok_capture.c:(.text+0x6c5): undefined reference to `tctreeiterinit'
grok_pattern.o: In function `grok_pattern_name_list':
grok_pattern.c:(.text+0xb): undefined reference to `tclistnew'
grok_pattern.c:(.text+0x16): undefined reference to `tctreeiterinit'
grok_pattern.c:(.text+0x2b): undefined reference to `tclistpush'
grok_pattern.c:(.text+0x38): undefined reference to `tctreeiternext'
grok_pattern.o: In function `grok_pattern_add':
grok_pattern.c:(.text+0xc7): undefined reference to `tctreeput'
grok_pattern.o: In function `grok_pattern_find':
grok_pattern.c:(.text+0x11c): undefined reference to `tctreeget'
predicates.o: In function `grok_predicate_regexp_init':
predicates.c:(.text+0x6b): undefined reference to `pcre_exec'
predicates.c:(.text+0x32f): undefined reference to `pcre_compile'
grok_program.o: In function `grok_collection_init':
grok_program.c:(.text+0x34): undefined reference to `event_init'
grok_program.c:(.text+0x69): undefined reference to `event_set'
grok_program.c:(.text+0x74): undefined reference to `event_add'
grok_program.o: In function `grok_collection_check_end_state':
grok_program.c:(.text+0x155): undefined reference to `event_base_loopexit'
grok_program.o: In function `_collection_sigchld':
grok_program.c:(.text+0x380): undefined reference to `event_once'
grok_program.o: In function `grok_collection_loop':
grok_program.c:(.text+0x6a5): undefined reference to `event_base_dispatch'
grok_input.o: In function `_program_file_buferror':
grok_input.c:(.text+0x6c): undefined reference to `event_once'
grok_input.o: In function `grok_input_eof_handler':
grok_input.c:(.text+0x14b): undefined reference to `bufferevent_disable'
grok_input.c:(.text+0x275): undefined reference to `bufferevent_disable'
grok_input.c:(.text+0x2c6): undefined reference to `event_once'
grok_input.c:(.text+0x316): undefined reference to `event_once'
grok_input.o: In function `_program_file_repair_event':
grok_input.c:(.text+0x6ea): undefined reference to `event_once'
grok_input.o: In function `_program_file_read_buffer':
grok_input.c:(.text+0x97a): undefined reference to `bufferevent_get_input'
grok_input.c:(.text+0x982): undefined reference to `evbuffer_readline'
grok_input.o: In function `_program_process_stdout_read':
grok_input.c:(.text+0x9da): undefined reference to `bufferevent_get_input'
grok_input.c:(.text+0x9e2): undefined reference to `evbuffer_readline'
grok_input.o: In function `grok_program_add_input_process':
grok_input.c:(.text+0xc2e): undefined reference to `bufferevent_new'
grok_input.c:(.text+0xc3e): undefined reference to `bufferevent_enable'
grok_input.c:(.text+0xc9d): undefined reference to `event_once'
grok_input.c:(.text+0xccf): undefined reference to `bufferevent_new'
grok_input.c:(.text+0xcdc): undefined reference to `bufferevent_enable'
grok_input.o: In function `grok_program_add_input_file':
grok_input.c:(.text+0xe11): undefined reference to `bufferevent_new'
grok_input.c:(.text+0xe21): undefined reference to `bufferevent_enable'
grok_input.c:(.text+0xe48): undefined reference to `event_once'
grok_input.o: In function `_program_file_read_real':
grok_input.c:(.text+0x44a): undefined reference to `event_once'
grok_matchconf.o: In function `grok_matchconfig_init':
grok_matchconf.c:(.text+0x15): undefined reference to `tclistnew'
grok_matchconf.o: In function `grok_matchconfig_close':
grok_matchconf.c:(.text+0x13e): undefined reference to `tclistval'
grok_matchconf.c:(.text+0x14e): undefined reference to `tclistnum'
grok_matchconf.c:(.text+0x15a): undefined reference to `tclistdel'
grok_matchconf.o: In function `grok_matchconfig_exec':
grok_matchconf.c:(.text+0x1487): undefined reference to `tclistnum'
grok_matchconf.c:(.text+0x14dc): undefined reference to `tclistval'
grok_discover.o: In function `grok_discover_init':
grok_discover.c:(.text+0x25): undefined reference to `tccmpint32'
grok_discover.c:(.text+0x2c): undefined reference to `tctreenew2'
grok_discover.c:(.text+0x5c): undefined reference to `tclistnum'
grok_discover.c:(.text+0x86): undefined reference to `tclistval'
grok_discover.c:(.text+0x142): undefined reference to `tctreeputkeep'
grok_discover.c:(.text+0x15f): undefined reference to `tclistdel'
grok_discover.o: In function `grok_discover_clean':
grok_discover.c:(.text+0x288): undefined reference to `tctreedel'
grok_discover.o: In function `grok_discover':
grok_discover.c:(.text+0x363): undefined reference to `tctreeiterinit'
grok_discover.c:(.text+0x38c): undefined reference to `tctreeiternext'
grok_discover.c:(.text+0x3ad): undefined reference to `tctreeget'
grok_config.o: In function `conf_match_set_debug':
grok_config.c:(.text+0x564): undefined reference to `tclistnum'
grok_config.c:(.text+0x5a4): undefined reference to `tclistval'
grok_config.c:(.text+0x5de): undefined reference to `tclistover'
grok_config.o: In function `conf_new_match_pattern':
grok_config.c:(.text+0x4f7): undefined reference to `tclistpush'
collect2: ld returned 1 exit status
make: *** [grok] Error 1
rm filters.c grok_matchconf_macro.c

I just installed jls-grok 0.9.0 and when I fetch code from the github repository and manually run tests, I get this:

/Users/victorng/dev/grok/ruby/test/pure-ruby $ jruby alltests.rb

Loading tests: ./general/basic_test.rb
Loading tests: ./general/captures_test.rb
Loading tests: ./patterns/day.rb
Loading tests: ./patterns/host.rb
Loading tests: ./patterns/ip.rb
Loading tests: ./patterns/iso8601.rb
Loading tests: ./patterns/month.rb
Loading tests: ./patterns/number.rb
Loading tests: ./patterns/path.rb
Loading tests: ./patterns/prog.rb
Loading tests: ./patterns/quotedstring.rb
Loading tests: ./patterns/uri.rb
Loaded suite alltests
Started
E..EE..EEEEEFEEEEEEEEEEEEEEEEEEEE
Finished in 0.524 seconds.

1. Error:
   test_days(DayPatternsTest):
   TypeError: can't convert String into Integer
   org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/day.rb:10:insetup'
   org/jruby/RubyArray.java:1603:in each' org/jruby/RubyArray.java:1603:ineach'

2. Error:
   test_grok_expanded_pattern_works_correctly(GrokBasicTests):
   TypeError: can't convert String into Integer
   org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./general/basic_test.rb:45:intest_grok_expanded_pattern_works_correctly'
   org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
   org/jruby/RubyArray.java:1603:in `each'

3. Error:
   test_grok_load_patterns_from_file(GrokBasicTests):
   TypeError: can't convert String into Integer
   org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./general/basic_test.rb:55:intest_grok_load_patterns_from_file'
   org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
   org/jruby/RubyArray.java:1603:in `each'

4. Error:
   test_basic_capture(GrokPatternCapturingTests):
   TypeError: can't convert String into Integer
   org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./general/captures_test.rb:23:intest_basic_capture'
   org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
   org/jruby/RubyArray.java:1603:in `each'

5. Error:
   test_capture_methods(GrokPatternCapturingTests):
   TypeError: can't convert String into Integer
   org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./general/captures_test.rb:12:intest_capture_methods'
   org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
   org/jruby/RubyArray.java:1603:in `each'

6. Error:
   test_multiple_captures(GrokPatternCapturingTests):
   TypeError: can't convert String into Integer
   org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./general/captures_test.rb:64:intest_multiple_captures'
   org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
   org/jruby/RubyArray.java:1603:in `each'

7. Error:
   test_multiple_captures_with_same_name(GrokPatternCapturingTests):
   TypeError: can't convert String into Integer
   org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./general/captures_test.rb:52:intest_multiple_captures_with_same_name'
   org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
   org/jruby/RubyArray.java:1603:in `each'

8. Error:
   test_nested_captures(GrokPatternCapturingTests):
   TypeError: can't convert String into Integer
   org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./general/captures_test.rb:77:intest_nested_captures'
   org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
   org/jruby/RubyArray.java:1603:in `each'

9. Failure:
   test_nesting_recursion(GrokPatternCapturingTests)
   [./general/captures_test.rb:89:in test_nesting_recursion' org/jruby/RubyKernel.java:2092:insend'
   org/jruby/RubyArray.java:1603:in each' org/jruby/RubyArray.java:1603:ineach']:
   exception expected but was
   Class:
   Message: <"can't convert String into Integer">
   ---Backtrace---
   org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
   /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./general/captures_test.rb:90:intest_nesting_recursion'
   ./general/captures_test.rb:89:in test_nesting_recursion' org/jruby/RubyKernel.java:2092:insend'
   org/jruby/RubyArray.java:1603:in `each'

org/jruby/RubyArray.java:1603:in `each'

10. Error:
    test_valid_capture_subnames(GrokPatternCapturingTests):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./general/captures_test.rb:98:intest_valid_capture_subnames'
    org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
    org/jruby/RubyArray.java:1603:in `each'

11. Error:
    test_hosts(HostPattternTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/host.rb:10:insetup'
    org/jruby/RubyArray.java:1603:in each' org/jruby/RubyArray.java:1603:ineach'

12. Error:
    test_ips(IPPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/ip.rb:13:intest_ips'
    org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
    org/jruby/RubyArray.java:1603:in `each'

13. Error:
    test_non_ips(IPPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/ip.rb:23:intest_non_ips'
    org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
    org/jruby/RubyArray.java:1603:in `each'

14. Error:
    test_iso8601(ISO8601PatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/iso8601.rb:10:insetup'
    org/jruby/RubyArray.java:1603:in each' org/jruby/RubyArray.java:1603:ineach'

15. Error:
    test_iso8601_nomatch(ISO8601PatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/iso8601.rb:10:insetup'
    org/jruby/RubyArray.java:1603:in each' org/jruby/RubyArray.java:1603:ineach'

16. Error:
    test_months(MonthPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/month.rb:10:insetup'
    org/jruby/RubyArray.java:1603:in each' org/jruby/RubyArray.java:1603:ineach'

17. Error:
    test_match_base16num(NumberPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/number.rb:58:intest_match_base16num'
    org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
    org/jruby/RubyArray.java:1603:in `each'

18. Error:
    test_match_number(NumberPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/number.rb:13:intest_match_number'
    org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
    org/jruby/RubyArray.java:1603:in `each'

19. Error:
    test_match_number_amid_things(NumberPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/number.rb:37:intest_match_number_amid_things'
    org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
    org/jruby/RubyArray.java:1603:in `each'

20. Error:
    test_match_number_float(NumberPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/number.rb:27:intest_match_number_float'
    org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
    org/jruby/RubyArray.java:1603:in `each'

21. Error:
    test_no_match_number(NumberPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/number.rb:50:intest_no_match_number'
    org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
    org/jruby/RubyArray.java:1603:in `each'

22. Error:
    test_unix_paths(PathPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/path.rb:10:insetup'
    org/jruby/RubyArray.java:1603:in each' org/jruby/RubyArray.java:1603:ineach'

23. Error:
    test_windows_paths(PathPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/path.rb:10:insetup'
    org/jruby/RubyArray.java:1603:in each' org/jruby/RubyArray.java:1603:ineach'

24. Error:
    test_progs(ProgPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/prog.rb:9:insetup'
    org/jruby/RubyArray.java:1603:in each' org/jruby/RubyArray.java:1603:ineach'

25. Error:
    test_escaped_quotes_no_match_quoted_string(QuotedStringPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/quotedstring.rb:38:intest_escaped_quotes_no_match_quoted_string'
    org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
    org/jruby/RubyArray.java:1603:in `each'

26. Error:
    test_non_quoted_strings_no_match(QuotedStringPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/quotedstring.rb:47:intest_non_quoted_strings_no_match'
    org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
    org/jruby/RubyArray.java:1603:in `each'

27. Error:
    test_quoted_string_common(QuotedStringPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/quotedstring.rb:13:intest_quoted_string_common'
    org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
    org/jruby/RubyArray.java:1603:in `each'

28. Error:
    test_quoted_string_inside_escape(QuotedStringPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/quotedstring.rb:27:intest_quoted_string_inside_escape'
    org/jruby/RubyKernel.java:2092:in send' org/jruby/RubyArray.java:1603:ineach'
    org/jruby/RubyArray.java:1603:in `each'

29. Error:
    test_urls(URIPatternsTest):
    TypeError: can't convert String into Integer
    org/jruby/RubyArray.java:1338:in []' org/jruby/RubyMatchData.java:403:in[]'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:78:in compile' org/jruby/RubyKernel.java:1419:inloop'
    /Users/victorng/dev/grok/ruby/lib/grok-pure.rb:70:in compile' ./patterns/uri.rb:9:insetup'
    org/jruby/RubyArray.java:1603:in each' org/jruby/RubyArray.java:1603:ineach'

33 tests, 14 assertions, 1 failures, 28 errors

Testing with jruby 1.9 - I get failures with the pure-ruby tests.

$ make test-pure
JRUBY_OPTS=--1.9 LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:$PWD/../../" RUBYLIB="$PWD/../lib" ruby pure-ruby/alltests.rb
Loading tests: pure-ruby/general/basic_test.rb
/Users/victorng/dev/grok/ruby/test/../lib/grok-pure.rb:22: warning: regexp has invalid interval
./pure-ruby/general/basic_test.rb:2:in require': /Users/victorng/dev/grok/ruby/test/../lib/grok-pure.rb:22: undefined (?...) sequence: /%{ # match '%{' not prefixed with '\' (SyntaxError) (?<name> # match the pattern name (?<pattern>[A-z0-9]+) (?::(?<subname>[A-z0-9_:]+))? ) (?:=(?<definition> (?: (?:[^{}\\]+|\\.+)+ | (?<curly>\{(?:(?>[^{}]+|(?>\\[{}])+)|(\g<curly>))*\})+ )+ ))? [^}]* }/ from ./pure-ruby/general/basic_test.rb:2 from pure-ruby/alltests.rb:7:inload'
from pure-ruby/alltests.rb:7
from pure-ruby/alltests.rb:5:in `each'
from pure-ruby/alltests.rb:5
make: *** [test-pure] Error 1
Victors-MacBook-Air:test victorng$

grok script used:

program {
  load-patterns: "patterns/base"
  load-patterns: "patterns/linux-syslog"
  load-patterns: "patterns/postfix"
#  exec "grep outbound- /var/log/maillog"
#  file "/var/log/maillog.1" {
  file "/var/lib/hadoop-0.20/ajay.m/maillog.1" {
    follow: true
  }

  match {
    pattern: "%{POSTFIXSMTPLOG}"
    #reaction: "%{QUEUEID}: %{logsource} %{to} %{relay} got response: %{reason}"
    reaction: "%{QUEUEID}: %{logsource} %{to} via %{relay}"
  }
}

Input logs which match the pattern look like:

Mar 26 04:02:11 outbound-us2 postfix/smtp[23113]: 3C8C7ED4D11: to=, relay=none, delay=0, delays=0/0/0/0, dsn=5.4.6, status=bounced (mail for domain.com loops back to myself)

postfix patterns used: https://gist.github.com/be1b531485eaca695ab7
(markdown was eating up line 4, hence gisted)

Without follow: true, the grok script returns.

Libraries:
libtokyocabinet-dev/trusty,now 1.4.48-2 amd64 [installed]
libtokyocabinet9/trusty,now 1.4.48-2 amd64 [installed]
libpcre3/trusty-updates,now 1:8.31-2ubuntu2.3 amd64 [installed]

Command:
valgrind --leak-check=full ./grok -f samples/ifconfig.grok

Stdout:

==23776== Memcheck, a memory error detector
==23776== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==23776== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==23776== Command: ./grok -f samples/ifconfig.grok
==23776==
==23776== Source and destination overlap in memcpy(0x6465fd0, 0x6465fd0, 24)
==23776== at 0x4C2F71C: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x54D6953: tctreeput (in /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0)
==23776== by 0x405D7A: grok_capture_add (in /root/grok/grok)
==23776== by 0x404EB7: grok_compilen (in /root/grok/grok)
==23776== by 0x409DE6: grok_matchconfig_init (in /root/grok/grok)
==23776== by 0x40E9D2: conf_new_matchconf (in /root/grok/grok)
==23776== by 0x40CCC6: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== Source and destination overlap in memcpy(0x6465310, 0x6465310, 104)
==23776== at 0x4C2F71C: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x54D6953: tctreeput (in /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0)
==23776== by 0x405C4D: grok_capture_add (in /root/grok/grok)
==23776== by 0x4056D5: grok_compilen (in /root/grok/grok)
==23776== by 0x409DE6: grok_matchconfig_init (in /root/grok/grok)
==23776== by 0x40E9D2: conf_new_matchconf (in /root/grok/grok)
==23776== by 0x40CCC6: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== Source and destination overlap in memcpy(0x6465a00, 0x6465a00, 24)
==23776== at 0x4C2F71C: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x54D6953: tctreeput (in /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0)
==23776== by 0x405CF2: grok_capture_add (in /root/grok/grok)
==23776== by 0x4056D5: grok_compilen (in /root/grok/grok)
==23776== by 0x409DE6: grok_matchconfig_init (in /root/grok/grok)
==23776== by 0x40E9D2: conf_new_matchconf (in /root/grok/grok)
==23776== by 0x40CCC6: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== Source and destination overlap in memcpy(0x6465fd0, 0x6465fd0, 24)
==23776== at 0x4C2F71C: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x54D6953: tctreeput (in /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0)
==23776== by 0x405D7A: grok_capture_add (in /root/grok/grok)
==23776== by 0x4056D5: grok_compilen (in /root/grok/grok)
==23776== by 0x409DE6: grok_matchconfig_init (in /root/grok/grok)
==23776== by 0x40E9D2: conf_new_matchconf (in /root/grok/grok)
==23776== by 0x40CCC6: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
Found: 172.17.0.1
Found: 192.168.0.20
Found: 127.0.0.1
==23776==
==23776== HEAP SUMMARY:
==23776== in use at exit: 33,710 bytes in 73 blocks
==23776== total heap usage: 225 allocs, 152 frees, 76,441 bytes allocated
==23776==
==23776== 105 bytes in 1 blocks are definitely lost in loss record 20 of 53
==23776== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x54D1BB0: tclistpush (in /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0)
==23776== by 0x405CD9: grok_capture_add (in /root/grok/grok)
==23776== by 0x404EB7: grok_compilen (in /root/grok/grok)
==23776== by 0x40EAEA: conf_new_match_pattern (in /root/grok/grok)
==23776== by 0x40CDD5: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== 105 bytes in 1 blocks are definitely lost in loss record 21 of 53
==23776== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x54D1BB0: tclistpush (in /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0)
==23776== by 0x405D61: grok_capture_add (in /root/grok/grok)
==23776== by 0x404EB7: grok_compilen (in /root/grok/grok)
==23776== by 0x40EAEA: conf_new_match_pattern (in /root/grok/grok)
==23776== by 0x40CDD5: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== 108 (104 direct, 4 indirect) bytes in 1 blocks are definitely lost in loss record 22 of 53
==23776== at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x404DF4: grok_compilen (in /root/grok/grok)
==23776== by 0x40EAEA: conf_new_match_pattern (in /root/grok/grok)
==23776== by 0x40CDD5: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== 142 (136 direct, 6 indirect) bytes in 1 blocks are definitely lost in loss record 24 of 53
==23776== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x40EA77: conf_new_match_pattern (in /root/grok/grok)
==23776== by 0x40CDD5: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== 315 bytes in 3 blocks are definitely lost in loss record 34 of 53
==23776== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x54D1BB0: tclistpush (in /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0)
==23776== by 0x405CD9: grok_capture_add (in /root/grok/grok)
==23776== by 0x404EB7: grok_compilen (in /root/grok/grok)
==23776== by 0x409DE6: grok_matchconfig_init (in /root/grok/grok)
==23776== by 0x40E9D2: conf_new_matchconf (in /root/grok/grok)
==23776== by 0x40CCC6: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== 315 bytes in 3 blocks are definitely lost in loss record 35 of 53
==23776== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x54D1BB0: tclistpush (in /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0)
==23776== by 0x405D61: grok_capture_add (in /root/grok/grok)
==23776== by 0x404EB7: grok_compilen (in /root/grok/grok)
==23776== by 0x409DE6: grok_matchconfig_init (in /root/grok/grok)
==23776== by 0x40E9D2: conf_new_matchconf (in /root/grok/grok)
==23776== by 0x40CCC6: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== 321 (312 direct, 9 indirect) bytes in 3 blocks are definitely lost in loss record 36 of 53
==23776== at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x404DF4: grok_compilen (in /root/grok/grok)
==23776== by 0x409DE6: grok_matchconfig_init (in /root/grok/grok)
==23776== by 0x40E9D2: conf_new_matchconf (in /root/grok/grok)
==23776== by 0x40CCC6: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== 1,153 (24 direct, 1,129 indirect) bytes in 1 blocks are definitely lost in loss record 47 of 53
==23776== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x54D190A: tclistnew (in /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0)
==23776== by 0x405DDA: grok_capture_add (in /root/grok/grok)
==23776== by 0x404EB7: grok_compilen (in /root/grok/grok)
==23776== by 0x40EAEA: conf_new_match_pattern (in /root/grok/grok)
==23776== by 0x40CDD5: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== 1,153 (24 direct, 1,129 indirect) bytes in 1 blocks are definitely lost in loss record 48 of 53
==23776== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x54D190A: tclistnew (in /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0)
==23776== by 0x405DCD: grok_capture_add (in /root/grok/grok)
==23776== by 0x404EB7: grok_compilen (in /root/grok/grok)
==23776== by 0x40EAEA: conf_new_match_pattern (in /root/grok/grok)
==23776== by 0x40CDD5: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== 1,377 (24 direct, 1,353 indirect) bytes in 1 blocks are definitely lost in loss record 49 of 53
==23776== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x54D190A: tclistnew (in /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0)
==23776== by 0x405DCD: grok_capture_add (in /root/grok/grok)
==23776== by 0x404EB7: grok_compilen (in /root/grok/grok)
==23776== by 0x409DE6: grok_matchconfig_init (in /root/grok/grok)
==23776== by 0x40E9D2: conf_new_matchconf (in /root/grok/grok)
==23776== by 0x40CCC6: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== 3,459 (72 direct, 3,387 indirect) bytes in 3 blocks are definitely lost in loss record 52 of 53
==23776== at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==23776== by 0x54D190A: tclistnew (in /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0)
==23776== by 0x405DDA: grok_capture_add (in /root/grok/grok)
==23776== by 0x404EB7: grok_compilen (in /root/grok/grok)
==23776== by 0x409DE6: grok_matchconfig_init (in /root/grok/grok)
==23776== by 0x40E9D2: conf_new_matchconf (in /root/grok/grok)
==23776== by 0x40CCC6: yyparse (in /root/grok/grok)
==23776== by 0x4045CE: main (in /root/grok/grok)
==23776==
==23776== LEAK SUMMARY:
==23776== definitely lost: 1,536 bytes in 19 blocks
==23776== indirectly lost: 7,017 bytes in 23 blocks
==23776== possibly lost: 0 bytes in 0 blocks
==23776== still reachable: 25,157 bytes in 31 blocks
==23776== suppressed: 0 bytes in 0 blocks
==23776== Reachable blocks (those to which a pointer was found) are not shown.
==23776== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==23776==
==23776== For counts of detected and suppressed errors, rerun with: -v
==23776== ERROR SUMMARY: 25 errors from 15 contexts (suppressed: 0 from 0)

https://github.com/jordansissel/grok/blob/master/Makefile#L216:

@if $(GPERF) --version | head -1 | egrep -v '3\.[0-9]+\.[0-9]+' ; then \
	echo "We require gperf version >= 3.0.3" ; \
	exit 1; \
fi

This test fails with gperf 3.1:

We require gperf version >= 3.0.3

All the necessary libraries are present:
rpm -qa | egrep -i 'bison|flex|gperf|make|libevent|libpcre|tokyo|cunit'
flex-2.5.35-54.4.x86_64
automake-1.11.1-7.1.noarch
libtokyocabinet-devel-1.4.46-1.1.x86_64
libpcre0-32bit-8.10-4.1.x86_64
make-3.82-140.1.x86_64
gperf-3.0.4-6.1.x86_64
libtokyocabinet9-1.4.46-1.1.x86_64
libpacemaker3-1.1.5-3.2.x86_64
libpcre0-8.10-4.1.x86_64
libevent-1_4-2-1.4.14b-2.1.x86_64
bison-2.4.1-6.1.x86_64
libpcreposix0-8.10-4.1.x86_64
libevent-devel-1.4.14b-2.1.x86_64
pacemaker-1.1.5-3.2.x86_64
libpcrecpp0-8.10-4.1.x86_64

System is OpenSuse 11.4 (x86_64)

Thanks for any hints.

$ make grok
[ -f grok_capture_xdr.h ] && rm grok_capture_xdr.h || true
rpcgen -h grok_capture.x -o grok_capture_xdr.h
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux grok.c -o grok.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux grokre.c -o grokre.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux grok_capture.c -o grok_capture.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux grok_pattern.c -o grok_pattern.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux stringhelper.c -o stringhelper.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux predicates.c -o predicates.o
[ -f grok_capture_xdr.c ] && rm grok_capture_xdr.c || true
rpcgen -c grok_capture.x -o grok_capture_xdr.c
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux grok_capture_xdr.c -o grok_capture_xdr.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux grok_match.c -o grok_match.o
grok_match.c: In function ‘grok_match_get_named_capture’:
grok_match.c:6:3: warning: passing argument 1 of ‘grok_capture_get_by_name’ discards qualifiers from pointer target type
grok_capture.h:13:21: note: expected ‘struct grok_t _’ but argument is of type ‘const struct grok_t * const’
grok_match.c:10:5: warning: passing argument 1 of ‘grok_capture_get_by_subname’ discards qualifiers from pointer target type
grok_capture.h:14:21: note: expected ‘struct grok_t *’ but argument is of type ‘const struct grok_t * const’
grok_match.c: In function ‘grok_match_walk_init’:
grok_match.c:40:18: warning: initialization discards qualifiers from pointer target type
grok_match.c: In function ‘grok_match_walk_next’:
grok_match.c:49:3: warning: passing argument 1 of ‘grok_capture_walk_next’ discards qualifiers from pointer target type
grok_capture.h:20:21: note: expected ‘struct grok_t *’ but argument is of type ‘const struct grok_t * const’
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux grok_logging.c -o grok_logging.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux grok_program.c -o grok_program.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux grok_input.c -o grok_input.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux grok_matchconf.c -o grok_matchconf.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux libc_helper.c -o libc_helper.o
/usr/bin/gperf grok_matchconf_macro.gperf > grok_matchconf_macro.c
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux grok_matchconf_macro.c -o grok_matchconf_macro.o
/usr/bin/gperf filters.gperf > filters.c
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux filters.c -o filters.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux grok_discover.c -o grok_discover.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux conf.tab.c -o conf.tab.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux conf.yy.c -o conf.yy.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux main.c -o main.o
cc -c -pipe -fPIC -I. -O2 -I/usr/local/include -DPLATFORM_GNULinux grok_config.c -o grok_config.o
gcc -ldl -lpcre -levent -rdynamic -ltokyocabinet -L/usr/local/lib -L/lib -levent grok.o grokre.o grok_capture.o grok_pattern.o stringhelper.o predicates.o grok_capture_xdr.o grok_match.o grok_logging.o grok_program.o grok_input.o grok_matchconf.o libc_helper.o grok_matchconf_macro.o filters.o grok_discover.o conf.tab.o conf.yy.o main.o grok_config.o -o grok
grok.o: In function grok_pcre_callout': grok.c:(.text+0x66): undefined reference todlopen'
grok.c:(.text+0x72): undefined reference to dlsym' grok.o: In functiongrok_init':
grok.c:(.text+0x1b5): undefined reference to pcre_callout' grok.c:(.text+0x213): undefined reference totctreenew'
grok.c:(.text+0x21c): undefined reference to tctreenew' grok.c:(.text+0x225): undefined reference totctreenew'
grok.c:(.text+0x22e): undefined reference to tctreenew' grok.c:(.text+0x237): undefined reference totctreenew'
grok.c:(.text+0x273): undefined reference to pcre_compile' grok.c:(.text+0x2a0): undefined reference topcre_fullinfo'
grok.c:(.text+0x2b3): undefined reference to pcre_get_stringnumber' grok.c:(.text+0x2cc): undefined reference topcre_get_stringnumber'
grok.c:(.text+0x2e5): undefined reference to pcre_get_stringnumber' grok.c:(.text+0x2fe): undefined reference topcre_get_stringnumber'
grokre.o: In function grok_free_clone': grokre.c:(.text+0x10): undefined reference topcre_free'
grokre.c:(.text+0x3c): undefined reference to tctreedel' grokre.c:(.text+0x4a): undefined reference totctreedel'
grokre.c:(.text+0x58): undefined reference to tctreedel' grokre.o: In functiongrok_compilen':
grokre.c:(.text+0xe2): undefined reference to tctreeclear' grokre.c:(.text+0xeb): undefined reference totctreeclear'
grokre.c:(.text+0xf4): undefined reference to tctreeclear' grokre.c:(.text+0xfd): undefined reference totctreeclear'
grokre.c:(.text+0x269): undefined reference to pcre_exec' grokre.c:(.text+0x2cc): undefined reference topcre_get_substring'
grokre.c:(.text+0x37e): undefined reference to pcre_get_substring' grokre.c:(.text+0x3a2): undefined reference topcre_get_substring'
grokre.c:(.text+0x608): undefined reference to pcre_free_substring' grokre.c:(.text+0xad5): undefined reference topcre_compile'
grokre.c:(.text+0xaf5): undefined reference to pcre_fullinfo' grokre.c:(.text+0xb2a): undefined reference topcre_fullinfo'
grokre.c:(.text+0xb42): undefined reference to pcre_fullinfo' grokre.c:(.text+0xb5a): undefined reference topcre_fullinfo'
grokre.o: In function grok_execn': grokre.c:(.text+0xe5a): undefined reference topcre_exec'
grokre.o: In function grok_free_clone': grokre.c:(.text+0x67): undefined reference totctreedel'
grokre.o: In function grok_free': grokre.c:(.text+0x94): undefined reference totctreedel'
grok_capture.o: In function grok_capture_add': grok_capture.c:(.text+0x9a): undefined reference totctreeput'
grok_capture.c:(.text+0xb5): undefined reference to tctreeput' grok_capture.c:(.text+0xc7): undefined reference totctreeget'
grok_capture.c:(.text+0xdb): undefined reference to tclistnum' grok_capture.c:(.text+0x103): undefined reference totclistval'
grok_capture.c:(.text+0x119): undefined reference to tclistremove' grok_capture.c:(.text+0x129): undefined reference totclistpush'
grok_capture.c:(.text+0x141): undefined reference to tctreeput' grok_capture.c:(.text+0x154): undefined reference totctreeget'
grok_capture.c:(.text+0x168): undefined reference to tclistnum' grok_capture.c:(.text+0x193): undefined reference totclistval'
grok_capture.c:(.text+0x1a9): undefined reference to tclistremove' grok_capture.c:(.text+0x1b9): undefined reference totclistpush'
grok_capture.c:(.text+0x1d2): undefined reference to tctreeput' grok_capture.c:(.text+0x222): undefined reference totclistnew'
grok_capture.c:(.text+0x231): undefined reference to tclistnew' grok_capture.o: In functiongrok_capture_get_by_id':
grok_capture.c:(.text+0x25c): undefined reference to tctreeget' grok_capture.o: In functiongrok_capture_get_by_name':
grok_capture.c:(.text+0x2a4): undefined reference to tctreeget' grok_capture.c:(.text+0x2b8): undefined reference totclistval'
grok_capture.o: In function grok_capture_get_by_subname': grok_capture.c:(.text+0x304): undefined reference totctreeget'
grok_capture.c:(.text+0x318): undefined reference to tclistval' grok_capture.o: In functiongrok_capture_get_by_capture_number':
grok_capture.c:(.text+0x34c): undefined reference to tctreeget' grok_capture.o: In functiongrok_capture_walk_next':
grok_capture.c:(.text+0x703): undefined reference to tctreeiternext' grok_capture.c:(.text+0x726): undefined reference totctreeget'
grok_capture.o: In function grok_capture_walk_init': grok_capture.c:(.text+0x6e5): undefined reference totctreeiterinit'
grok_pattern.o: In function grok_pattern_name_list': grok_pattern.c:(.text+0xd): undefined reference totclistnew'
grok_pattern.c:(.text+0x1d): undefined reference to tctreeiterinit' grok_pattern.c:(.text+0x33): undefined reference totclistpush'
grok_pattern.c:(.text+0x3e): undefined reference to tctreeiternext' grok_pattern.o: In functiongrok_pattern_add':
grok_pattern.c:(.text+0xd7): undefined reference to tctreeput' grok_pattern.o: In functiongrok_pattern_find':
grok_pattern.c:(.text+0x12c): undefined reference to tctreeget' predicates.o: In functiongrok_predicate_regexp_init':
predicates.c:(.text+0x69): undefined reference to pcre_exec' predicates.c:(.text+0x31f): undefined reference topcre_compile'
grok_program.o: In function grok_collection_init': grok_program.c:(.text+0x34): undefined reference toevent_init'
grok_program.c:(.text+0x69): undefined reference to event_set' grok_program.c:(.text+0x74): undefined reference toevent_add'
grok_program.o: In function grok_collection_check_end_state': grok_program.c:(.text+0x162): undefined reference toevent_base_loopexit'
grok_program.o: In function _collection_sigchld': grok_program.c:(.text+0x3a8): undefined reference toevent_once'
grok_program.o: In function grok_collection_loop': grok_program.c:(.text+0x6e5): undefined reference toevent_base_dispatch'
grok_input.o: In function _program_file_buferror': grok_input.c:(.text+0x6c): undefined reference toevent_once'
grok_input.o: In function grok_input_eof_handler': grok_input.c:(.text+0x1ab): undefined reference tobufferevent_disable'
grok_input.c:(.text+0x2d5): undefined reference to bufferevent_disable' grok_input.c:(.text+0x326): undefined reference toevent_once'
grok_input.c:(.text+0x376): undefined reference to event_once' grok_input.o: In function_program_file_repair_event':
grok_input.c:(.text+0x723): undefined reference to event_once' grok_input.o: In function_program_file_read_buffer':
grok_input.c:(.text+0x95e): undefined reference to evbuffer_readline' grok_input.o: In function_program_process_stdout_read':
grok_input.c:(.text+0x9be): undefined reference to evbuffer_readline' grok_input.o: In functiongrok_program_add_input_process':
grok_input.c:(.text+0xbbe): undefined reference to bufferevent_new' grok_input.c:(.text+0xbce): undefined reference tobufferevent_enable'
grok_input.c:(.text+0xc2d): undefined reference to event_once' grok_input.c:(.text+0xc5f): undefined reference tobufferevent_new'
grok_input.c:(.text+0xc6c): undefined reference to bufferevent_enable' grok_input.o: In functiongrok_program_add_input_file':
grok_input.c:(.text+0xd72): undefined reference to bufferevent_new' grok_input.c:(.text+0xd82): undefined reference tobufferevent_enable'
grok_input.c:(.text+0xda9): undefined reference to event_once' grok_input.o: In functionprogram_file_read_real':
grok_input.c:(.text+0x4aa): undefined reference to event_once' grok_matchconf.o: In functiongrok_matchconfig_init':
grok_matchconf.c:(.text+0x15): undefined reference to tclistnew' grok_matchconf.o: In functiongrok_matchconfig_close':
grok_matchconf.c:(.text+0x13c): undefined reference to tclistval' grok_matchconf.c:(.text+0x14c): undefined reference totclistnum'
grok_matchconf.c:(.text+0x158): undefined reference to tclistdel' grok_matchconf.o: In functiongrok_matchconfig_exec':
grok_matchconf.c:(.text+0x153f): undefined reference to tclistnum' grok_matchconf.c:(.text+0x158c): undefined reference totclistval'
grok_discover.o: In function grok_discover_init': grok_discover.c:(.text+0x27): undefined reference totccmpint32'
grok_discover.c:(.text+0x2e): undefined reference to tctreenew2' grok_discover.c:(.text+0x64): undefined reference totclistnum'
grok_discover.c:(.text+0xa6): undefined reference to tclistval' grok_discover.c:(.text+0x16a): undefined reference totctreeputkeep'
grok_discover.c:(.text+0x187): undefined reference to tclistdel' grok_discover.o: In functiongrok_discover_clean':
grok_discover.c:(.text+0x2b8): undefined reference to tctreedel' grok_discover.o: In functiongrok_discover':
grok_discover.c:(.text+0x3c5): undefined reference to tctreeiterinit' grok_discover.c:(.text+0x3f7): undefined reference totctreeiternext'
grok_discover.c:(.text+0x413): undefined reference to tctreeget' grok_discover.c:(.text+0x532): undefined reference totctreeiternext'
grok_config.o: In function conf_match_set_debug': grok_config.c:(.text+0x4d6): undefined reference totclistnum'
grok_config.c:(.text+0x51a): undefined reference to tclistval' grok_config.c:(.text+0x554): undefined reference totclistover'
grok_config.o: In function conf_new_match_pattern': grok_config.c:(.text+0x467): undefined reference totclistpush'
collect2: ld returned 1 exit status
make: ** [grok] Error 1
rm filters.c grok_matchconf_macro.c

Hello,
I tried to report this issue on the google groups already,
but in the meantime I made some more googling and found out why + solution.
But as I'm not a gcc/make expert, it is best for you to review it than for me to make a pull request.

So as I understood, the issue comes from gcc4.6 being more restrictive on the LDFLAGS position,
I found a similar bug report on another project, applied the same change (moving LDFLAGS to end of line) in your Makefile and it worked for me.
bumptech/stud#62 (commits)

So in the grok and dicogrok directive, I changed
$(CC) $(LDFLAGS) $^ -o $@
to
$(CC) $^ -o $@ $(LDFLAGS)

and in the libgrok.$(LIBSUFFIX):, I changed
$(CC) $(LDFLAGS) -fPIC $(DYNLIBFLAG) $(LIBNAMEFLAG) $^ -o $@
to
$(CC) -fPIC $(DYNLIBFLAG) $(LIBNAMEFLAG) $^ -o $@ $(LDFLAGS)

I found the explanation here:
http://ubuntuforums.org/showthread.php?t=1859400

grok_execn called on an object that has not pattern compiled. Did you call grok_compile yet?

Using OSX with brew-installed pcre.

Initially saw this with pcre 8.37, but it appears to be broken for me back until pcre 8.32

Hi @jordansissel,

I was going through grok to see if it is possible to spew the number of times the pattern was found.
Instead of emitting all the patterns found, is it possible to aggregate the number of times the specific pattern/message was found and emit the number and the message instead of just a filter?

Thank you for the help.

Could we clean up the project organization a bit by moving the source code into a directory (src/)?

Could grok get a README describing its purpose, and maybe some example usage snippets?

It looks like the grok gemspec that is published with the 0.9.4 jls-grok is incorrect.

When I try and install the gem with jruby 1.6.5.1 running in an rbenv environment, I get an error:

Invalid gemspec in [/opt/rubies/jruby-1.6.5.1/lib/ruby/gems/1.8/specifications/jls-grok-0.9.4.gemspec]: Illformed requirement ["#YAML::Yecht::DefaultKey:0x6d3d422d 0.1.7"]

I tried the 0.9.2 gem and it worked ok.

I've tried looking about in the code, but I can't see the gemspec in the master branch, only in the pure-ruby branches, but I'm not sure how you're building the gem.

Let me know if you're wanting any further information.

Gavin

After removing the lines 217-219 from the Makefile (#28), the compilation failes with gperf 3.1:

>>> Compiling source in /var/tmp/portage/dev-libs/grok-0.9.2-r1/work/grok-0.9.2 ...
make -j1 
sh ./version.sh --header > grok_version.h
[ -f grok_capture_xdr.h ] && rm grok_capture_xdr.h || true
rpcgen -h grok_capture.x -o grok_capture_xdr.h
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  grok.c -o grok.o
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  grokre.c -o grokre.o
grokre.c: In function ‘grok_pattern_expand’:
grokre.c:312:12: warning: passing argument 1 of ‘free’ discards ‘const’ qualifier from pointer target type
    free(pattern_regex);
            ^
In file included from /usr/include/pcre.h:90:0,
                from grokre.c:1:
/usr/include/stdlib.h:483:13: note: expected ‘void *’ but argument is of type ‘const char *’
extern void free (void *__ptr) __THROW;
            ^
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  grok_capture.c -o grok_capture.o
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  grok_pattern.c -o grok_pattern.o
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  stringhelper.c -o stringhelper.o
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  predicates.c -o predicates.o
[ -f grok_capture_xdr.c ] && rm grok_capture_xdr.c || true
rpcgen -c grok_capture.x -o grok_capture_xdr.c
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  grok_capture_xdr.c -o grok_capture_xdr.o
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  grok_match.c -o grok_match.o
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  grok_logging.c -o grok_logging.o
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  grok_program.c -o grok_program.o
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  grok_input.c -o grok_input.o
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  grok_matchconf.c -o grok_matchconf.o
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  libc_helper.c -o libc_helper.o
/usr/bin/gperf grok_matchconf_macro.gperf > grok_matchconf_macro.c
x86_64-pc-linux-gnu-gcc -c -O2 -pipe -march=core-avx2 -mssse3 -msse4 -fPIC -I. -DPLATFORM_GNULinux  grok_matchconf_macro.c -o grok_matchconf_macro.o
grok_matchconf_macro.gperf:69:1: error: conflicting types for ‘patname2macro’
In file included from grok_matchconf_macro.gperf:4:0:
grok_matchconf_macro.h:22:24: note: previous declaration of ‘patname2macro’ was here
const struct strmacro *patname2macro(const char *str, unsigned int len);
                        ^
Makefile:238: recipe for target 'grok_matchconf_macro.o' failed
make: *** [grok_matchconf_macro.o] Error 1
rm grok_matchconf_macro.c

Works fine with gperf 3.0.4.

I am not sure there's a point reporting bugs, since this project seems to be dormant at best. I just summarize.

• debug masks sometimes get lost, for example grok_config.c never logs for me, which is bad for filename errors (or, actually, config errors)
• plenty of segfaults due to non-checked return values from extrenal libs, mainly libpcre
  ** for eample grok_match_walk_next accepts position 1852178432 happily for a 80 characters long string - and also crashes happily immediately after.
• libpcre seems to limit regexp length, none of my (test) grok files was below the limit. :-)
• non-follow file read doesn't work; I haven't checked how it was designed, but definitely not as expected.
• Makefile clean does not clean libgrok* and discogrok binaries

Hi,

Do not match after a specific line in my test case. Is it a known bug?

My test procedure is,
[mitsu-ko@webg203 grok]$ ./grok -f samples/tomcat1.grok
Jun 29, 2008 11:16:20 AM org.apache.catalina.core.ApplicationContext log
Jun 29, 2008 11:16:20 AM org.apache.catalina.core.ApplicationContext log
Jun 29, 2008 11:22:43 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:23:02 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:23:30 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:33:32 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:33:54 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:34:24 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:45:40 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:46:10 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:46:30 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:46:39 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:47:25 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:47:51 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:48:40 AM org.apache.catalina.core.StandardWrapperValve invoke

Then I grep the 'Jun' keyword, then run grok the new file.
[mitsu-ko@webg203 grok]$ grep Jun tomcat1.log >tomcat2.log
[mitsu-ko@webg203 grok]$ ./grok -f samples/tomcat2.grok
Jun 29, 2008 11:16:20 AM org.apache.catalina.core.ApplicationContext log
Jun 29, 2008 11:16:20 AM org.apache.catalina.core.ApplicationContext log
Jun 29, 2008 11:22:43 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:23:02 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:23:30 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:33:32 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:33:54 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:34:24 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:45:40 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:46:10 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:46:30 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:46:39 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:47:25 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:47:51 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:48:40 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:48:54 AM org.apache.catalina.core.ApplicationContext log
Jun 29, 2008 11:48:54 AM org.apache.catalina.core.ApplicationContext log
Jun 29, 2008 11:49:06 AM org.apache.catalina.core.ApplicationContext log
Jun 29, 2008 11:49:06 AM org.apache.catalina.core.ApplicationContext log
Jun 29, 2008 11:49:12 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:49:13 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:49:16 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:49:17 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:49:46 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:53:10 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:56:14 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:57:37 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:57:48 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 11:58:13 AM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 12:03:51 PM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 12:08:19 PM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 12:10:02 PM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 12:10:13 PM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 12:13:27 PM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 12:14:07 PM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 12:23:42 PM org.apache.catalina.core.ApplicationContext log
Jun 29, 2008 12:23:42 PM org.apache.catalina.core.ApplicationContext log
Jun 29, 2008 12:30:09 PM org.apache.catalina.core.ApplicationContext log
Jun 29, 2008 12:30:09 PM org.apache.catalina.core.ApplicationContext log
Jun 29, 2008 12:36:48 PM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 12:36:51 PM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 12:36:54 PM org.apache.catalina.core.StandardWrapperValve invoke
Jun 29, 2008 1:26:34 PM org.apache.catalina.core.ApplicationContext log
Jun 29, 2008 1:26:34 PM org.apache.catalina.core.ApplicationContext log

All of the log lines are outputted.

My environment is ,

OS: CentOS7.6.1810(64bit) (latest)
other RPM: installed by yum

Best regards,

grok_file.zip

Any reason that the Makefile only installs grok/patterns/base? There are also java, ruby, and nagios files in the distro.

Any reason to not do this?

diff --git a/Makefile b/Makefile
index adfe869..f8cf5f2 100644
--- a/Makefile
+++ b/Makefile
@@ -103,6 +103,9 @@ install: libgrok.$(LIBSUFFIX) grok discogrok $(GROKHEADER)
        install -d $(DESTDIR)$(PREFIX)/share/grok
        install -d $(DESTDIR)$(PREFIX)/share/grok/patterns
        install patterns/base $(DESTDIR)$(PREFIX)/share/grok/patterns/
+       install patterns/java $(DESTDIR)$(PREFIX)/share/grok/patterns/
+       install patterns/nagios $(DESTDIR)$(PREFIX)/share/grok/patterns/
+       install patterns/ruby $(DESTDIR)$(PREFIX)/share/grok/patterns/

 uninstall:
        rm -f $(DESTDIR)$(PREFIX)/bin/grok

looks like it is almost 10 years since the last release.

Also the version scheme is kind of weird (v1.0 -> v0.9)