robcowart / synesis_lite_syslog Goto Github PK
View Code? Open in Web Editor NEWSyslog collection with the Elastic Stack.
License: Other
Syslog collection with the Elastic Stack.
License: Other
hello
im facing a little problem now in syslog solution
the parser cant parse the cisco syslog format
(gns3 is creating the syslog... router 7200)
any help would be appreciated
thanks...
Hello.
We use elastiflow 4 with elasticsearch 7.8.1
Will you plan to update synesis_lite_syslog to work with 7.8.1 version .
It allows to use one elasticsearch cluster to collect netflow and syslog data
It seems that ES username/password defined in the systemd file don't take effect and logstash still tries to use the default "changeme". The workaround is to specify the ES password in the output conf file.
Hi Rob, followed the instructions but the 514 port listener fails to start for some reason -
[2019-05-15T12:05:05,250][INFO ][logstash.inputs.tcp ] Starting tcp input listener {:address=>"0.0.0.0:514", :ssl_enable=>"false"} [2019-05-15T12:05:05,256][ERROR][logstash.javapipeline ] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:synesis_lite_syslog Plugin: <LogStash::Inputs::Tcp host=>"0.0.0.0", dns_reverse_lookup_enabled=>false, id=>"dbb44d7e7b498a36341cfa76571b5c366d668ed2b496086bcc69ec34add2adb2", type=>"syslog", port=>514, enable_metric=>true, codec=><LogStash::Codecs::Line id=>"line_ed18c78c-5642-4b13-9fc6-67f140d700c5", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, mode=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>, tcp_keep_alive=>false> Error: Permission denied Exception: Java::JavaNet::SocketException Stack: sun.nio.ch.Net.bind0(Native Method) sun.nio.ch.Net.bind(sun/nio/ch/Net.java:461) sun.nio.ch.Net.bind(sun/nio/ch/Net.java:453) sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:227) io.netty.channel.socket.nio.NioServerSocketChannel.doBind(io/netty/channel/socket/nio/NioServerSocketChannel.java:128) io.netty.channel.AbstractChannel$AbstractUnsafe.bind(io/netty/channel/AbstractChannel.java:558) io.netty.channel.DefaultChannelPipeline$HeadContext.bind(io/netty/channel/DefaultChannelPipeline.java:1283) io.netty.channel.AbstractChannelHandlerContext.invokeBind(io/netty/channel/AbstractChannelHandlerContext.java:501) io.netty.channel.AbstractChannelHandlerContext.bind(io/netty/channel/AbstractChannelHandlerContext.java:486) io.netty.channel.DefaultChannelPipeline.bind(io/netty/channel/DefaultChannelPipeline.java:989) io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:254) io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:364) io.netty.util.concurrent.AbstractEventExecutor.safeExecute(io/netty/util/concurrent/AbstractEventExecutor.java:163) io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(io/netty/util/concurrent/SingleThreadEventExecutor.java:403) io.netty.channel.nio.NioEventLoop.run(io/netty/channel/nio/NioEventLoop.java:463) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(io/netty/util/concurrent/SingleThreadEventExecutor.java:858) io.netty.util.concurrent.FastThreadLocalRunnable.run(io/netty/util/concurrent/FastThreadLocalRunnable.java:30) java.lang.Thread.run(java/lang/Thread.java:834)
Hello,
Since event.message
and log.message
are pretty much a duplicate of the logs, I decided to drop event.message
since we have a cleaner log.message
- this allows to save space as some logs are quite long.
But when I search using "query strings" in Kibana, it doesn't search log.message
at all. It does search event.message
when it's there as well as other fields such as log.process
.
I don't know why Kibana refuses to search log.message
when using "query strings" (just typing a word or sentence with double-quotes in the KQL box), can you help?
Thanks!
Hi Rob,
I've installed this on a docker container, this is the current docker-compose.yaml:
version: '2'
services:
elasticsearch:
image: elasticsearch-img:6.3.2
container_name: elasticsearch-container
volumes:
- /data/elasticsearch-1/:/usr/share/elasticsearch/data
ports:
- 9200:9200 #Elasticsearch HTTP
- 9300:9300 #Elasticsearch TCP transport
network_mode: bridge
restart: always
environment:
# - cluster.name=docker-cluster
# - bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms4g -Xmx4g"
ulimits:
memlock:
soft: -1
hard: -1
logstash:
image: logstash-img:6.3.2
container_name: logstash-container
ports:
- 5000:5000 #logstash TCP input
- 514:5140 #listent to syslog on 514(host) map it to 5140(container) 541 is reserved and needs root
- 514:5140/udp #listent to syslog on 514(host) map it to 5140(container) 541 is reserved and needs root
restart: always #restarts on reboot
environment:
- "LS_JAVA_OPTS=-Xms8g -Xmx8g"
- "SYNLITE_SYSLOG_TEMPLATE_PATH=/usr/share/logstash/syslog/templates"
- "SYNLITE_SYSLOG_GROK_PATTERNS_DIR=/usr/share/logstash/syslog/patterns"
- "SYNLITE_SYSLOG_RESOLVE_IP2HOST=true"
- "SYNLITE_SYSLOG_NAMESERVER=8.8.8.8"
- "SYNLITE_SYSLOG_ES_HOSTS=elasticsearch:9200"
# - "SYNLITE_SYSLOG_ES_USER=elastic"
# - "SYNLITE_SYSLOG_ES_PASSWORD=changeme"
- "SYNLITE_SYSLOG_TCP_HOST=0.0.0.0"
- "SYNLITE_SYSLOG_TCP_PORT=514"
- "SYNLITE_SYSLOG_UDP_HOST=0.0.0.0"
- "SYNLITE_SYSLOG_UDP_PORT=514"
- "SYNLITE_SYSLOG_MSG_TIMESTAMP=true"
- "SYNLITE_SYSLOG_TZ=UTC"
network_mode: bridge
links:
- elasticsearch
depends_on:
- elasticsearch
Initially, port mapping of 514:514
made docker crib stating that permission was denied. I'm guessing this is because it's a port < 1000 and hence is previlaged. I've mapped 514:5140
within the container.
My /etc/rsyslog.conf looks like below:
...
...
# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")
# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
...
...
I'm able to see syslog being written on to /var/log/syslog
. It works when I do something like: logger -s " This is a test "
However, I do not see anyting being picked up by LS/ES. What am I missing?
Thanks
@robcowart Hi. Thanks for the work put in here. Please consider adding docker support.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.