Comments (10)
Can you clarify this, Peter? I think I'm missing it :) Thanks,
from remote_syslog.
Hey Troy, sorry for the late reply.
We have the need to store / query all of our logs from our various network services in a central place, just like papertrail. We've been evaluating different products in the market and here are some of my thoughts.. hope it's helpful.
First, most application developers will send their logs to an app-specific log file that is local to that application, in a structure decided by the app developer. It makes sense to use syslog as a conduit for sending all of the logs to papertrail, but the downside is it makes it difficult to send structured data that is easily queryable once its all together. But either way, syslog is workable, just not very elegant once you want to start querying the aggregated data.
The app developer could use a syslog adapter in their program (like we have done by using Yell and a syslog adapter), but in the case they don't want to do that and keep with their usual app log file, then enter remote_syslog (as I understand). Remote_syslog will proxy that file to syslog, or even directly to papertrail.
I think remote_syslog, or something like it, could be the best way to send those application logs to a central source as it would be easier to structure the data, without having to worry about fitting into the basic format of syslog as well; offering more freedom. But also, it's less to think about, just setup the config for remote_syslog to the app files, specify some mapping and away you go.
The end goal for us once all of our logs are centralized is to be able to query the data, group the data, and set alerts on certain events that happen in the system. The more structured, the easier that is to do.
Okay, back to my original comment about Go :) .. I've been coding Ruby for the last 10 years and I love it as a language, and we even have a bunch of it in our company. But, as I've been writing more Go code lately, I see the benefits of a low-level language that is super efficient (on memory / cpu), compiles super fast, and even produces cross-platform binaries. I feel an agent that proxies data to a central server would be a good fit for Go, using less system resources and even more friendly to deploy then having to install a specific version of Ruby with some specific gem dependencies. You'd have to make sure the right version of Ruby is on the system, in this case some MRI that is friendly with Eventmachine. EM is becoming an out-dated technology, IMO, and I'm not sure even how/if it works with MRI 2.1. Does it even do proper file async i/o or does it fake it somehow? It's been a few years since I've worked with it. The devops person on our team also mentioned he wasn't able to get the current version of remote_syslog to work for us, so we opted to setting up a syslog adapter right in our application... but I'd prefer to not even have to think about that. Next I'm going to see how the querying interface works.
We produce a lot of logs :) and for something that is just proxying plain-text files, I like to keep the cpu/memory as low as possible. Just trying to help offer suggestions to make this important utility be even better.
from remote_syslog.
Yeah, we're 95% complete rewriting it in Go (already), I just was trying to figure out what "Recommended rewrite in Go" with no detail meant. It sounds like you were suggesting rewriting it in Go, which I obviously agree with, but couldn't tell from the subject alone :)
from remote_syslog.
Great to hear. Iād love to hear about that when its ready.
On Feb 6, 2014, at 3:49 PM, Troy Davis [email protected] wrote:
Yeah, we're 95% complete rewriting it in Go (already), I just was trying to figure out what "Recommended rewrite in Go" with no detail meant. It sounds like you were suggesting rewriting it in Go, which I obviously agree with, but couldn't tell from the subject alone :)
ā
Reply to this email directly or view it on GitHub.
from remote_syslog.
Can do. It was neat to see it come together.
On Thu, Feb 6, 2014 at 12:50 PM, Peter Kieltyka [email protected]:
Great to hear. I'd love to hear about that when its ready.
On Feb 6, 2014, at 3:49 PM, Troy Davis [email protected] wrote:
Yeah, we're 95% complete rewriting it in Go (already), I just was trying
to figure out what "Recommended rewrite in Go" with no detail meant. It
sounds like you were suggesting rewriting it in Go, which I obviously agree
with, but couldn't tell from the subject alone :)Reply to this email directly or view it on GitHub.
Reply to this email directly or view it on GitHubhttps://github.com//issues/77#issuecomment-34369191
.
from remote_syslog.
How is the Go client comin?
from remote_syslog.
Still progressing =) We work a lot like holman/ama#122 (comment), which is to say, most release estimates are worthless. It's pretty far along, so the next update will be a downloadable release.
from remote_syslog.
thats cute :) I come from a similar school of thought.. where deadlines are a poison to quality.. however, there is also FTPPTF... as well the Zuckerberg, done is better then perfect. Or the pragmatic programmer one, that there is no such thing as perfect software.
Just sayin.. :) good luck.. no rush on my end, we're setup with the syslogd.. was more curious
from remote_syslog.
btw this is a pretty cool project that might be useful for papertrail to support -- http://docs.fluentd.org/articles/architecture described as "It's like syslogd, but uses JSON for log messages"
from remote_syslog.
Just an update that remote_syslog2 has had a few production releases now and we're recommending it for all new deployments (as covered in the older remote_syslog's README).
from remote_syslog.
Related Issues (20)
- Catch exceptions from YAML parser
- Compression option HOT 3
- is there a way to filter out some strings? HOT 2
- Specify Different Facility Per Logfile HOT 3
- remote_syslog 1.6.14 crashes on: attempted to remove invalid watch signature HOT 25
- max_message_size = 1024 for UDP - why? HOT 1
- remote_syslog 1.6.14 keeps deleted logfiles open HOT 6
- remote_syslog 1.6.14 keeps old logfiles open HOT 4
- PID file is not created when passing "no detach" option (-D)
- Constant 3-5% CPU usage per process HOT 7
- Rails Logfile, newlines are converted to <13> HOT 7
- Log all error events by default
- Can't seem to use relative path to ssl_server_cert HOT 5
- Anyone keen to rewrite this in Golang? HOT 10
- Can't process logrotate properly when use the copytruncate option on the logrotate HOT 1
- init script gives error on start HOT 9
- Recent commits to servolux break this gem HOT 4
- RemoteSyslogLogger HOT 8
- The 'exclude_patterns' in config sample is implicit and lack of instruction HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from remote_syslog.