Recommended rewrite in Go about remote_syslog

[troy]

Can you clarify this, Peter? I think I'm missing it :) Thanks,

[pkieltyka]

Hey Troy, sorry for the late reply.

We have the need to store / query all of our logs from our various network services in a central place, just like papertrail. We've been evaluating different products in the market and here are some of my thoughts.. hope it's helpful.

First, most application developers will send their logs to an app-specific log file that is local to that application, in a structure decided by the app developer. It makes sense to use syslog as a conduit for sending all of the logs to papertrail, but the downside is it makes it difficult to send structured data that is easily queryable once its all together. But either way, syslog is workable, just not very elegant once you want to start querying the aggregated data.

The app developer could use a syslog adapter in their program (like we have done by using Yell and a syslog adapter), but in the case they don't want to do that and keep with their usual app log file, then enter remote_syslog (as I understand). Remote_syslog will proxy that file to syslog, or even directly to papertrail.

I think remote_syslog, or something like it, could be the best way to send those application logs to a central source as it would be easier to structure the data, without having to worry about fitting into the basic format of syslog as well; offering more freedom. But also, it's less to think about, just setup the config for remote_syslog to the app files, specify some mapping and away you go.

The end goal for us once all of our logs are centralized is to be able to query the data, group the data, and set alerts on certain events that happen in the system. The more structured, the easier that is to do.

Okay, back to my original comment about Go :) .. I've been coding Ruby for the last 10 years and I love it as a language, and we even have a bunch of it in our company. But, as I've been writing more Go code lately, I see the benefits of a low-level language that is super efficient (on memory / cpu), compiles super fast, and even produces cross-platform binaries. I feel an agent that proxies data to a central server would be a good fit for Go, using less system resources and even more friendly to deploy then having to install a specific version of Ruby with some specific gem dependencies. You'd have to make sure the right version of Ruby is on the system, in this case some MRI that is friendly with Eventmachine. EM is becoming an out-dated technology, IMO, and I'm not sure even how/if it works with MRI 2.1. Does it even do proper file async i/o or does it fake it somehow? It's been a few years since I've worked with it. The devops person on our team also mentioned he wasn't able to get the current version of remote_syslog to work for us, so we opted to setting up a syslog adapter right in our application... but I'd prefer to not even have to think about that. Next I'm going to see how the querying interface works.

We produce a lot of logs :) and for something that is just proxying plain-text files, I like to keep the cpu/memory as low as possible. Just trying to help offer suggestions to make this important utility be even better.

[troy]

Yeah, we're 95% complete rewriting it in Go (already), I just was trying to figure out what "Recommended rewrite in Go" with no detail meant. It sounds like you were suggesting rewriting it in Go, which I obviously agree with, but couldn't tell from the subject alone :)

[pkieltyka]

Great to hear. I’d love to hear about that when its ready.

On Feb 6, 2014, at 3:49 PM, Troy Davis [email protected] wrote:

Yeah, we're 95% complete rewriting it in Go (already), I just was trying to figure out what "Recommended rewrite in Go" with no detail meant. It sounds like you were suggesting rewriting it in Go, which I obviously agree with, but couldn't tell from the subject alone :)

—
Reply to this email directly or view it on GitHub.

[troy]

Can do. It was neat to see it come together.

On Thu, Feb 6, 2014 at 12:50 PM, Peter Kieltyka [email protected]:

Great to hear. I'd love to hear about that when its ready.

On Feb 6, 2014, at 3:49 PM, Troy Davis [email protected] wrote:

Yeah, we're 95% complete rewriting it in Go (already), I just was trying
to figure out what "Recommended rewrite in Go" with no detail meant. It
sounds like you were suggesting rewriting it in Go, which I obviously agree
with, but couldn't tell from the subject alone :)

Reply to this email directly or view it on GitHub.

Reply to this email directly or view it on GitHubhttps://github.com//issues/77#issuecomment-34369191
.

[pkieltyka]

How is the Go client comin?

[troy]

Still progressing =) We work a lot like holman/ama#122 (comment), which is to say, most release estimates are worthless. It's pretty far along, so the next update will be a downloadable release.

[pkieltyka]

thats cute :) I come from a similar school of thought.. where deadlines are a poison to quality.. however, there is also FTPPTF... as well the Zuckerberg, done is better then perfect. Or the pragmatic programmer one, that there is no such thing as perfect software.

Just sayin.. :) good luck.. no rush on my end, we're setup with the syslogd.. was more curious

[pkieltyka]

btw this is a pretty cool project that might be useful for papertrail to support -- http://docs.fluentd.org/articles/architecture described as "It's like syslogd, but uses JSON for log messages"

[troy]

Just an update that remote_syslog2 has had a few production releases now and we're recommending it for all new deployments (as covered in the older remote_syslog's README).