dleung / gotail

Unix tail-like library written in Go

Gotail

Blog: (http://capykoa.com/articles/11)

Background

I needed a simple utility to follow rapidly growing adserver logrotating logs that I can process dynamically and in real time.

Features

• Default behavior supports logrotating files.
• Timeout option to wait on new files that hasn't been created yet.
• Supports high reads. On my 2013 Macbook Pro, I was able benchmark at more than 150,000 lines/second (go test). However, expected performance on production should be much higher.
• Lightweight and low memory footprint
• Partly inspired by ActiveState/tail

Usage

package main

import (
	"flag"
	"fmt"
	"log"

	"github.com/dleung/gotail"
)

var fname string

func main() {
	flag.StringVar(&fname, "file", "", "File to tail")
	flag.Parse()

	// Set timeout to 0 to automatically fail if file isn't found
	tail, err := gotail.NewTail(fname, gotail.Config{Timeout: 10})
	if err != nil {
		log.Fatalln(err)
	}

	// lines on the tail.Lines channel for new lines.
	for line := range tail.Lines {
		fmt.Println(line)
	}
}

gotail's Issues

License?

It'd be easier for others to use this if it included a license. Jeff Atwood has a nice writeup on licensing: http://blog.codinghorror.com/pick-a-license-any-license/.

Why not use fsnotify to detect writes instead of polling?

I was hoping to use this library for parsing some logs, but in Go 1.5 it eats a full 2 CPU cores smashing its head against bufio.Reader.ReadString when no data is even available. Not too suitable when it is not the lone (nor primary) application on the server.

I only realized this as I went to deploy and saw the load averages on the machines explode, going into strace and seeing millions of syscall.Read.

fsnotify provides a Write event - why not let the system wake you when there is more data to read?

Edit: I have rewritten with fsnotify, and CPU is totally idle, using epoll on Linux, compared to fully loaded with gotail.