alvinj / scalaapacheaccesslogparser

An Apache access log parser written in Scala

A Scala Apache Access Log Parser

This project can be used to parse Apache access log records in JVM applications (Scala, Java, etc.) It is specifically written to work with "combined records", as that's the only access log format I've used since the 1990s.

Discussion

In short, I needed an Apache access log parser, and after looking at some other code, I decided to write my own.

Usage

The API is in flux, but right now the usage starts like this:

val rawRecord = """89.166.165.223 - - [21/Jul/2009:02:48:12 -0700] "GET /foo HTTP/1.1" 404 970 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.11) Firefox/3.0.11""""

val parser = AccessLogParser
val accessLogRecord = parser.parse(rawRecord)    // an AccessLogRecord instance

The AccessLogRecord class definition looks like this:

case class AccessLogRecord (
    clientIpAddress: String,         // should be an ip address, but may also be the hostname if hostname-lookups are enabled
    rfc1413ClientIdentity: String,   // typically '-'
    remoteUser: String,              // typically '-'
    dateTime: String,                // [day/month/year:hour:minute:second zone]
    request: String,                 // 'GET /foo ...'
    httpStatusCode: String,          // 200, 404, etc.
    bytesSent: String,               // may be '-'
    referer: String,                 // where the visitor came from
    userAgent: String                // long string to represent the browser and OS
)

In the test code you'll see that I use the parser like this:

val parser = new AccessLogParser
val rec = parser.parseRecord(rawRecord)
it("the result should not be None") {
    assert(rec != None)
}
it("the individual fields should be right") {
    rec.foreach { r =>
        assert(r.clientIpAddress == "66.249.70.10")
        assert(r.rfc1413ClientIdentity == "-")
        assert(r.remoteUser == "-")
        assert(r.dateTime == "[23/Feb/2014:03:21:59 -0700]")
        assert(r.request == "GET /blog/post/java/how-load-multiple-spring-context-files-standalone/ HTTP/1.0")
        assert(r.httpStatusCode == "301")
        assert(r.bytesSent == "-")
        assert(r.referer == "-")
        assert(r.userAgent == "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")
    }
}

If you don't like using the Option/Some/None pattern, I added a method named parseRecordReturningNullObjectOnFailure that returns a "Null Object" version of an AccessLogRecord instead of an Option.

I also added some methods to parse the date and request fields, and I'll document those here on another day. You can see all of the current, up-to-date API by looking at the tests in the AccessLogRecordSpec class.

Building

This project is a typical Scala/SBT project, so just use commands like this:

sbt compile
sbt test
sbt package

More information

I've added more documentation about this library at the following URLs. First, the basic documentation on this library is at this URL:

• My Scala Apache access log parser library

Next, I've written two articles on how to use this library to analyze Apache access log records with Apache Spark and Scala:

• Analyzing Apache access logs with Spark and Scala
• Generating a list of URLs from Apache access log files, sorted by hit count, using Apache Spark (and Scala)

For more information about yours truly:

• See my website
• Find me here on Twitter

All the best,
Alvin Alexander
http://alvinalexander.com

scalaapacheaccesslogparser's Issues

Not able to parse

I just added the jar and try to run sample u have tried, got parse is not a member, kindly help

scala> import com.alvinalexander.accesslogparser._
import com.alvinalexander.accesslogparser._

scala> val parser = AccessLogParser
parser: com.alvinalexander.accesslogparser.AccessLogParser.type = com.alvinalexander.accesslogparser.AccessLogParser$@230c374c

scala> val accessLogRecord = parser.parse(rawRecord)
:26: error: value parse is not a member of object com.alvinalexander.accesslogparser.AccessLogParser
val accessLogRecord = parser.parse(rawRecord)

Unfortunately MyApp has stopped. How can I solve this?

Unfortunately MyApp has stopped. How can I solve this?

Cant Compile with SBT

Izham@osx ~/s/ScalaApacheAccessLogParser> sbt compile
[info] Set current project to ScalaApacheAccessLogParser (in build file:/Users/Izham/scala/ScalaApacheAccessLogParser/)
[info] Updating {file:/Users/Izham/scala/ScalaApacheAccessLogParser/}scalaapacheaccesslogparser...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 2 Scala sources to /Users/Izham/scala/ScalaApacheAccessLogParser/target/scala-2.10/classes...
[info] 'compiler-interface' not yet compiled for Scala 2.10.0. Compiling...
error: error while loading CharSequence, class file '/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home/jre/lib/rt.jar(java/lang/CharSequence.class)' is broken
(class java.lang.RuntimeException/bad constant pool tag 18 at byte 10)
error: error while loading Comparator, class file '/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home/jre/lib/rt.jar(java/util/Comparator.class)' is broken
(class java.lang.RuntimeException/bad constant pool tag 18 at byte 20)
error: error while loading AnnotatedElement, class file '/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home/jre/lib/rt.jar(java/lang/reflect/AnnotatedElement.class)' is broken
(class java.lang.RuntimeException/bad constant pool tag 18 at byte 76)
error: error while loading Arrays, class file '/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home/jre/lib/rt.jar(java/util/Arrays.class)' is broken
(class java.lang.RuntimeException/bad constant pool tag 18 at byte 765)
/var/folders/62/7m3r5f0954v4v_vljpx29_ym0000gn/T/sbt_54774fba/xsbt/ExtractAPI.scala:479: error: java.util.Comparator does not take type parameters
private[this] val sortClasses = new Comparator[Symbol] {
^
5 errors found
error Error compiling sbt component 'compiler-interface'
[error] Total time: 7 s, completed 16-May-2016 19:35:26