twitter / finatra

Fast, testable, Scala services built on TwitterServer and Finagle

Finatra

Status

This project is used in production at Twitter (and many other organizations), and is being actively developed and maintained.

Finatra is a lightweight framework for building fast, testable, scala applications on top of TwitterServer and Finagle. Finatra provides an easy-to-use API for creating and testing Finagle servers and apps as well as powerful JSON support, modern logging via SLF4J, Finagle client utilities, and more.

Getting involved

• Website: https://twitter.github.io/finatra/
• Latest news: Blog
• Github Source: https://github.com/twitter/finatra/
• Gitter: https://gitter.im/twitter/finatra
• Mailing List: [email protected]

Features

• Production use @Twitter.
• ~50 times faster than v1.6 in several benchmarks.
• Powerful Feature and Integration test support.
• Optional JSR-330 Dependency Injection using Google Guice.
• Jackson-based JSON parsing supporting required fields, default values, and validations.
• Logback MDC integration with com.twitter.util.Local for contextual logging across futures.

Documentation

To get started, see the Getting Started section of our User Guide to get up and running. Or check out the specific sections for building HTTP or Thrift servers.

Examples

An HTTP controller and server:

import com.twitter.finatra.http._

@Singleton
class ExampleController extends Controller {
  get("/") { request: Request =>
    "<h1>Hello, world!</h1>"
  }
}

import com.twitter.finatra.http._

class ExampleServer extends HttpServer {
  override def configureHttp(router: HttpRouter): Unit = {
    router
      .filter[CommonFilters]
      .add[ExampleController]
  }
}

A Thrift controller and server:

import com.twitter.finatra.thrift._
import com.twitter.scrooge.{Request, Response}

@Singleton
class ExampleThriftController
  extends Controller(MyThriftService) {

  handle(MyFunction).withFn { request: Request[MyFunction.Args] =>
    ...
  }
}

import com.twitter.finatra.thrift._

class ExampleServer extends ThriftServer {
  override def configureThrift(router: ThriftRouter): Unit = {
    router
      .add[ExampleThriftController]
  }
}

Example Projects

Finatra includes working examples which highlight various features of the framework and include tests. These examples are included in the root sbt build and are thus buildable as part of the entire project.

Please take a look through the examples for more detailed information on features, testing, building, and running.

Latest version

The release branch in Github tracks the latest stable release, which is currently:

available on Maven Central. See the First Steps section in the User Guide for how to add dependencies.

Releases are done on an approximately monthly schedule. While semver is not followed, the changelogs are detailed and include sections on public API breaks and changes in runtime behavior.

Development version

The develop branch in Github tracks the latest code which is updated every week. If you want to contribute a patch or fix, please use this branch as the basis of your Pull Request.

We feel that a welcoming community is important and we ask that you follow Twitter's Open Source Code of Conduct in all interactions with the community. For more information on providing contributions, please see our CONTRIBUTING.md documentation.

Presentations

Check out our list of presentations: Finatra Presentations.

Authors

• Steve Cosenza https://github.com/scosenza
• Christopher Coco https://github.com/cacoco

A full list of contributors can be found on GitHub.

Follow @finatra on Twitter for updates.

License

Copyright 2013 Twitter, Inc.

Licensed under the Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0

finatra's Issues

verify heroku uploads works

apache-like directory browser for files

seems useful

0 deprecation/warnings

Startup / Shutdown hooks

E.g. initialize / shutdown a redis connection pool. I didn't see a place to put these. Many require a call for the threadpool to be shutdown for the app to exit cleanly.

I suppose one could put stuff on a singleton object, that the controller could access, and have the thing that starts up FinatraServer manage starting up and shutting down the stuff that lives there?

support ETAGS and/or Cache-Control headers in file server

libthrift outdated

I just saw that the project are using libthrift 0.5.0, and the last version is 0.9.1.

Version 0.5.0 isn't available in commom mvn repos, only at twitter mvn repo.

There should not be an issue at all, but, I'm under a corporate proxy with all sort of funny things that a MS-based network can provide, and we also have a nexus mirror. This nexus mirrors seems to not been able to mirror the twitter repo.

Is there any special reason for finatra not use the lastest version of libthrift?

Thanks!

an config

move to twitter-server once published

Sending redirect require a body

Doing this:

render.status(HttpResponseStatus.MOVED_PERMANENTLY.getCode).header("Location", redirectUrl).toFuture

does not work because of the missing body. Responds with a 500 instead.

allow insertion of userland filters into the finagle stack

implement a routes.txt in admin

Would be nice to have an admin endpoint to return all registered routes

update gem

update docs

LogLevel doesn't seem to work

Hi!

I've tried to set the logging level to DEBUG and WARNING or ERROR by

1. using the flag: -com.twitter.finatra.config.logLevel='ERROR' or
2. setting System.setProperty("com.twitter.finatra.config.logLevel", "DEBUG") before registering the controllers

However, I always seem to have info messages coming through and never debug. If I call log.getLevel, I always get null.

Maybe I'm doing something wrong and I'm stuck. BTW, the documentation about the flags is wrong: it's not -usage but -help seems to work :-)

Unable to retrieve post parameters

I'm unable to retrieve post parameters to handle the form submit.

Here is a simple app that reproduces the problem

object Main {

  class ExampleApp extends Controller {

    post("/param") {
      request =>
        request.params.get("test") match {
          case Some(value) => render.plain(value.toString).status(200).toFuture
          case None => render.plain("No param").status(404).toFuture
        }
    }

  }


  def main(args: Array[String]) {
    val app = new ExampleApp

    FinatraServer.register(app)
    FinatraServer.start()

  }

}

When I execute

curl -iH 'Content-Type: application/x-www-form-urlencoded' -XPOST http://localhost:7070/param -d 'test=myparam'

I get no param response

HTTP/1.1 404 Not Found
Content-Type: text/plain
Content-Length: 8

No param

add logging

RequestAdapter does not support multiple values for query params

If you want to send an array as query parameters to a GET request, jQuery encodes it as follows

var a = {key: [1,2,3]}
$.param(a) // key[]=1&key[]=2&key[]=3

For nested hashes:

var b = {key1: {field1:1, field2: 2}, key2: {field1: 2, field2: 89}}
$.param(b) // key1[field1]=1&key1[field2]=2&key2[field1]=2&key2[field2]=89

There can obviously be some pretty complicated nesting of these.

Currently Finatra only pulls the head off a set of query parameters that all have the same key (first example would give only one of them). It would be nice to return the set of all values, otherwise it requires a bit of application logic/coordination to figure out which keys are actually present.

https://github.com/capotej/finatra/blob/master/src/main/scala/com/twitter/finatra/RequestAdapter.scala#L39

do a perf review

benchmark suite with caliper

micro benchmark all the things!

Filter invoked 4 times per single request?

Using the ugly patch described previously, I've added a single filter to the application. When tracing the filter, I see that it gets invoked 4 times per http request. I've verified that I'm indeed only making 1 http request and not 4. Any idea what's going on here?

Here's a fragment of a stack trace that I captured that shows the caller of the filter:

com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:69)
com.twitter.finagle.Service$$anon$1.apply(Service.scala:14)
com.twitter.finatra.FileService.apply(FileService.scala:121)
com.twitter.finatra.FileService.apply(FileService.scala:85)
com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:69)
com.twitter.finagle.Service$$anon$1.apply(Service.scala:14)
com.twitter.finatra.LoggingFilter.apply(LoggingFilter.scala:11)
com.twitter.finatra.LoggingFilter.apply(LoggingFilter.scala:8)
com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:69)
com.twitter.finagle.Service$$anon$1.apply(Service.scala:14)
com.twitter.finagle.Service$$anon$1.apply(Service.scala:11)
com.twitter.finatra.FinatraServer$$anonfun$2.apply(FinatraServer.scala:59)
com.twitter.finatra.FinatraServer$$anonfun$2.apply(FinatraServer.scala:58)
com.twitter.finagle.Filter$$anon$6.apply(Filter.scala:117)
com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:69)
com.twitter.finagle.ServiceProxy.apply(Service.scala:103)
com.twitter.finagle.Service$$anon$1.apply(Service.scala:14)
com.twitter.finagle.Filter$$anon$4.apply(Filter.scala:111)
com.twitter.finagle.Filter$$anon$5$$anon$1.apply(Filter.scala:53)
com.twitter.finagle.service.StatsFilter.apply(StatsFilter.scala:34)
com.twitter.finagle.Filter$$anon$5.apply(Filter.scala:52)
com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:69)
com.twitter.finagle.Service$$anon$1.apply(Service.scala:14)
com.twitter.finagle.tracing.ServerDestTracingProxy$$anon$1.apply(DestinationTracing.scala:26)
com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:69)
com.twitter.finagle.Service$$anon$1.apply(Service.scala:14)
com.twitter.finagle.filter.MkJvmFilter$$anon$1.apply(JvmFilter.scala:29)
com.twitter.finagle.Filter$$anon$5$$anon$1.apply(Filter.scala:53)
com.twitter.finagle.tracing.TracingFilter$$anonfun$apply$1.apply(TracingFilter.scala:18)
com.twitter.finagle.tracing.TracingFilter$$anonfun$apply$1.apply(TracingFilter.scala:16)
com.twitter.finagle.tracing.Trace$.unwind(Trace.scala:183)
com.twitter.finagle.tracing.TracingFilter.apply(TracingFilter.scala:16)
com.twitter.finagle.Filter$$anon$5$$anon$1.apply(Filter.scala:53)
com.twitter.finagle.filter.MonitorFilter$$anonfun$apply$1.apply(MonitorFilter.scala:16)
com.twitter.finagle.filter.MonitorFilter$$anonfun$apply$1.apply(MonitorFilter.scala:16)
com.twitter.util.Future$$anonfun$monitored$1.apply$mcV$sp(Future.scala:93)
com.twitter.util.Monitor$$anonfun$apply$1.apply$mcV$sp(Monitor.scala:38)
com.twitter.util.Monitor$$anonfun$apply$1.apply(Monitor.scala:38)
com.twitter.util.Monitor$$anonfun$apply$1.apply(Monitor.scala:38)
com.twitter.util.Monitor$$anonfun$using$1.apply(Monitor.scala:103)
com.twitter.util.Monitor$.restoring(Monitor.scala:110)
com.twitter.util.Monitor$.using(Monitor.scala:101)
com.twitter.util.Monitor$class.apply(Monitor.scala:37)
com.twitter.util.Monitor$$anon$1.apply(Monitor.scala:141)
com.twitter.util.Future$.monitored(Future.scala:92)
com.twitter.finagle.filter.MonitorFilter.apply(MonitorFilter.scala:15)
com.twitter.finagle.Filter$$anon$5$$anon$1.apply(Filter.scala:53)
com.twitter.finagle.filter.HandletimeFilter.apply(HandletimeFilter.scala:15)
com.twitter.finagle.Filter$$anon$5.apply(Filter.scala:52)
com.twitter.finagle.Filter$$anon$5.apply(Filter.scala:52)
com.twitter.finagle.Filter$$anon$5.apply(Filter.scala:52)
com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:69)

handle param routing for static file handling

localhost:7070/mypage.html works fine
localhost:7070/mypage.html?name=value == 404

asset pipeline filter

A filter that compiles coffeescript/sass/compass on the fly

Add public/ dir in src/main/resources as new docroot

Right now, anything in src/main/resources will be served from the FileService, this may not be desirable, move to a public/ dir.

bubble up view/mustache errors

enable full admin endpoints besides metrics.json

Only admin/metrics.json works, enable the others.

Filters not working

Hey, I'm using the latest version of finatra, but it seem that filters are not working as described here:

http://finatra.info/docs/index.html#filters

I built a single file example with a filter that throws an exception to isolate the problem:

package com.fintest

import com.twitter.finatra._
import com.twitter.finagle._
import com.twitter.app.App
import com.twitter.finagle.http.{Request => FinagleRequest}
import com.twitter.finagle.http.{Response => FinagleResponse}

class Example extends Controller {
  get("/") { request =>
     render.plain("hi").toFuture
  }
}

class MyServer extends FinatraServer {
    val controller = new Example()
    addFilter(new TestFilter)
    register(controller)
}

class TestFilter extends SimpleFilter[FinagleRequest, FinagleResponse] with App {

  def apply(
    request: FinagleRequest,
    service: Service[FinagleRequest, FinagleResponse]) = {
    service(request) map { response =>
        throw new Exception("!") // this is never executed
        response
    }
  }
}

object Runner {
    def main(args: Array[String]): Unit = {
        val server = new MyServer()
        server.start()
    }  
}

The exception is never thrown. Any idea why this is the case?

make redirects RFC compliant

According to the spec, redirects SHOULD include a note in the body with the hyperlink of the new location.

mvn package doesnt fully package

./finatra generator doesnt work on linux

Seems to be an incompatibility between bsd and gnu version of sed

filters for select routes only

Is it currently possible to apply filters just to select routes? For instance if I don't want a filter to run on every static file that's served. I'm sure I can figure out a way to hack this together in my own app logic but am wondering if there is anything that I could use out of the box.

thanks!

support a "rails style" render.action to render arbitrary actions from any other action without a redirect

move to com.twitter.common.metrics instead of ostrich.stats

port back apache's multiupload handler

netty's postdecoder isnt all that great, put back the multiupload one

regexed routes

flight/bower and bootstrap built in

It'd be nice if the public directory contained a version of flight/bower and bootstrap already in there.

Serve static files

There should be a possibility to serve static resources

• Images
• CSS files
• JS files
• etc.

PUT and PATCH command param issue

Looks like this only works for POST params: mairbek@7956428

Extend this to work for all verbs.

Custom error handlers

There should be a possibility to define custom error handler for specified exception.

Right now it is possible to define custom filter as a workaround but this could be simplified.

maven => sbt

sbt is the de facto build tool in scala-land, finatra should probably use it.

PID management

support in release scripts for dual publishing scala 2.9 and 2.10

./finatra update-readme no longer works

probably due to the m4 changes

Make mustache factory use baseTemplatePath local_docroot and template_path

I have a couple of mustache files stored in resources/mustache. I have set -Dlocal_docroot=resources and -Dtemplate_path=mustache/ when running my finatra server.

Whenever I load a partial (which is in the resources/mustache directory) i need to prefix the partial name with mustache/ in the file that loads the partial. I'd expect to be able to do

{{> my_partial.mustache}}

instead of having to do

{{> mustache/my_partial.mustache}}

Is this expected behavior, or is there a bug on my end? Should this be changed? If so I can come up with a patch.

request.routeParams should be decoded

When requesting /hello%3Aworld?bar=hello%3Aworld

get("/:foo") { request =>
    request.routeParams.get("foo").get should equal("hello:world") // it's actually `hello%3Aworld`
    request.params.get("bar").get should equal("hello:world")
}

Workaround:

import org.jboss.netty.handler.codec.http.QueryStringDecoder
import org.jboss.netty.util.CharsetUtil

QueryStringDecoder.decodeComponent(param, CharsetUtil.UTF_8)

1.4.1

using websockets

Hi..is there any tutorial or explanation of how use websockets with finatra?...is possible use atmosphere websocket with finatra similar to scalatra or many others frameworks??...thanks!!

Publish to Maven Central

Publish to the Sonatype OSS system:
https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide

Once this is done, you can sync to Maven Central easily.

Sonatype issue: https://issues.sonatype.org/browse/OSSRH-3988

unbreak file upload/form support

Test/Harden logging

We seem to always mess up logging, let's get some tests around this to ensure high quality logging.

Test Cases:

• Log levels are respected
• Ability to turn off logging

unbreak cookie support