wildart / thrift.jl

Thrift for Julia

Thrift.jl

Apache Thrift is a lightweight, language-independent software stack with an associated code generation mechanism for RPC.

Thrift.jl is an implementation of Thrift for Julia, including a plugin for the Thrift IDL compiler.

Getting Started

Setting up Thrift for Julia

• Install the Julia Thrift package: Pkg.add("Thrift")
• Patch the Thrift IDL compiler with the Julia code generator plugin. Any of the following methods can be followed:
  • Place Julia plugin t_jl_generator.cc into compiler/cpp/src/generate folder of Thrift source code. Update makefiles to include the new source and rebuild.
  • Clone Thrift sources from here.
• Build and install the Thrift compiler. Some instructions here.

Generating "Hello Julia"

A sample Hello Julia IDL and implementation is bundled along with the Thrift.jl package. It can be found under test/hello folder of the package.

It contains a Thrift IDL named hello.thrift, which contains a service SayHello with a hello method that returns a hello message for the supplied name in a randomly chosen language.

1. Generate Julia sources from the IDL.

   Run command: thrift --gen jl hello.thrift. This should result in a gen-jl folder with sources generated from the IDL placed in a folder hello (named after the IDL file name).

2. Examine the generated files. Below is a brief explanation of the contents of the generated files.

   • hello_types.jl: contains Julia types for Thrift structs, exceptions and enums declared explicitly in the IDL along with other implicit types generated by the code generator.
   • hello_constants.jl: contains any constants declared in the IDL
   • SayHello.jl: code generated for the SayHello service.
   • hello.jl: contains a module named hello (named after the IDL file name), that includes the above mentioned generated files. It also includes a file named hello_impl.jl that is not generated, but must be created by the user.

Implementing "Hello Julia"

• An implementation of the service methods are already provided as hello_impl.jl in the test/hello folder. It has an implementation of hello service method, that appends a randomly chosen greeting from the constant array GREETINGS to the supplied name.

• Place the hello_impl.jl file in the gen-jl/hello folder.

• The client and server implementations for this are already provided as clnt.jl and srvr.jl. Start the server with julia srvr.jl. Run the client with the command julia clnt.jl.

Implementation Details

Following is the status of protocols, transports and servers supported in the current implementation:

Protocols

Implementation	Implemented as	Notes
Binary	TBinaryProtocol
Compact	TCompactProtocol
JSON		Not implemented yet

Transports

Implementation	Implemented as	Notes
Socket	TSocket and TServerSocket
Framed	TFramedTransport
SSL Socket		Not implemented yet
HTTP		Not implemented yet
Buffered		Not implemented yet

Servers

Implementation	Implemented as	Notes
Blocking. Single Task.	TSimpleServer	Single process, blocking
Non Blocking Tasks.	TTaskServer	Single process. Asynchronous task spawned for each connection.
Non Blocking Multi Process.	TProcessPoolServer	Multi process, non blocking.

Setting and Getting Fields

Types used as Thrift structures are regular Julia types and the Julia syntax to set and get fields can be used on them. But with fields that are set as optional, it is quite likely that some of them may not have been present in the instance that was read. Similarly, fields that need to be sent need to be explicitly marked as being set. The following methods are exported to assist doing this:

• get_field(obj::Any, fld::Symbol) : Gets obj.fld if it has been set. Throws an error otherwise.
• set_field(obj::Any, fld::Symbol, val) : Sets obj.fld = val and marks the field as being set. The value would be written on the wire when obj is serialized. Fields can also be set the regular way, but then they must be marked as being set using the fillset method.
• has_field(obj::Any, fld::Symbol) : Checks whether field fld has been set in obj.
• clear(obj::Any, fld::Symbol) : Marks field fld of obj as unset.
• clear(obj::Any) : Marks all fields of obj as unset.

Other Methods

• copy!(to::Any, from::Any) : shallow copy of objects
• isfilled(obj::Any, fld::Symbol) : same as has_field
• isfilled(obj::Any) : same as isinitialized
• fillset(obj::Any, fld::Symbol) : mark field fld of object obj as set
• fillunset(obj::Any) : mark all fields of this object as not set
• fillunset(obj::Any, fld::Symbol) : mark field fld of object obj as not set

On the Generated Code Structure

The generated code largely follows the scheme used in other languages, e.g. Python and C++. Each Thrift program (IDL file) is placed into a separate folder.

Code is also generated to bundle the generated methods into a module which can optionally be used. The reason for making it optional is to not enforce any particular module structure on the user. The example in test/calculator illustrates how to include multiple thrift generated services in a single Julia module, without using the autogenerated modules.

The generated service Processor now assumes that the implemented methods are present in the current module. Thus the generated code is not a complete module and requires the user to supply a service implementation to be complete. An alternative would be to make the generated code a complete module, and have the user supply an implementation module.

Service extensions are supported. The thrift processor on the server side passes on any methods it can not handle to the processor it extends from. Extensions of service clients are supported through Julia type extension.

The code generator can be tweaked in the future towards any preferred way of usage that may appear with further usage.

Thrift Metadata

Thrift serialization can be customized for a type by defining a meta method on it. The meta method provides an instance of ThriftMeta that allows specification of optional fields, field numbers, and default values for fields for a type. The Thrift code generator generates appropriate meta methods wherever required. The below information will however help in understanding and tweaking the generated code if required.

Defining a specialized meta is done simply as below:

import Thrift.meta

meta(t::Type{MyType}) = meta(t,                          # the type which this is for
        Symbol[:intval],                                 # optional fields
        Int[8, 10],                                      # field numbers
        Dict{Symbol,Any}({:strval => "default value"}))  # default values

Without any specialized meta method:

• All fields are marked as required.
• Field numbers are assigned serially starting from -1, and decremented in the order of their declaration.
• No default values are assigned.

When the default behavior is fine, just passing empty values would do. E.g., if just field numbers need to be specified, the following would do:

meta(t::Type{MyType}) = meta(t, [], [8,10], Dict())