flowenol / avro-fastserde

Fast Apache Avro serialization/deserialization library

avro-fastserde

avro-fastserde is an alternative approach to Apache Avro serialization and deserialization. It generates dedicated code responsible for handling serialization and deserialization, which achieves better performance results than native implementation. Learn more here.

Version

Current version is 1.0.5

Requirements

You need Java 8 to use this library.

Installation

Releases are distributed on Maven central:

<dependency>
    <groupId>com.rtbhouse</groupId>
    <artifactId>avro-fastserde</artifactId>
    <version>1.0.5</version>
</dependency>

Usage

Just use avro-fastserde DatumReader and DatumWriter interface implementation:

import com.rtbhouse.utils.avro.FastGenericDatumReader;
import com.rtbhouse.utils.avro.FastGenericDatumWriter;
import com.rtbhouse.utils.avro.FastSpecificDatumReader;
import com.rtbhouse.utils.avro.FastSpecificDatumWriter;

...

FastGenericDatumReader<GenericData.Record> fastGenericDatumReader = new FastGenericDatumReader<>(writerSchema, readerSchema);
fastGenericDatumReader.read(null, binaryDecoder);

FastGenericDatumWriter<GenericData.Record> fastGenericDatumWriter = new FastGenericDatumWriter<>(schema);
fastGenericDatumWriter.read(data, binaryEncoder);

FastSpecificDatumReader<T> fastSpecificDatumReader = new FastSpecificDatumReader<>(writerSchema, readerSchema);
fastSpecificDatumReader.read(null, binaryDecoder);

FastSpecificDatumWriter<T> fastSpecificDatumWriter = new FastSpecificDatumWriter<>(schema);
fastSpecificDatumWriter.write(data, binaryEncoder);

You can alter class generation behaviour via system properties:

 // Set compilation classpath
 System.setProperty(FastSerdeCache.CLASSPATH, compileClasspath);
 
 // Set generated classes directory
 System.setProperty(FastSerdeCache.GENERATED_CLASSES_DIR, generatedClassesDir);

Or FastSerdeCache class:

import com.rtbhouse.utils.avro.FastGenericDatumReader;
import com.rtbhouse.utils.avro.FastSerdeCache;

...

FastSerdeCache cache = new FastSerdeCache(compileClassPath);
FastGenericDatumReader<GenericData.Record> fastGenericDatumReader = new FastGenericDatumReader<>(writerSchema, readerSchema, cache);

Limitations

• no support for reuse parameter in DatumReader interface.
• no support for SchemaConstructable marker interface for specific Avro records.
• FastSpecificDatumReader will not read data into GenericRecord if the specific classes are not available but will result in compilation failure and fall back to default SpecificDatumReader implementation.