sksamuel / centurion

Kotlin Bigdata Toolkit

Centurion

Introduction

Centurion is a JVM (written in Kotlin) toolkit for columnar and streaming formats.

This library allows you to read, write and convert between the following formats:

• Apache Parquet
• Apache Orc
• Apache Arrow IPC
• Apache Avro

See changelog for release notes.

Schema Conversions

Centurion allows easy conversion of schemas between any of the supported formats, via Centurion's own internal format.

This internal format is a superset of the functionality of all the supported formats, and is intended as an intermediate format only to allow for conversions.

The following table shows how types map between each of the formats.

Centurion Type	Avro	Parquet	Orc	Arrow
Strings	String	Binary (String)	String	Utf8
UUID	String (UUID)	Binary (String)	String	Utf8
Booleans	Boolean	Boolean	Boolean	Bool
Int64	Long	Int64	Long	Int64 Signed
Int32	Int	Int32	Int	Int32 Signed
Int16	N/A (Int)	Int32 (Signed Int16)	Short	Int16 Signed
Int8	N/A (Int)	Int32 (Signed Int8)	Byte	Int8 Signed
Float64	Double	Double	Double	FloatingPointDouble
Float32	Float	Float	Float	FloatingPointSingle
Enum	Enum	Enum	String	String
Decimal	Binary / Fixed with annotation Decimal	Decimal(precision, scale)	Decimal)	Decimal
Varchar	Fixed)	N/A (String)	Varchar	N/A (String)
TimestampMillis	Long (TimestampMillis)	Int64 (Timestamp)	Timestamp	Timestamp (Millis)
TimestampMicros	Long (TimestampMicros)	Int64 (Timestamp)	Unsupported	Timestamp (Micros)
Map	Map	Map	Map	Map

centurion's Issues

Resequencer will fail in the following test

  it should "reorder out of order messages" in {
    val reseq = TestActorRef(new Resequencer(probe.ref))
    reseq ! msg3
    reseq ! msg2
    reseq ! msg1

    probe.expectMsg(1 second, msg1)
    probe.expectMsg(1 second, msg2)
    probe.expectMsg(1 second, msg3)
  }

It would never send the message 3

Gradle repository config to find the jar

Hi,

I am using gradle as my build tool and trying to fetch this jar from the dependency you mentioned in the readme. The gradle, however, fails to find this using the jitpack plugin with the following lines in my build.gradle.

repositories {
        // ...
        maven { url "https://jitpack.io" }
    }

dependencies {
        compile 'com.sksamuel.akka:akka-patterns_2.11:0.12.1-SNAPSHOT'
    }

Kindly help.

GroupingActor would need to clean up the buffer before stop

Would add this part of code to the GroupingActor OK?

  override def postStop():Unit = {
      if (received.size>0) {
        target ! received.toList
      }   
  }