Support logical types when converting schemas, especially with Hive support. about kafka-connect-hdfs

[xianzhen]

How about adding "logical type" attribute in connect.schema?

1. Hive schema depends on "type" and "logical type";
2. Column type stored in target file system only depends on "type".

The code translating between connect.schema and avro.schema is hard to be understood.
Adding "DATE"/"TIMESTAMP" type in connect.schema can be it more complex.

[Ishiihara]

@xianzhen The logical type information is stored in the name of the schema. The HDFS connector can potentially discover the logical type from that can covert it to a proper Hive data type.

[yangfeiran]

Hello,
From my point of view, this issue is: when I have timestamp type in ORACLE, after data is extracted into kafka, it is still timestamp type in schema registry, but when I use HDFS sink to put data into the HIVE, the type goes wrong, it became BIGINT.

If so, I am facing this problem too.

[bharatviswa504]

Hi,
Any updates on this issue, from which version it will be available?

[bharatviswa504]

Hi @ewencp

I have updated the code
static {

LOGICAL_TO_TYPEINFO = new HashMap<>();
LOGICAL_TO_TYPEINFO.put(Date.LOGICAL_NAME, TypeInfoFactory.dateTypeInfo);
LOGICAL_TO_TYPEINFO.put(Timestamp.LOGICAL_NAME, TypeInfoFactory.timestampTypeInfo);
LOGICAL_TO_TYPEINFO.put(Decimal.LOGICAL_NAME, TypeInfoFactory.decimalTypeInfo);
}

if (schema.name() != null) {
switch (schema.name()) {
case Date.LOGICAL_NAME:
return LOGICAL_TO_TYPEINFO.get(schema.name());
case Decimal.LOGICAL_NAME:
return LOGICAL_TO_TYPEINFO.get(schema.name());
case Timestamp.LOGICAL_NAME:
return LOGICAL_TO_TYPEINFO.get(schema.name());
}
}

Updated HiveSchemaConverter to add above code when creating Avro Hive table. So, this handle schema conversion. Data conversion where to be handled and how it needs to be handled.
But When table is getting created still i am seeing table has int column. So, further looked in to Avro Hive Conversion. Also, Avro 1.7.7 does not have logical date data type from 1.8 logical date type is available.
https://cwiki.apache.org/confluence/display/Hive/AvroSerDe
But there is no date data type in that. So, could you please explain more and help me on this to fix this issue. Thanks in advance.

[pdambrauskas]

Any updates on this issue?
We're also facing this problem with kafka-connect-hdfs 4.0.0.

[Chintanvpatel]

+1
Any update on this ? Right now we are facing the issue for DATE, DATETIME and DECIMAL.