Cannot project Schema when adding new column about kafka-connect-hdfs

[shikhar]

@talnicolas can you post a stacktrace?

We represent optional Avro fields like that (union with null & null as defval) as optional fields in Connect. If I were to make an educated guess, this seems like the bug in SchemaProjector.

diff --git a/connect/api/src/main/java/org/apache/kafka/connect/data/SchemaProjector.java b/connect/api/src/main/java/org/apache/kafka/connect/data/SchemaProjector.java
index ad0caf8..2cfd384 100644
--- a/connect/api/src/main/java/org/apache/kafka/connect/data/SchemaProjector.java
+++ b/connect/api/src/main/java/org/apache/kafka/connect/data/SchemaProjector.java
@@ -111,7 +111,7 @@ public class SchemaProjector {
                 } catch (SchemaProjectorException e) {
                     throw new SchemaProjectorException("Error projecting " + sourceField.name(), e);
                 }
-            } else {
+            } else if (!targetField.schema().isOptional()) {
                 Object targetDefault;
                 if (targetField.schema().defaultValue() != null) {
                     targetDefault = targetField.schema().defaultValue();

UPDATE: created https://issues.apache.org/jira/browse/KAFKA-4173 for this

[talnicolas]

@shikhar Yes this is exactly the faulty line, we were wondering if it was the intended behaviour or if it was really a bug. Thanks for creating the JIRA ticket, I will follow that.

The stacktrace just to confirm:

ERROR Task some-sink-connector-1 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerSinkTask) java.lang.RuntimeException: org.apache.kafka.connect.errors.SchemaProjectorException: Cannot project Schema{com.some.class:STRUCT} to Schema{com.some.class:STRUCT} at io.confluent.connect.hdfs.TopicPartitionWriter.write(TopicPartitionWriter.java:292) at io.confluent.connect.hdfs.DataWriter.write(DataWriter.java:234) at io.confluent.connect.hdfs.HdfsSinkTask.put(HdfsSinkTask.java:91) at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:370) at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:227) at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:170) at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:142) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.kafka.connect.errors.SchemaProjectorException: Cannot project Schema{com.some.class:STRUCT} to Schema{com.some.class:STRUCT} at org.apache.kafka.connect.data.SchemaProjector.projectStruct(SchemaProjector.java:119) at org.apache.kafka.connect.data.SchemaProjector.projectRequiredSchema(SchemaProjector.java:92) at org.apache.kafka.connect.data.SchemaProjector.project(SchemaProjector.java:72) at io.confluent.connect.hdfs.schema.SchemaUtils.project(SchemaUtils.java:66) at io.confluent.connect.hdfs.TopicPartitionWriter.write(TopicPartitionWriter.java:263) ... 13 more

[shikhar]

Great, thanks, closing this as it's a framework issue