Mirroring not working about mirus

[pdavidson100]

I recommend you look closely at your worker.properties file, as the configuration problem could be there. It's also a good idea to look at your log file, particularly the Kafka consumer and producer properties dumps which show exactly what Mirus and Kafka Connect are attempting to connect to.

[saritago]

@pdavidson100 Below is the worker properties file. Note that mirroring works fine when i do a mirroring within the cluster ie when i pass "destination.bootstrap.servers": "localhost:9092" and "consumer.bootstrap.servers": "localhost:9092" in connector. But when i change to "destination.bootstrap.servers": "localhost:9093" and try to consume from the broker running with 9093 port, it does not work. Should i be adding any additional parameter in connector for destination cluster as well?

##
# Mirus Quickstart worker properties file
#
# This file can be passed to `bin/mirus-worker-start.sh` to quickly get a basic Mirus worker up and
# running.
#
# ---
# Kafka broker bootstrap server - this is Source cluster
bootstrap.servers=localhost:9092
group.id=mirus-quickstart
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
header.converter=org.apache.kafka.connect.converters.ByteArrayConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
config.storage.topic=mirus-config
status.storage.topic=mirus-status
offset.storage.topic=connect-offsets

# In a production environment these values should be increased
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1

[pdavidson100]

Yes, you need to configure the producers created by the Kafka Connect workers in the usual way. This config lives under producer.* in the worker properties file. Try adding this:

producer.bootstrap.servers=localhost:9093

[saritago]

@pdavidson100 Thanks, the above configuration worked for 1 on 1 mirroring. However when i have 2 source clusters and make below configuration entries mirroring happens only from 1 source and not the other source.

# Kafka broker bootstrap server - this is Source cluster
bootstrap.servers=localhost:9092
consumer.bootstrap.servers=localhost:9094,localhost:9092
producer.bootstrap.servers=localhost:9093
group.id=mirus-quickstart
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
header.converter=org.apache.kafka.connect.converters.ByteArrayConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
config.storage.topic=mirus-config
status.storage.topic=mirus-status
offset.storage.topic=connect-offsets

# In a production environment these values should be increased
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1

connector config

curl localhost:8083/connectors/mirus-quickstart-source/config \
      -X PUT \
      -H 'Content-Type: application/json' \
      -d '{
           "name": "mirus-quickstart-source",
           "connector.class": "com.salesforce.mirus.MirusSourceConnector",
           "tasks.max": "5",
           "topics.whitelist": "test-mir",
           "destination.topic.name.suffix": "",
           "destination.bootstrap.servers": "localhost:9093",
           "consumer.bootstrap.servers": "localhost:9092,localhost:9094",
           "consumer.client.id": "mirus-quickstart",
           "consumer.key.deserializer": "org.apache.kafka.common.serialization.ByteArrayDeserializer",
           "consumer.value.deserializer": "org.apache.kafka.common.serialization.ByteArrayDeserializer"
       }'

Also when i add additional servers in producer.bootstrap.servers and destination.bootstrap.servers mirroring happens only to one node and not both destination nodes.

Is this the right way to configure multi cluster mirroring?

[pdavidson100]

For multiple source clusters you just need to configure multiple connector config instances and give each one a unique name (e.g. PUT different config to /connectors/quickstart1/config and /connectors/quickstart2/config) . One connector config should have consumer.bootstrap.servers=localhost:9092, the other should have consumer.bootstrap.servers=localhost:9094. No need to configure multiple bootstrap servers in your Worker properties file.

Kafka Connect Source Connectors do no directly support multiple destination Kafka clusters within the same Kafka Connect cluster. This is because the producer bootstrap is a Worker-level property (i.e., all workers must produce to the same destination cluster). The work-around is to configure multiple independent Kafka Connect sub-clusters, each with different admin topic names and worker configurations. However, this is not usually necessary because the recommended approach is to co-locate your Kafka Connect cluster with your destination Kafka cluster. That means there should always be one independent Kafka Connect cluster for each destination Kafka cluster anyway.