Comments (7)
Hi @marcaguzman , can you post your configs related to this issue to better help you? (the ones that worked and the ones that didn't)
from mirus.
Wow! Thanks for such a fast reply @dalassi1. I used the default config initially with a few benign modifications to ensure I knew which kafka instance Mirus was connecting to because at one point I had Mirus running on both VMs (both in loopback), but now I have only one instance running. Mostly default loopback quickstart worker properties file:
`bootstrap.servers=localhost:9092
group.id=mirus-quickstart
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
header.converter=org.apache.kafka.connect.converters.ByteArrayConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
config.storage.topic=mg2-config
status.storage.topic=mg2-status
offset.storage.topic=mg2-offsets
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1`
Loopback connection config is the default:
> curl localhost:8083/connectors/mirus-quickstart-source/config \ -X PUT \ -H 'Content-Type: application/json' \ -d '{ "name": "mirus-quickstart-source", "connector.class": "com.salesforce.mirus.MirusSourceConnector", "tasks.max": "5", "topics.whitelist": "test", "destination.topic.name.suffix": ".mirror", "destination.bootstrap.servers": "localhost:9092", "consumer.bootstrap.servers": "localhost:9092", "consumer.client.id": "mirus-quickstart", "consumer.key.deserializer": "org.apache.kafka.common.serialization.ByteArrayDeserializer", "consumer.value.deserializer": "org.apache.kafka.common.serialization.ByteArrayDeserializer" }'
I made no additional changes to the worker file above, but did set destination.bootstrap.servers and name fields in the connection setup in my attempt to replicate to a separate system:
curl -o - localhost:8083/connectors/MIRUS-quickstart-source/config\ -X PUT \ -H 'Content-Type: application/json' \ -d '{ "name": "MIRUS-quickstart-source", "connector.class": "com.salesforce.mirus.MirusSourceConnector", "tasks.max": "5", "topics.whitelist": "test", "destination.topic.name.suffix": ".mirror", "destination.bootstrap.servers": "172.29.82.125:9092", "consumer.bootstrap.servers": "localhost:9092", "consumer.client.id": "MIRUS-quickstart", "consumer.key.deserializer": "org.apache.kafka.common.serialization.ByteArrayDeserializer", "consumer.value.deserializer": "org.apache.kafka.common.serialization.ByteArrayDeserializer" }'
from mirus.
This config looks right to me. Anything interesting in the logs? Also, did you make sure you have these configs right in the worker config file https://github.com/salesforce/mirus#mirus-specific-configuration ? Specifically, I can think of producer.bootstrap.servers=172.29.82.125:9092
from mirus.
This is an example of what producer configs you might need to have in your worker config file
producer.bootstrap.servers=<server-list>:<port-num>
producer.compression.type=gzip
from mirus.
The log dir (mirus-master/quickstart/logs/) is empty. I'll restart Mirus with the addition of the producer configuration and tee the output .
from mirus.
Mirus, with the producer controls added to the worker config, appears to start cleanly, but replication is not occurring between the two systems (separate VMs with kafka running on both). No errors in the Mirus logs. It's a pull configuration with Mirus running on the sink system which is supposed to receive messages posted in the source system. I'll start from zero tomorrow and if it still does not work will post the config files, connector creation curl, and logs. Thanks.
from mirus.
No luck. I deleted the kafka logs in both nodes (Hub, GP), added the provider config lines, and ran through the setup steps. Msgs are not replicating from the Hub test topic to the GP test.mirror topic. Worker config file:
##
# Mirus Quickstart worker properties file
#
# This file can be passed to `bin/mirus-worker-start.sh` to quickly get a
# basic Mirus worker up and running.
#
# ---
# Kafka broker bootstrap server - Source cluster is on a remote machine
bootstrap.servers=172.29.82.125:9092
group.id=mirus-quickstart
key.converter=org.apache.kafka.connect.converters.ByteArrayConverter
value.converter=org.apache.kafka.connect.converters.ByteArrayConverter
header.converter=org.apache.kafka.connect.converters.ByteArrayConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
config.storage.topic=gp-pull-config
status.storage.topic=gp-pull-status
offset.storage.topic=gp-pull-offsets
# In a production environment these values should be increased
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
# Recommended by Derar Alassi [mailto:[email protected]]
# This Mirus instance is on the same VM as the instance of Kafka the will
# receive the replicated messages.
producer.bootstrap.servers=localhost:9092
producer.client.id=replikator-sink
producer.compression.type=gzip
producer.security.protocol=SSL
Connector curl:
curl localhost:8083/connectors/mirus-quickstart-source/config \
-X PUT \
-H 'Content-Type: application/json' \
-d '{
"name": "mirus-quickstart-source",
"connector.class": "com.salesforce.mirus.MirusSourceConnector",
"tasks.max": "5",
"topics.whitelist": "test",
"destination.topic.name.suffix": ".mirror",
"destination.bootstrap.servers": "localhost:9092",
"consumer.bootstrap.servers": "172.29.82.125:9092",
"consumer.client.id": "mirus-quickstart",
"consumer.key.deserializer": "org.apache.kafka.common.serialization.ByteArrayDeserializer",
"consumer.value.deserializer": "org.apache.kafka.common.serialization.ByteArrayDeserializer"
}'
Mirus log:
mirus.log
One odd thing - when I created the connector the three storage config topics (gp-pull-*) appeared in both kafka instances.
from mirus.
Related Issues (20)
- Mirror not correct HOT 1
- Getting similar function as "measuring lag" (mirrormaker) in mirus HOT 6
- Deprecate Prefix & Suffix settings in favor of RegexRouter HOT 1
- Enabling SSL connectivity HOT 9
- failed to run the quick start example HOT 4
- Mirus failing to start when brokers are deployed to new ip's HOT 7
- Can we manipulate the filter any of the config parameters(eg. password configuration) during the GET call HOT 5
- mirus.connector.auto.restart.enabled doesnt work HOT 1
- org.apache.kafka.common.errors.RecordTooLargeException produce duplicated messages
- API end point for health check HOT 2
- Mirus deployed in k8s pod is failing with "java.lang.OutOfMemoryError" HOT 2
- Mirus tasks failing with error "Could not initialize class org.xerial.snappy.Snappy"
- Can Mirus run as a Kafka Connect connector? HOT 2
- Not able to convert header value of replicated topic from ByteArray to String HOT 1
- Produce msg to topic test does not reflect on test.mirror HOT 1
- Default consumer offset reset policy - earliest HOT 3
- ReplayPolicy.IGNORE does not process new records HOT 2
- SSL configuration
- Excessive logging (Exception thrown while calling task.commitRecord())
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mirus.