Detailed video of every step here:
---->
https://drive.google.com/file/d/1MQC5X1bzX7G6sivb77Nao2t7I-O15nA8/view?usp=sharing
First clone the git repo, and enter in the folder:
git clone https://github.com/angietd94/confluent-cloud-hybrid-demo.git
cd confluent-cloud-hybrid-demo
###Guide: to find values in Confluent Cloud:
CLOUD_KEY=""
CLOUD_SECRET=""
export CLUSTERID="lkc-***"
insidde the cluster information . Cluster > Cluster Settings.
#Either us, eu
export GEO="eu"
export REGION="eu-central-1"
CLUSTERAPIKEY=""
CLUSTERAPISECRET=""
#for example, aws or gcp
CLOUDPROVIDER="aws"
SR_API_KEY=""
SR_API_SECRET=""
SCHEMA_REGISTRY_URL="***+"
The URL in the down-right side!
BOOTSTRAP_SERVERS="*****:****"
![IMAGE_DESCRIPTION](img/bootstrap.png)
export TOPIC=""
export DATABASE_NAME="default"
export DATABASE_PASSWORD=""
export DATABASE_HOSTNAME="**"
export CLICKHOUSE_HOSTNAME=""
############
then:
chmod +x executable.sh
chmod +x config_files.sh
chmod +x config.sh
chmod +x ccloud-generate-cp-configs.sh
./executable.sh
More information on the part of Auto generating:
Example: Autogenerate Self-Managed Component Configs for Confluent Cloud
https://docs.confluent.io/cloud/current/cp-component/auto-generate-configs.html
Can you see the Control center here?
http://localhost:9021/clusters
and the plugins here?
http://localhost:8083/connector-plugins
What I see:
if YES!
CREATE A TOPIC , I called it simple_example_avro for example with a simple schema:
{
"doc": "Sample schema to help you get started.",
"fields": [
{
"name": "age",
"type": "int"
},
{
"doc": "The string is a unicode character sequence.",
"name": "name",
"type": "string"
}
],
"name": "sampleRecord",
"namespace": "com.mycorp.mynamespace",
"type": "record"
}
and then after this launch config_files.sh making sure all values are good (they should) This will add a connector for clickhouse
./config_file.sh
You can verify it here, or in the control center!
http://localhost:8083/connectors/
CREATE A TABLE IN CLICKHOUSE SIMILAR TO THE STRUCTURE OF YOUR TOPIC
for example:
CREATE OR REPLACE TABLE simple_example_avro
(
age Int64,
name String
)
ENGINE = MergeTree
ORDER BY name;
Try send messages in CC something lik:
{
"name": "angelica",
"orderid": 29
}
And the outcome would look like this!
Also some videos here that go through all the process: https://drive.google.com/file/d/1xUpOMX-cFQDH5BVCfKHOc4IMn8uYmNxV/view https://drive.google.com/file/d/1Zr-Nkpmpd2leGuErGQRAPJWxmiheWZTB/view?usp=sharing