confluentinc / ccloud-tools

Running Tools from Confluent Platform along with your Confluent Cloud™ Cluster

Confluent Cloud Tools

This is an unofficial open-source project created to aid in quickly provisioning Confluent Platform Tools (i.e.: REST Proxy, Kafka Connect, KSQL, Control Center) to be used while developing applications with Confluent Cloud.

This project is based on Terraform and allows you to have the ecosystem running in the Cloud—and automatically connected to the Apache Kafka cluster. All you have to do is:

The provisioning process may take about ~5 minutes. Once it is finished, the tools will be available to use right way via the publicly exposed load balancers. The diagram below shows how this works.

Quickstart

The first thing you need to do is clone the repository. So go ahead and get yourself a copy of the Confluent Cloud Tools:

    $ git clone [email protected]:confluentinc/ccloud-tools.git <ENTER>

Alternatively, if you don't have Git installed in your computer you can simply click on the green button 'Clone or Download' available in the top of this Github repository. This will download a fresh copy of the code straight from the master branch.

Next, navigate to the folder that contains your Cloud provider implementation (i.e.: terraform/aws) and set up the credentials information from your Cloud provider in the cloud.auto.tfvars.example file. Before running Terraform, make sure to remove the ".example" suffix. Also note that each Cloud provider has their own way to specify these credentials.

You are going need to edit the ccloud.auto.tfvars.example file as well to specify information from your Confluent Cloud cluster. Before running Terraform, make sure to remove the ".example" suffix. The information from the Confluent Cloud cluster can be easily obtained via the dashboard. Go to your cluster and then access the Client Config tab. There, you can create new API keys and secret, as well as retrieve your cluster bootstrap servers. If you are new to Confluent Cloud -- you might want to watch the Getting Started with Confluent Cloud video.

Finally, you will need to run Terraform (which you can download from here if you do not have it already). While under the folder that contains your Cloud provider implementation, run:

    $ terraform init <ENTER>
    $ terraform plan <ENTER>
    $ terraform apply <ENTER>

Once the script finishes, there will be an output showing this:

As you can see in the output shown above, there is a bastion server you can use to SSH into the internal VMs. The bastion server is the only way in into the VMs, so for this reason it is disabled by default. To enable it, edit the variables.tf file and modify the instance_count variable, setting the value of bastion_server to 1.

License

The project is licensed under the Apache 2 license.

ccloud-tools's Issues

confluent.metrics.topic.max.message.bytes

Confluent Cloud has a maximum message size of 8388608 so the default value for confluent.metrics.topic.max.message.bytes of 10485760 causes a C3 infinite reboot loop:

(io.confluent.controlcenter.KafkaHelper)Feb 6 05:31:21 control-center-mcg4 control-center-start: org.apache.kafka.common.errors.PolicyViolationException: Config property 'max.message.bytes' with value '10485760' exceeded max limit of 8388608.Feb 6 05:31:21 control-center-mcg4 control-center-start: [2019-02-06 05:31:21,236] ERROR [main] Startup failed to create necessary topics (io.confluent.controlcenter.KafkaHelper)Feb 6 05:31:21 control-center-mcg4 control-center-start: [2019-02-06 05:31:21,237] INFO [main] Shutting down Kafka Streams

Please add this line to
/etc/confluent/confluent-5.1.0/etc/confluent-control-center/c3-ccloud.properties :

confluent.metrics.topic.max.message.bytes=8388608

add schema registry variables to ccloud.auto.tfvars

The variables var.ccloud_schema_registry_basic_auth and var.ccloud_schema_registry_url must be entered when prompted when doing "plan" and "apply". These should be added to the config file instead.

Feature Request: Provide terraform provider

hello confluent,

it would be greatly apprechiated if your customers could experience the same level of abstraction that you provide via your gui also when choosing to provision their kafka cluster via terraform - to achieve this, it would be really helpful if you would provide a terraform provider, who hides the underlying complexity of a kafka cluster..

Thank you for considering this request

feature request: Azure support

This is a feature request to add support for Microsoft Azure

How to delete consumer groups from non active consumers or Reset the messages lag

Hello,

We ran a trial KSQL container pointing to our cloud data topic, that created a consumer group for ingestion.
After our test, we wish:

1. To understand that these consumer groups (with constantly increasing number of messages lag) do consume disk space.
2. How to delete the consumer group or reset its pending messages (lag) count.

This is in regards to ccloud deployment of Kafka cluster.

Regards,

add interceptors

Connect distributed (properties file kafka-connect-ccloud.properties) should include these two properties to enable Consumer Lag & Data Streams in C3:

producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor

502 Bad Gateway for all components (Control Center, Kafka Connect) after applying terraform script

Hi,
We are a recent Confluent Cloud customer and I have successfully deployed the Confluent Platform tools from this repo onto AWS EC2 (Ubuntu) using the steps in your Readme, resulting in this output after applying the Terraform script:

Bastion_Server = ssh ec2-user@REDACTED -i cert.pem
Control_Center = http://control-center-ccloud-tools-REDACTED.eu-west-2.elb.amazonaws.com
KSQL_Server = http://ksql-server-ccloud-tools-REDACTED.eu-west-2.elb.amazonaws.com
Kafka_Connect = http://kafka-connect-ccloud-tools-REDACTED.eu-west-2.elb.amazonaws.com
REST_Proxy = http://rest-proxy-ccloud-tools-REDACTED.eu-west-2.elb.amazonaws.com

However, when I navigate to any of those urls I receive a 502 Bad Gateway response.

Is this the desired output? Im not sure how to go about accessing any of these tools.

I tried (for instance):

curl -sX GET "http://ksql-server-ccloud-tools-REDACTED.eu-west-2.elb.amazonaws.com/info"

and I still receive a 502 Error.

Any ideas if I am doing something wrong? Additionally, the topics that should be created by Control Center when it starts, as listed at this link:

https://docs.confluent.io/current/cloud/connect/c3-cloud-config.html

Are not listed when running ccloud kafka topic list

It seems as though none of the Cloud Tools have initialised properly...

Resource 'google_compute_instance_template.kafka_connect' not found

After modifying variables.tf to change the region from us-east1 to asia-southeast1, and the number of Connect nodes from 1->3 and the number of ksql nodes from 1->3; the plan stage fails with:

Error: Error running plan: 3 error(s) occurred:

• google_compute_region_instance_group_manager.kafka_connect: 3 error(s) occurred:

• google_compute_region_instance_group_manager.kafka_connect[1]: Resource 'google_compute_instance_template.kafka_connect' not found for variable 'google_compute_instance_template.kafka_connect.self_link'

• google_compute_region_instance_group_manager.kafka_connect[0]: Resource 'google_compute_instance_template.kafka_connect' not found for variable 'google_compute_instance_template.kafka_connect.self_link'

• google_compute_region_instance_group_manager.kafka_connect[2]: Resource 'google_compute_instance_template.kafka_connect' not found for variable 'google_compute_instance_template.kafka_connect.self_link'

• data.template_file.control_center_properties: 1 error(s) occurred:

• data.template_file.control_center_properties: Resource 'google_compute_global_address.kafka_connect' not found for variable 'google_compute_global_address.kafka_connect.address'

• google_compute_region_instance_group_manager.ksql_server: 3 error(s) occurred:

• google_compute_region_instance_group_manager.ksql_server[0]: Resource 'google_compute_instance_template.ksql_server' not found for variable 'google_compute_instance_template.ksql_server.self_link'

• google_compute_region_instance_group_manager.ksql_server[2]: Resource 'google_compute_instance_template.ksql_server' not found for variable 'google_compute_instance_template.ksql_server.self_link'

• google_compute_region_instance_group_manager.ksql_server[1]: Resource 'google_compute_instance_template.ksql_server' not found for variable 'google_compute_instance_template.ksql_server.self_link'

need a way to specify ssh key pair

I don't see a way to specify my existing key pair, just tried this and can't figure out how to log into any instance.

SSL handshake failed

instructions: add "clone or Download" + Download ZIP

As non-developer users may not have git cmd line installed, please edit the setup instructions to add "clone or Download" + Download ZIP; unzip and cd to the cloud provider subdirectory as an alternative to "git clone".

C3 property confluent.controlcenter.ksql.url must include a port (:8088)

C3 property confluent.controlcenter.ksql.url must include a port (:8088).

At present, kSQL does not start, returning an error message that kSQL is not properly configured.
After modifying the property to add :8088 in /etc/confluent/confluent-5.1.0/etc/confluent-control-center/c3-ccloud.properties and bouncing C3, kSQL then works

kafka-connect missing connectors

Due to invalid configuration kafka-connect loads only the org.apache.kafka.connect.file.FileStreamSinkConnector (visible by visiting kafka-connect-url/connector-plugins).
Temporary fix is to add plugin.path=/etc/confluent/confluent-5.1.0/share/java to /etc/confluent/confluent-5.1.0/etc/kafka-connect/kafka-connect-ccloud.properties.