oracle-quickstart / oci-confluent Goto Github PK

Terraform module to deploy Confluent on Oracle Cloud Infrastructure (OCI)

License: Apache License 2.0

HCL 67.07% Shell 32.93%

terraform oci cloud confluent kafka partner-led oracle

oci-confluent's Introduction

oci-confluent

These are Terraform modules that deploy Confluent Platform on Oracle Cloud Infrastructure (OCI). They are developed jointly by Oracle and Confluent.

community deploys the Community Edition
enterprise deploys the Enterprise Edition

Architecture

Prerequisites

First off you'll need to do some pre deploy setup. That's all detailed here.

Clone the Module

Now, you'll want a local copy of this repo. You can make that with the commands:

git clone https://github.com/oracle-quickstart/oci-confluent.git

If you want to deploy community edition:

cd oci-confluent/community
ls

If you want to deploy enterprise edition (comes with 30 day free trial):

cd oci-confluent/enterprise
ls

We now need to initialize the directory with the module in it. This makes the module aware of the OCI provider. You can do this by running:

terraform init

This gives the following output:

Deploy

Now for the main attraction. Let's make sure the plan looks good:

terraform plan

That gives:

This command details what will be deployed based on the variables.tf file. If that's good, we can go ahead and apply the deploy:

terraform apply

You'll need to enter yes when prompted. The apply should take about five minutes to run. Once complete, you'll see something like this:

When the apply is complete, the infrastructure will be deployed, but cloud-init scripts will still be running. Those will wrap up asynchronously. The cluster might take ten minutes. Now is a good time to get a coffee.

The outputs of the deploy list the public ips of all the deployed instances. You can ssh into any of the instances by running a command like:

ssh -i ~/.ssh/oci opc@<instance ip>

Confluent Control Center

If you installed the enterprise version, you can login to Confluent Control Center.

View the Cluster in OCI Console

You can also login to the web console here to view the IaaS that is running from the deployment.

Virtual Cloud Network (vcn) page:

Instances page:

Create Topics, Produce and Consume Messages

First off, let's try creating a topic.

ssh opc@<broker_instance_ip>

Now create a topic by running the command

/usr/bin/kafka-topics --zookeeper zookeeper-0:2181 --create --topic demo --partitions 1 --replication-factor 3

Alternatively, if you installed Enterprise Edition, you can create a topic through the Confluent Control Center Web Console.

Now we can try adding a few messages to the topic. For instance, we can use the REST API to publish 10 messages. This can be done from any machine which has access to Kafka REST API endpoint. For example:

export RPURL=http://rest-0:8082
curl -X POST -H "Content-Type: application/vnd.kafka.json.v1+json"  --data '{"records":[{"value":{"foo":"bar"}}]}' $RPURL/topics/demo

Now let's trying consuming messages:

curl -X POST -H "Content-Type: application/vnd.kafka.v1+json" --data '{"name": "ext_consumer_demo","format": "json", "auto.offset.reset": "smallest"}' $RPURL/consumers/c1
curl -X GET -H "Accept: application/vnd.kafka.json.v1+json" $RPURL/consumers/c1/instances/ext_consumer_demo/topics/demo

oci-confluent's People

Contributors

Stargazers

Watchers

Forkers

oci-portfolio rcarrascosps iafboy

oci-confluent's Issues

consolidate and rename scripts

It's difficult to understand what all these are doing. Rework so there's a script per type of node, ie a zookeeper.sh script that runs in cloud init on the zookeeper node.

https://github.com/oci-quickstart/oci-confluent/tree/master/scripts

Control Center - Both Terraform and Script

We'll want to deploy control center on its own node and only in the commercial case.

Confluent Connect Script

add the script

Minimal Open Ports

Here's what Pinkesh has open today. The architecture diagram I drew has a public and private subnet. Need to think about that.


resource "oci_core_security_list" "PublicSubnet" {
    compartment_id = "${var.compartment_ocid}"
    display_name = "Public Subnet"
    vcn_id = "${oci_core_virtual_network.confluent_vcn.id}"
    egress_security_rules = [{
        destination = "0.0.0.0/0"
        protocol = "6"
    }]
    ingress_security_rules = [{
        tcp_options {
            "max" = 22
            "min" = 22
        }
        protocol = "6"
        source = "0.0.0.0/0"
    }]
    ingress_security_rules = [{
        protocol = "6"
	source = "${var.VPC-CIDR}"
    }]
    ingress_security_rules = [{
        tcp_options {
            "max" = 9092
            "min" = 9092
        }
        protocol = "6"
        source = "0.0.0.0/0"
    }]
    ingress_security_rules = [{
        tcp_options {
            "max" = 9021
            "min" = 9021
        }
        protocol = "6"
        source = "0.0.0.0/0"
    }]
    ingress_security_rules = [{
        tcp_options {
            "max" = 8083
            "min" = 8083
        }
        protocol = "6"
        source = "0.0.0.0/0"
    }]
    ingress_security_rules = [{
        tcp_options {
            "max" = 8088
            "min" = 8088
        }
        protocol = "6"
        source = "0.0.0.0/0"
    }]
    ingress_security_rules = [{
        tcp_options {
            "max" = 8082
            "min" = 8082
        }
        protocol = "6"
        source = "0.0.0.0/0"
    }]
    ingress_security_rules = [{
        tcp_options {
            "max" = 8081
            "min" = 8081
        }
        protocol = "6"
        source = "0.0.0.0/0"
    }]
    ingress_security_rules = [{
        tcp_options {
            "max" = 2181
            "min" = 2181
        }
        protocol = "6"
        source = "0.0.0.0/0"
    }]

}

S3 README.md Update

It seems silly to have a folder with only one item.

Use Fault Domains

We're focusing on single AD because performance is better and new regions only have a single AD. Need to map racks to FDs.

Broker - Format and Mount Disks

Crib from here:
https://github.com/oci-quickstart/oci-cloudera/tree/development

README.md Update

This would include dropping the env-vars construct for oci-prereq.

Examples of ones we should emulate are:
https://github.com/oci-quickstart/oci-couchbase
https://github.com/oci-quickstart/oci-datastax

Support for community and commercial

Use Instance Pools

Parameterize Version Number

It looks like this is going to be a little gross...
https://github.com/oci-quickstart/oci-confluent/blob/81b1cc6b4054eb429ade3c110e8a280bee360faa/scripts/install.sh#L19-L34

REST Proxy script

fill out the script

remote remote-exec dependency

use cloud-init instead

ZooKeeper Script

Currently getting this when it tries to start:

[root@zookeeper-2 log]# systemctl start confluent-zookeeper
[root@zookeeper-2 log]# systemctl status confluent-zookeeper
● confluent-zookeeper.service - Apache Kafka - ZooKeeper
Loaded: loaded (/usr/lib/systemd/system/confluent-zookeeper.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2019-03-11 00:32:20 GMT; 14s ago
Docs: http://docs.confluent.io/
Process: 12789 ExecStart=/usr/bin/zookeeper-server-start /etc/kafka/zookeeper.properties (code=exited, status=127)
Main PID: 12789 (code=exited, status=127)

Mar 11 00:32:20 zookeeper-2 systemd[1]: Started Apache Kafka - ZooKeeper.
Mar 11 00:32:20 zookeeper-2 systemd[1]: confluent-zookeeper.service: main process exited, code=exited, status=127/n/a
Mar 11 00:32:20 zookeeper-2 systemd[1]: Unit confluent-zookeeper.service entered failed state.
Mar 11 00:32:20 zookeeper-2 systemd[1]: confluent-zookeeper.service failed.
[root@zookeeper-2 log]#

Deploy Simple Cluster per Customer Ask

We have a customer asking for a simple community cluster that looks like this:

We need to be able to deploy this config solely by modifying variables.tf. The dev branch here has a similar example for Cloudera: https://github.com/oci-quickstart/oci-cloudera/tree/development/terraform

Kafka Broker Script