Coder Social home page Coder Social logo

vanoord / kafka-topology-builder Goto Github PK

View Code? Open in Web Editor NEW

This project forked from kafka-ops/julie

0.0 1.0 0.0 1.79 MB

A solution to help you build automation and gitops in your Apache Kafka deployments. The Kafka gitops!

License: MIT License

Java 94.92% Shell 4.65% Dockerfile 0.43%

kafka-topology-builder's Introduction

The Topology Builder for Apache Kafka (Automation, Gitops, SelfService)

CI tests Gitter Documentation Status

Topology Builder for Apache Kafka helps you automate the management of your things within Apache Kafka, from Topics, Configuration to Metadata but as well Access Control, Schemas. More items are plan, check here for details.

The motivation

A typical question many teams face when building, or growing, an Apache Kafka infrastructure is how to manage topics, configuration, metadata, ACLs, etc in a more automated way to provide their teams with the most autonomy possible while reducing the time involved in this administrative tasks.

The Topology Builder for Apache Kafka, in close collaboration with Git and CI/CD, (Jenkins, etc) is here to help you build it.

Where's the docs?

We recommend taking time to read the docs. There's quite a bit of detailed information about Gitops, Apache Kafka and how this project can help you automate the common operational tasks.

Automating Management with CI/CD and Gitops

KafkaTopologyBuilder

You might be wondering what is the usual workflow to implement this approach:

Action: As a user, part of a developer team (for example), I like to have some changes in Apache Kafka.

Change Request: As a user:

  • Go to the git repository where the topology is described
  • Create a new branch
  • Perform the changes need
  • Make a pull request targeting master branch

Approval process: As an ops admin, I can:

  • Review the pull request (change request) initiated by teams
  • Request changes when need
  • Merge the requests.

Considerations:

  • Using webhooks, the git server (github, gitlab or bitbucket) will inform the CI/CD system changes had happened and they need to be applied to the cluster.
  • All changes (git push) to master branch are disabled directly. Changes only can happen with a pull request.

Help??

If you are using the Kafka Topology Builder, or plan to use it in your project? might be you have encounter a bug? or a challenge? need a certain future? feel free to reach out into our gitter community.

Gitter

Feature list, not only bugs ;-)

What can you achieve with this tool:

  • Support for multiple access control mechanisms:
    • Traditional ACLs
    • Role Bases Access Control as provided by Confluent
  • Automatically set access control rules for:
    • Kafka Consumers
    • Kafka Producers
    • Kafka Connect
    • Kafka Streams applications ( microservices )
    • Schema Registry instances
    • Confluent Control Center
  • Manage topic naming with a topic name convention
    • Including the definition of projects, teams, datatypes and for sure the topic name
    • Some of the topics are flexible defined by user requirements
  • Allow for creation, delete and update of:
    • topics, following the topic naming convention
    • Topic configuration, variables like retention, segment size, etc
    • Acls, or RBAC rules
    • Service Accounts (Experimental feature only available for now in Confluent Cloud)
  • Manage your cluster schemas.
    • Support for Confluent Schema Registry

Out of the box support for Confluent Cloud and other clouds that enable you to use the AdminClient API.

How can I run the topology builder directly?

This tool is available in multiple formats:

  • As a Docker image, available from docker hub
  • As an RPM package, for the RedHat alike distributions
  • As a DEB package, for Debian based distros
  • Directly as a fat jar (zip/tar.gz)
  • As a fat jar.

The latest version are available from the releases page.

How to execute the tool

This is how you can run the tool directly as a docker image:

docker run purbon/kafka-topology-builder:latest kafka-topology-builder.sh  --help
Parsing failed cause of Missing required options: topology, brokers, clientConfig
usage: cli
    --allowDelete          Permits delete operations for topics and
                           configs.
    --brokers <arg>        The Apache Kafka server(s) to connect to.
    --clientConfig <arg>   The AdminClient configuration file.
    --dryRun               Print the execution plan without altering
                           anything.
    --help                 Prints usage information.
    --quiet                Print minimum status update
    --topology <arg>       Topology config file.
    --version              Prints useful version information.

If you install the tool as rpm, you will have available in your $PATH the kafka-topology-builder.sh. You can run this script with the same options observed earlier.

An example topology

An example topology should look like this (in yaml format):

context: "context"
source: "source"
projects:
- name: "foo"
  consumers:
  - principal: "User:app0"
  - principal: "User:app1"
  streams:
  - principal: "User:App0"
    topics:
      read:
      - "topicA"
      - "topicB"
      write:
      - "topicC"
      - "topicD"
  connectors:
  - principal: "User:Connect1"
    topics:
      read:
      - "topicA"
      - "topicB"
  - principal: "User:Connect2"
    topics:
      write:
      - "topicC"
      - "topicD"
  topics:
  - name: "foo" # topicName: context.source.foo.foo
    config:
      replication.factor: "2"
      num.partitions: "3"
  - name: "bar" # topicName: context.source.foo.bar
    config:
      replication.factor: "2"
      num.partitions: "3"
- name: "bar"
  topics:
  - name: "bar" # topicName: context.source.bar.bar
    config:
      replication.factor: "2"
      num.partitions: "3"

more examples can be found at the example/ directory.

Also, please check, the documentation in the docs for extra information and examples on managing ACLs, RBAC, Principales, Schemas and many others.

Interested in contributing back?

Interested on contributing back? might be have an idea for a great future? or wanna fix a bug? Check our contributing doc for guidance.

Building the Kafka Topology Builder from scratch (source code)

The project is build using Java and Maven, so both are required if you aim to build the tool from scratch. The minimum version of Java supported is Java 8, note it soon will be deprecated here, it is only keep as supported for very legacy environments.

It is recommended to run the Topology Builder with Java 11 and an open JDK version.

Building a release

If you are interested on building a release artifact from the source code, check our release doc for guidance.

Nightly builds as well as release builds are regularly available from the Actions in this project.

kafka-topology-builder's People

Contributors

akselh avatar christophschubert avatar dependabot[bot] avatar kikulikov avatar liko9 avatar michaelandrepearce avatar michaelpearce-gain avatar piotrsmolinski avatar purbon avatar sknop avatar sverrehu avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.