topicctl for Mac

A tool for easy, declarative management of Kafka topics. Includes the ability to "apply" topic changes from YAML as well as a repl for interactive exploration of brokers, topics, consumer groups, messages, and more.

Motivation

Managing Kafka topics via the standard tooling can be tedious and error-prone; there is no standard, declarative way to define topics (e.g., YAML files that can be checked-in to git), and understanding the state of a cluster at any given point in time requires knowing and using multiple, different commands with different interfaces.

We created topicctl to make the management of our Kafka topics more transparent and user-friendly. The project was inspired by kubectl and other tools that we've used in non-Kafka-related contexts.

See this blog post for more details.

? Upgrading from v0

We recently revamped topicctl to support ZooKeeper-less cluster access as well as some additional security options (TLS/SSL and SASL)! See this blog post to learn more about why and how we did this.

All changes should be backwards compatible, but you'll need to update your cluster configs if you want to take advantage of these new features; see the clusters section below for more details on the latest config format.

The code for the old version has been preserved in the v0 branch if you run into problems and need to revert.

Related projects

Check out the data-digger for a command-line tool that makes it easy to tail and summarize structured data in Kafka.

Getting started

Installation

Either: Run go install github.com/segmentio/topicctl/cmd/topicctl@latest Clone this repo and run make install in the repo root Use the Docker image: docker pull segment/topicctl

If you use (1) or (2), the binary will be placed in $GOPATH/bin.

If you use Docker, you can run the tool via docker run -ti --rm segment/topicctl [subcommand] [flags]. Depending on your Docker setup and what you're trying to do, you may also need to run in the host network via --net=host and/or mount local volumes with -v.

Quick tour Start up a 6 node Kafka cluster locally: Run the net alias script to make the broker addresses available on localhost: Apply the topic configs in examples/local-cluster/topics: Send some test messages to the topic-default topic: Open up the repl (while keeping the tester running in a separate terminal): Run some test commands: Increase the number of partitions in the topic-default topic by changing the partitions: ... value in topic-default.yaml to 9 and re-applying: Bring down the local cluster:

Usage

Subcommands

apply

The apply subcommand ensures that the actual state of a topic in the cluster matches the desired state in its config. If the topic doesn't exist, the tool will create it. If the topic already exists but its cluster state is out-of-sync, then the tool will initiate the necessary changes to bring it into compliance.

See the Config formats section below for more information on the expected file formats.

bootstrap