How to Install and Set Up Apache Kafka on Windows, Mac, and Linux: Step-by-Step Guide

Apache Kafka is a distributed event streaming platform capable of handling trillions of events a day. This guide will walk you through setting up a single-node Kafka environment on your local machine.

1: Prerequisites

Before installing Kafka, ensure you have the following:

• Java Development Kit (JDK) 8 or later: Kafka is built with Java, so you need a compatible JDK installed.

2: Download and Extract Kafka

1. Download Kafka: Visit the official Apache Kafka website and download the binary for the latest stable version (e.g., Scala 2.13).
2. Extract the Files: Extract the downloaded `.tgz` file to a location of your choice.

3: Start ZooKeeper and Kafka

Kafka relies on ZooKeeper for managing the cluster's state. You need to start ZooKeeper before starting the Kafka server.

1. Start ZooKeeper: Open a terminal, navigate to your Kafka directory, and run:

   On Mac/Linux:

   bin/zookeeper-server-start.sh config/zookeeper.properties

   On Windows:

   .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

2. Start Kafka Broker: Open a new terminal, navigate to your Kafka directory, and run:

   On Mac/Linux:

   bin/kafka-server-start.sh config/server.properties

   On Windows:

   .\bin\windows\kafka-server-start.bat .\config\server.properties

4: Create a Topic and Test

Now, let's test your Kafka installation by creating a topic and sending some messages.

1. Create a Topic: Open a new terminal and create a topic named `test-topic`:

   On Mac/Linux:

   bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1

   On Windows:

   .\bin\windows\kafka-topics.bat --create --topic test-topic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1

2. Start a Producer: In the same terminal, start a console producer to send messages to your topic:

   On Mac/Linux:

   bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092

   On Windows:

   .\bin\windows\kafka-console-producer.bat --topic test-topic --bootstrap-server localhost:9092

3. Start a Consumer: Open a new terminal and start a console consumer to receive messages:

   On Mac/Linux:

   bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092

   On Windows:

   .\bin\windows\kafka-console-consumer.bat --topic test-topic --from-beginning --bootstrap-server localhost:9092

4. Send Messages: Go back to the producer terminal and type some messages. You should see them appear in the consumer terminal.

5: Conclusion

Congratulations! You have successfully installed and tested Apache Kafka on your system. You can now explore more advanced features like building streaming applications.

FAQ

Q: Is this setup suitable for production?
A: This single-node setup is for development and learning purposes only. For production, you should set up a multi-broker cluster for high availability and fault tolerance.

Q: How do I stop Kafka and ZooKeeper?
A: First, stop the producer and consumer with `Ctrl+C`. Then, stop the Kafka broker server and the ZooKeeper server using `Ctrl+C` in their respective terminals.

Read Next