DEV Community

Vijay Ashley Rodrigues
Vijay Ashley Rodrigues

Posted on

No Docker, No Problem: Run Apache Kafka on Windows

#kafka #docker #developers #microsoft

apache-kafka-without-docker

Getting Kafka up and running without Docker on a Windows machine might seem like navigating a maze—most tutorials assume you're working on Linux or using containers. But what if you're working directly on Windows, or inside WSL, and want full visibility into every config and step?

I ran into the same challenge—and after hours of tweaking, testing, and debugging, I finally got it all working. This article documents the entire process, step-by-step, so you can skip the trial and error and get straight to building.

Let’s dive into the full local setup: from environment setup to producing and consuming messages with Python.

Step 1: Activate Your Python Virtual Environment

Make sure your virtual environment is activated:

F:\python_virtual_environments\orchestration_env\Scripts\activate
Enter fullscreen mode Exit fullscreen mode

You should see your prompt change to:

(orchestration_env) F:\python_virtual_environments>
Enter fullscreen mode Exit fullscreen mode

Step 2: Install the Kafka Python Library

With your virtual environment active, install:

pip install kafka-python
Enter fullscreen mode Exit fullscreen mode

Verify:

python -c "import kafka; print(kafka.__version__)"
Enter fullscreen mode Exit fullscreen mode

Step 3: Configure Kafka and Java Environment Variables

Kafka Variables:

  1. Open System Properties → Environment Variables.
  2. Add:
  • KAFKA_HOME = F:\kafka_2_13__3_4_0
  • Add to Path: F:\kafka_2_13__3_4_0\bin\windows

Java Variables:
If not already set:

  • JAVA_HOME = C:\Program Files\Java\jdk-<version>

Restart your terminal to apply.

Step 4: Update Kafka Config Files

Navigate to:

F:\kafka_2_13__3_4_0\config
Enter fullscreen mode Exit fullscreen mode

Edit zookeeper.properties:

clientPort=2182
Enter fullscreen mode Exit fullscreen mode

Edit server.properties:

broker.id=0
zookeeper.connect=localhost:2182
log.dirs=F:/kafka_2_13__3_4_0/logs
Enter fullscreen mode Exit fullscreen mode

Step 5: Delete meta.properties (Fix for InconsistentClusterId)

Stop Kafka:

F:\kafka_2_13__3_4_0\bin\windows\kafka-server-stop.bat
Enter fullscreen mode Exit fullscreen mode

Stop Zookeeper:

F:\kafka_2_13__3_4_0\bin\windows\zookeeper-server-stop.bat
Enter fullscreen mode Exit fullscreen mode

Delete meta.properties:

Navigate to: F:\kafka_2_13__3_4_0\logs
Delete the file: meta.properties

Step 6: Start Zookeeper 

F:\kafka_2_13__3_4_0\bin\windows\zookeeper-server-start.bat F:\kafka_2_13__3_4_0\config\zookeeper.properties
Enter fullscreen mode Exit fullscreen mode

Wait for Zookeeper to fully initialize.

Step 7: Start Kafka Broker 

F:\kafka_2_13__3_4_0\bin\windows\kafka-server-start.bat F:\kafka_2_13__3_4_0\config\server.properties
Enter fullscreen mode Exit fullscreen mode

Step 8: Create a Kafka Topic 

F:\kafka_2_13__3_4_0\bin\windows\kafka-topics.bat -create -topic test-topic -bootstrap-server localhost:9092 -partitions 1 -replication-factor 1
Enter fullscreen mode Exit fullscreen mode

Verify:

F:\kafka_2_13__3_4_0\bin\windows\kafka-topics.bat -list -bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Step 9: Use CMD for Producer and Consumer

Start Producer:

F:\kafka_2_13__3_4_0\bin\windows\kafka-console-producer.bat -topic test-topic -bootstrap-server localhost:9092
Enter fullscreen mode Exit fullscreen mode

Type your messages in this window.

Start Consumer:

F:\kafka_2_13__3_4_0\bin\windows\kafka-console-consumer.bat -topic test-topic -bootstrap-server localhost:9092 -from-beginning
Enter fullscreen mode Exit fullscreen mode

This window will show the messages received.

Step 10: Send & Receive Messages Using Python

Create Scripts:

kafka_producer.py

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('test-topic', b'Hello from Python!')
producer.flush()

print("Message sent.")
Enter fullscreen mode Exit fullscreen mode

kafka_consumer.py

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'test-topic',
    bootstrap_servers='localhost:9092',
    auto_offset_reset='earliest',
    group_id='test-group'
)

for message in consumer:
    print(f"Received: {message.value.decode()}")
Enter fullscreen mode Exit fullscreen mode

Run the Scripts:

Before running any Kafka-related Python scripts, first activate your
virtual environment:

F:\python_virtual_environments\orchestration_env\Scripts\activate
Enter fullscreen mode Exit fullscreen mode

Once activated, your prompt will look like:

(orchestration_env) F:\python_virtual_environments>
Enter fullscreen mode Exit fullscreen mode

Now, navigate to the Kafka scripts directory:

cd F:\kafka-scripts
Enter fullscreen mode Exit fullscreen mode

Run the Producer script:

python kafka_producer.py
Enter fullscreen mode Exit fullscreen mode

Then, open another CMD window, activate the virtual environment again, and run the Consumer script:

F:\python_virtual_environments\orchestration_env\Scripts\activate
cd F:\kafka-scripts
python kafka_consumer.py
Enter fullscreen mode Exit fullscreen mode

Step 11: Stop Kafka and Zookeeper Safely

Once you're finished, stop both services gracefully:

F:\kafka_2_13__3_4_0\bin\windows\kafka-server-stop.bat
F:\kafka_2_13__3_4_0\bin\windows\zookeeper-server-stop.bat
Enter fullscreen mode Exit fullscreen mode

🙌 Final Thoughts

Setting up Apache Kafka on Windows without Docker can feel tricky at first, but once you walk through the steps, it’s actually a clean and flexible solution — perfect for local development or experimentation.

If this guide helped you or saved you time, consider leaving a ❤️, bookmarking it for later, or sharing your experience in the comments.
I’d love to hear how it went for you — or answer any questions you might have!

Top comments (0)