DEV Community

Luiz Fernando Reis Cendretti
Luiz Fernando Reis Cendretti

Posted on

Setting Up a Go Application with Kafka in a DevContainer

#go #docker #vscode #programming

The Pain of Setting Up Development Environments

How much time have you lost configuring your environment to run multiple applications at the same time? Maybe an hour? Or even a whole month? Then, after coding for weeks, your computer crashes, and the only solution is a hard reset. Now, you find yourself in the same situation, rebuilding everything from scratch. A personal nightmare, right? 😅

Fortunately, VSCode offers a solution: running your application inside a Docker container. With this approach, the only setup you need is Docker. Like magic! ✨

What is a DevContainer?

The DevContainer feature in VSCode allows developers to configure their application environment once and run it anywhere. In this article, I'll show you how to build a Go application with Kafka using a DevContainer.

Why Go and Kafka?

Go and Kafka make a powerful combination for building high-performance microservices. Kafka allows you to distribute workloads using partitions, while Go's lightweight goroutines help maximize performance. Imagine allocating threads based on Kafka partitions—Kafka efficiently manages message distribution while Go ensures optimal processing.

🚀 Let's Get Started

1️⃣ Initialize Your Go Project

First, create a project folder and initialize a Go module:

mkdir go-with-kafka
cd go-with-kafka && go mod init go-with-kafka
Enter fullscreen mode Exit fullscreen mode

2️⃣ Set Up a DevContainer

Open your project in VSCode and create a .devcontainer folder. This will contain all the Docker configurations needed for development.

Create boot.sh

This script initializes the environment and sets up a Kafka topic:

#!/bin/zsh

# Update package list
sudo apt update

# Install Go dependencies
go mod tidy

# Change ownership of Go directories to vscode user
sudo chown -R vscode:vscode .

# Creating a Kafka topic
echo "Waiting for Kafka to start..."
sleep 10

kafka-topics --create \
  --bootstrap-server kafka:29092 \
  --replication-factor 1 \
  --partitions 1 \
  --topic example-topic || echo "The topic already exists."

echo "Topic created."
Enter fullscreen mode Exit fullscreen mode

Create docker-compose.yml

This file defines services for your Go application, Kafka, and Zookeeper:

version: '3'
services:
  app:
    build:
      context: ..
      dockerfile: .devcontainer/Dockerfile
    environment:
      - VISUAL="code --wait"
      - APPNAME=go-with-kafka
    volumes:
      - ../..:/workspaces:cached
      - ~/.ssl:/home/vscode/.ssl:ro
    command: sleep infinity

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:29092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: true
Enter fullscreen mode Exit fullscreen mode

Create Dockerfile 

FROM mcr.microsoft.com/devcontainers/go

# [Optional] Uncomment this section to install additional OS packages.
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends
Enter fullscreen mode Exit fullscreen mode

🏗️ Building a Simple Kafka Pub/Sub in Go

Create main.go 

package main

import (
    "context"
    "log"
    "os"

    "github.com/segmentio/kafka-go"
)

func main() {
    if len(os.Args) < 2 {
        log.Fatalf("Usage: %s [pub|sub]", os.Args[0])
    }

    mode := os.Args[1]
    brokers := []string{"kafka:29092"}
    topic := "example-topic"

    switch mode {
    case "pub":
        publish(brokers[0], topic)
    case "sub":
        subscribe(brokers[0], topic)
    default:
        log.Fatalf("Invalid mode: %s. Use 'pub' to publish or 'sub' to consume.", mode)
    }
}

func publish(broker string, topic string) {
    writer := kafka.Writer{
        Addr:     kafka.TCP(broker),
        Topic:    topic,
        Balancer: &kafka.LeastBytes{},
    }

defer writer.Close()

    message := kafka.Message{
        Key:   []byte("Key"),
        Value: []byte("Hello, Kafka with kafka-go!"),
    }

    err := writer.WriteMessages(context.Background(), message)
    if err != nil {
        log.Fatalf("Error sending message: %v", err)
    }

    log.Println("Message sent successfully!")
}

func subscribe(broker string, topic string) {
    reader := kafka.NewReader(kafka.ReaderConfig{
        Brokers:   []string{broker},
        Topic:     topic,
        Partition: 0,
        MinBytes:  10e3, // 10KB
        MaxBytes:  10e6, // 10MB
    })

defer reader.Close()

    log.Println("Waiting for messages...")

    for {
        message, err := reader.ReadMessage(context.Background())
        if err != nil {
            log.Fatalf("Error reading message: %v", err)
        }
        log.Printf("Message received: key=%s value=%s\n", string(message.Key), string(message.Value))
    }
}
Enter fullscreen mode Exit fullscreen mode

🎯 Running the Application

In VSCode, press Ctrl + Shift + P (or Cmd + Shift + P on macOS) and select Rebuild and Reopen in Container. Wait for the setup to complete. Once inside the container, open a terminal and run:

Start the subscriber:

go run main.go sub
Enter fullscreen mode Exit fullscreen mode

Publish a message:

go run main.go pub
Enter fullscreen mode Exit fullscreen mode

Kafka in action

If everything works, you now have a Kafka-based Go application running inside a DevContainer! 🎉

🌟 Why Use DevContainers?

DevContainers are powerful for developers who don’t want to waste time configuring their local machine. Using containers ensures that your setup is portable and consistent across different environments.

🔗 Check out more DevContainer templates: GitHub - devcontainers/templates

Top comments (0)