DEV Community

Cover image for 🔭 OpenTelemetry in Action on Kubernetes: Part 2 - Instrument And Dockerizing
Kartik Dudeja
Kartik Dudeja

Posted on

🔭 OpenTelemetry in Action on Kubernetes: Part 2 - Instrument And Dockerizing

#opentelemetry #kubernetes #observability

🧭 What’s Happening in Part 2?

Welcome back, observability explorers! In Part 1, we built a basic FastAPI app to predict house prices with the help of linear regression. But let’s be honest — a naked app in production is like flying blind with no cockpit instruments.

So in this chapter of our telemetry tale, we’re going to:

  • Add OpenTelemetry instrumentation for traces, metrics, and logs
  • Generate custom metrics for traffic and latency
  • Log everything in beautiful JSON format
  • Dockerize our instrumented app
  • Run and observe logs in action

By the end, your app will be telemetry-ready, trace-emitting, log-spraying, and metric-capturing — just like a real production-grade microservice.

otel-k8s

🧪 Let’s Talk Observability Instrumentation

Here’s what we added to our Python FastAPI app:

✅ Tracing

  • We're using OpenTelemetry’s SDK to generate spans for each endpoint.
  • Each request to / or /predict is wrapped in a tracer span.
  • We tag the /predict span with the input features as metadata.

✅ Metrics

  • We define two custom metrics:

    • api_requests_total: A counter to track total API hits
    • api_latency_seconds: A histogram to record request duration

  • Each metric includes useful labels like endpoint and method.

✅ Logging

  • Logs are now output in structured JSON, perfect for parsing.

  • We log helpful events like:

    • Health check hits
    • Predictions made

Here’s a breakdown of the key code snippets:

📦 Tracing Setup 

trace.set_tracer_provider(TracerProvider(resource=resource))
tracer = trace.get_tracer(__name__)
span_exporter = OTLPSpanExporter()
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(span_exporter))
Enter fullscreen mode Exit fullscreen mode

⛳ OTLP = OpenTelemetry Protocol. We're exporting spans using OTLP over gRPC — which will be picked up by the OpenTelemetry Collector later in the series.

📊 Metrics Setup 

metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[metric_reader]))
api_counter = meter.create_counter("api_requests_total", ...)
api_latency = meter.create_histogram("api_latency_seconds", ...)
Enter fullscreen mode Exit fullscreen mode

🧠 Custom metrics give you way more flexibility than auto-generated ones — and they're fun to build!

🧾 Logging in JSON 

class JsonFormatter(logging.Formatter):
    def format(self, record):
        ...
Enter fullscreen mode Exit fullscreen mode

🧪 We format every log entry into JSON — making them machine-parseable and human-readable (if your brain speaks JSON).

🌐 Application Routes

We wrap each route in a span, log the action, increment counters, and record latency:

@app.get("/")
def read_root(request: Request):
    with tracer.start_as_current_span("GET /", kind=SpanKind.SERVER):
        logger.info("Health check hit")
        api_counter.add(1, {"endpoint": "/", "method": "GET"})
        api_latency.record(...)
Enter fullscreen mode Exit fullscreen mode

📜 Full Python Code — Instrumented with Tracing, Metrics, and Logs

Here's the Complete Code After Instrumentation:

from fastapi import FastAPI, Request
from pydantic import BaseModel
import pickle
import numpy as np
import logging
import json
import time

from opentelemetry import trace, metrics
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.logging import LoggingInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.trace import SpanKind


# --------------------------
# JSON Logging Setup
# --------------------------
class JsonFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        log_entry = {
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "name": record.name,
            "message": record.getMessage(),
        }
        return json.dumps(log_entry)

logger = logging.getLogger("house-price-service")
handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logger.addHandler(handler)
logger.setLevel(logging.INFO)

# --------------------------
# OpenTelemetry Tracing
# --------------------------
resource = Resource(attributes={"service.name": "house-price-service"})

# Tracer setup
trace.set_tracer_provider(TracerProvider(resource=resource))
tracer = trace.get_tracer(__name__)

span_exporter = OTLPSpanExporter()
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(span_exporter))

# Metrics setup
metric_reader = PeriodicExportingMetricReader(OTLPMetricExporter())
metrics.set_meter_provider(MeterProvider(resource=resource, metric_readers=[metric_reader]))
meter = metrics.get_meter(__name__)

# Metrics
api_counter = meter.create_counter(
    name="api_requests_total",
    unit="1",
    description="Total number of API requests",
)

api_latency = meter.create_histogram(
    name="api_latency_seconds",
    unit="s",
    description="API response latency in seconds",
)

# --------------------------
# FastAPI App Setup
# --------------------------
app = FastAPI()
FastAPIInstrumentor().instrument_app(app)
LoggingInstrumentor().instrument(set_logging_format=True)

# Load the model
with open("house_price_model.pkl", "rb") as f:
    model = pickle.load(f)

@app.get("/")
def read_root(request: Request):
    start_time = time.time()
    with tracer.start_as_current_span("GET /", kind=SpanKind.SERVER):
        logger.info("Health check hit")
        api_counter.add(1, {"endpoint": "/", "method": "GET"})
        api_latency.record(time.time() - start_time, {"endpoint": "/", "method": "GET"})
        return {"message": "House Price Prediction API is live!"}

class HouseFeatures(BaseModel):
    features: list[float]

@app.post("/predict/")
def predict(data: HouseFeatures, request: Request):
    start_time = time.time()
    with tracer.start_as_current_span("POST /predict", kind=SpanKind.SERVER) as span:
        span.set_attribute("input.features", str(data.features))
        api_counter.add(1, {"endpoint": "/predict", "method": "POST"})

        prediction = model.predict(np.array(data.features).reshape(1, -1))
        logger.info(f"Prediction made: {prediction[0]}")

        api_latency.record(time.time() - start_time, {"endpoint": "/predict", "method": "POST"})
        return {"predicted_price": prediction[0]}
Enter fullscreen mode Exit fullscreen mode

Updated requirements.txt:

annotated-types==0.7.0
anyio==4.9.0
asgiref==3.8.1
certifi==2025.4.26
charset-normalizer==3.4.2
click==8.2.1
Deprecated==1.2.18
exceptiongroup==1.3.0
fastapi==0.110.0
googleapis-common-protos==1.70.0
grpcio==1.71.0
h11==0.16.0
httptools==0.6.4
idna==3.10
importlib_metadata==7.1.0
joblib==1.5.1
numpy==1.26.4
opentelemetry-api==1.25.0
opentelemetry-exporter-otlp==1.25.0
opentelemetry-exporter-otlp-proto-common==1.25.0
opentelemetry-exporter-otlp-proto-grpc==1.25.0
opentelemetry-exporter-otlp-proto-http==1.25.0
opentelemetry-instrumentation==0.46b0
opentelemetry-instrumentation-asgi==0.46b0
opentelemetry-instrumentation-fastapi==0.46b0
opentelemetry-instrumentation-logging==0.46b0
opentelemetry-proto==1.25.0
opentelemetry-sdk==1.25.0
opentelemetry-semantic-conventions==0.46b0
opentelemetry-util-http==0.46b0
protobuf==4.25.7
pydantic==1.10.14
pydantic_core==2.33.2
python-dotenv==1.1.0
PyYAML==6.0.2
requests==2.32.3
scikit-learn==1.4.2
scipy==1.15.3
sniffio==1.3.1
starlette==0.36.3
threadpoolctl==3.6.0
typing-inspection==0.4.1
typing_extensions==4.13.2
urllib3==2.4.0
uvicorn==0.29.0
uvloop==0.21.0
watchfiles==1.0.5
websockets==15.0.1
wrapt==1.17.2
zipp==3.21.0
Enter fullscreen mode Exit fullscreen mode

🐳 Dockerizing the App (Now with Telemetry Magic!)

Now that the app can observe itself, it’s time to containerize it.

🧱 Dockerfile Summary 

# lightweight base image
FROM python:3.10.12-slim

# metadata
LABEL app="ml-prediction-model"
LABEL env="dev"
LABEL lab="observability"

# create a non-root user and group
RUN adduser --disabled-password --gecos '' appuser

# set working directory for application
WORKDIR /app

# copy list of required dependencies
COPY requirements.txt .

# install dependencies as root
RUN pip install --no-cache-dir -r requirements.txt

# cleanup dockerfile
RUN rm -rf /tmp/*

# add application code and ml model
COPY app.py .
COPY house_price_model.pkl .

# Change ownership to the non-root user
RUN chown -R appuser:appuser /app

# Switch to the non-root user
USER appuser

# export port for application listener and otlp
EXPOSE 8000
EXPOSE 4317

# start application
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
Enter fullscreen mode Exit fullscreen mode

Let’s take a look under the hood of our Dockerfile. This container is built with observability and security in mind — no root shenanigans here.

📦 Base Image 

FROM python:3.10.12-slim
Enter fullscreen mode Exit fullscreen mode

We start with a slim and clean Python 3.10 base image — small footprint, fewer vulnerabilities, and fast builds. It’s perfect for production-ready containers.

🏷️ Metadata Labels 

LABEL app="ml-prediction-model"
LABEL env="dev"
LABEL lab="observability"
Enter fullscreen mode Exit fullscreen mode

These labels help organize and identify the image in registries or orchestrators like Kubernetes. Think of it as giving your container a business card.

👤 Security First: Create a Non-Root User 

RUN adduser --disabled-password --gecos '' appuser
Enter fullscreen mode Exit fullscreen mode

Instead of running your app as root (a big no-no in production), we create a minimal non-root user called appuser. This is a best practice for container security.

📂 Set the Working Directory 

WORKDIR /app
Enter fullscreen mode Exit fullscreen mode

This sets /app as the working directory for all subsequent commands — keeping things tidy and predictable.

📜 Install Python Dependencies 

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

We copy the dependency list into the image and install everything in one go using pip. The --no-cache-dir keeps the image size lean by avoiding pip's cache bloat.

🧹 Clean Up Temporary Files 

RUN rm -rf /tmp/*
Enter fullscreen mode Exit fullscreen mode

Another space-saving move — we wipe /tmp to keep the image leaner and meaner.

🧠 Add the App and Model 

COPY app.py .
COPY house_price_model.pkl .
Enter fullscreen mode Exit fullscreen mode

We copy our instrumented FastAPI app and the trained ML model into the image.

🔐 Set Ownership and Use Non-Root User 

RUN chown -R appuser:appuser /app
USER appuser
Enter fullscreen mode Exit fullscreen mode

We change ownership of the /app directory to appuser and switch the active user. This enforces that your app runs without elevated privileges. Good security hygiene!

🌐 Expose Ports 

EXPOSE 8000
EXPOSE 4317
Enter fullscreen mode Exit fullscreen mode
  • 8000 is the app port served by Uvicorn.
  • 4317 is the default OTLP gRPC port used by OpenTelemetry exporters to communicate with the OpenTelemetry Collector.

🚀 Start the FastAPI App 

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
Enter fullscreen mode Exit fullscreen mode

The default command starts your FastAPI app using Uvicorn, binding it to all interfaces so it’s accessible from outside the container.

✅ Summary

This Dockerfile is:

  • ✅ Minimal (slim base image)
  • ✅ Secure (non-root user)
  • ✅ Telemetry-ready (OTLP and app ports exposed)
  • ✅ Production-friendly (clean and explicit)

🧪 Build & Run the Docker Container

Run these commands:

docker build -t house-price-predictor:v2 .
docker run -d -p 8000:8000 --name house-price-predictor house-price-predictor:v2
Enter fullscreen mode Exit fullscreen mode

ml-app-start-logs

🧪 Try It Out 

curl -i -X GET 'http://127.0.0.1:8000/'
Enter fullscreen mode Exit fullscreen mode

ml-app-health-endpoint-curl

ml-app-health-endpoint-logs

POST to /predict/ with:

curl -i -X POST 'http://127.0.0.1:8000/predict/' -H "Content-Type: application/json" -d '{"features": [1500]}'
Enter fullscreen mode Exit fullscreen mode

ml-app-predict-endpoint-curl

ml-app-predict-endpoint-logs

✅ What We've Achieved

🎯 In this part, we’ve:

  • Instrumented the app with OpenTelemetry SDK
  • Added spans, logs, and custom metrics
  • Dockerized and tested the telemetry-powered service

🛣️ What’s Next: Let’s Get This Thing on Kubernetes

Now that our app is fully instrumented — logging in structured JSON, generating traces and spans, emitting custom metrics — and Dockerized for portability, it’s time to take the next big leap:

🔮 Deploying the app to a Kubernetes cluster.

In Part 3, we’ll:

  • 📦 Create Kubernetes manifest files for deploying our FastAPI + ML app.
  • 📤 Expose the app so we can access it and start generating telemetry from a real-world setup.

Oh, and yes — our OTLP port (4317) is coming along for the ride, ready to chat with the OpenTelemetry Collector once it’s up.

Grab your kubectl, fire up your cluster, and get ready — it’s time to go full cloud-native.

{
    "author"   :  "Kartik Dudeja",
    "email"    :  "[email protected]",
    "linkedin" :  "https://linkedin.com/in/kartik-dudeja",
    "github"   :  "https://github.com/Kartikdudeja"
}
Enter fullscreen mode Exit fullscreen mode

Top comments (0)