Observability with OpenTelemetry

Fri, 10 Apr 2026 10:00:00 +0300

This guide walks through building observability into a service from scratch using OpenTelemetry. By the end you will have traces, metrics, and logs flowing into a collector and visualized in Grafana.

Part 1: Why Observability Matters

Logs tell you what happened. Metrics tell you how much. Traces tell you why. Together they give you the full picture when something breaks at 3 AM.

OpenTelemetry is the industry standard for instrumentation. It is vendor-neutral, has SDKs for every major language, and integrates with every major backend (Jaeger, Prometheus, Datadog, Grafana Cloud).

Part 2: Your First Trace

Install the Go SDK:

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk

Create a tracer provider and wrap your HTTP handler:

func main() {
 tp := initTracerProvider()
 defer tp.Shutdown(context.Background())

 mux := http.NewServeMux()
 mux.HandleFunc("/api/users", usersHandler)

 handler := otelhttp.NewHandler(mux, "server")
 http.ListenAndServe(":8080", handler)
}

Every incoming request now generates a span with method, route, status code, and duration.

Part 3: Adding Custom Spans

Automatic instrumentation covers the edges. For the interesting parts — database queries, cache lookups, business logic — add manual spans:

func getUser(ctx context.Context, id string) (*User, error) {
 ctx, span := tracer.Start(ctx, "getUser")
 defer span.End()

 span.SetAttributes(attribute.String("user.id", id))

 // ...
}

Part 4: Metrics and the Collector

Coming soon — configuring the OTel Collector, Prometheus scraping, and Grafana dashboards.

Monitoring on hxmn.dev

Observability with OpenTelemetry

Part 1: Why Observability Matters

Part 2: Your First Trace

Part 3: Adding Custom Spans

Part 4: Metrics and the Collector