This guide walks through building observability into a service from scratch using OpenTelemetry. By the end you will have traces, metrics, and logs flowing into a collector and visualized in Grafana.

Part 1: Why Observability Matters

Logs tell you what happened. Metrics tell you how much. Traces tell you why. Together they give you the full picture when something breaks at 3 AM.

OpenTelemetry is the industry standard for instrumentation. It is vendor-neutral, has SDKs for every major language, and integrates with every major backend (Jaeger, Prometheus, Datadog, Grafana Cloud).

Part 2: Your First Trace

Install the Go SDK:

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk

Create a tracer provider and wrap your HTTP handler:

func main() {
    tp := initTracerProvider()
    defer tp.Shutdown(context.Background())

    mux := http.NewServeMux()
    mux.HandleFunc("/api/users", usersHandler)

    handler := otelhttp.NewHandler(mux, "server")
    http.ListenAndServe(":8080", handler)
}

Every incoming request now generates a span with method, route, status code, and duration.

Part 3: Adding Custom Spans

Automatic instrumentation covers the edges. For the interesting parts — database queries, cache lookups, business logic — add manual spans:

func getUser(ctx context.Context, id string) (*User, error) {
    ctx, span := tracer.Start(ctx, "getUser")
    defer span.End()

    span.SetAttributes(attribute.String("user.id", id))

    // ...
}

Part 4: Metrics and the Collector

Coming soon — configuring the OTel Collector, Prometheus scraping, and Grafana dashboards.