OpenTelemetry is not just about tracing; it’s the future of observability, aiming to standardize how we collect telemetry data across all your services, regardless of language or framework.
Let’s see it in action with a simple Go web service.
main.go:
package main
import (
"context"
"fmt"
"io"
"log"
"net/http"
"time"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/resource"
"go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.12.0" // Use a specific version
)
var tracer = otel.Tracer("my-go-service")
func initTracer() (*trace.TracerProvider, error) {
// 1. Create an exporter. This one prints to stdout.
exporter, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
if err != nil {
return nil, fmt.Errorf("failed to create stdout exporter: %w", err)
}
// 2. Define service attributes.
res, err := resource.New(context.Background(),
resource.WithAttributes(
semconv.ServiceNameKey.String("go-web-server"),
semconv.ServiceVersionKey.String("1.0.0"),
),
)
if err != nil {
return nil, fmt.Errorf("failed to create resource: %w", err)
}
// 3. Create a TracerProvider.
tp := trace.NewTracerProvider(
trace.WithBatcher(exporter), // Use a batch exporter for efficiency
trace.WithResource(res),
)
otel.SetTracerProvider(tp)
// 4. Set the global text map propagator. This is crucial for distributed tracing.
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{}, // W3C Trace Context
propagation.Baggage{}, // W3C Baggage
))
return tp, nil
}
func helloHandler(w http.ResponseWriter, r *http.Request) {
// Extract the span context from the incoming request headers.
// This is how parent spans are identified in distributed tracing.
ctx := otel.GetTextMapPropagator().Extract(r.Context(), propagation.HeaderCarrier(r.Header))
// Start a new span for this request.
// `TraceID` and `SpanID` will be inherited from the incoming context if present.
ctx, span := tracer.Start(ctx, "helloHandler",
trace.WithSpanKind(trace.SpanKindServer), // Indicate this is a server-side span
trace.WithAttributes(
attribute.String("http.method", "GET"),
attribute.String("http.url", r.URL.String()),
),
)
defer span.End() // Ensure the span is always ended.
// Simulate some work.
time.Sleep(100 * time.Millisecond)
// Call another service (simulated).
callExternalService(ctx)
fmt.Fprintln(w, "Hello, World!")
}
func callExternalService(ctx context.Context) {
// Start a new span for the external call.
ctx, span := tracer.Start(ctx, "callExternalService",
trace.WithSpanKind(trace.SpanKindClient), // Indicate this is a client-side span
trace.WithAttributes(
attribute.String("peer.service", "external-api"),
attribute.String("db.system", "postgresql"),
),
)
defer span.End()
// Simulate network latency.
time.Sleep(50 * time.Millisecond)
}
func main() {
tp, err := initTracer()
if err != nil {
log.Fatalf("failed to initialize tracer: %v", err)
}
defer func() {
if err := tp.Shutdown(context.Background()); err != nil {
log.Printf("Error shutting down tracer provider: %v", err)
}
}()
http.HandleFunc("/hello", helloHandler)
log.Println("Server starting on :8080")
log.Fatal(http.ListenAndServe(":8080", nil))
}
When you run this Go service, you’ll see output like this on your console:
{
"traceid": "...",
"spanid": "...",
"parentspanid": "...",
"name": "helloHandler",
"kind": "SPAN_KIND_SERVER",
"start_time": "...",
"end_time": "...",
"attributes": [
{
"key": "http.method",
"value": {
"string_value": "GET"
}
},
{
"key": "http.url",
"value": {
"string_value": "/hello"
}
}
],
"events": [],
"status": {
"code": "STATUS_CODE_UNSET"
},
"trace_state": "",
"resource": {
"attributes": [
{
"key": "service.name",
"value": {
"string_value": "go-web-server"
}
},
{
"key": "service.version",
"value": {
"string_value": "1.0.0"
}
}
]
}
}
{
"traceid": "...",
"spanid": "...",
"parentspanid": "...",
"name": "callExternalService",
"kind": "SPAN_KIND_CLIENT",
"start_time": "...",
"end_time": "...",
"attributes": [
{
"key": "peer.service",
"value": {
"string_value": "external-api"
}
},
{
"key": "db.system",
"value": {
"string_value": "postgresql"
}
}
],
"events": [],
"status": {
"code": "STATUS_CODE_UNSET"
},
"trace_state": "",
"resource": {
"attributes": [
{
"key": "service.name",
"value": {
"string_value": "go-web-server"
}
},
{
"key": "service.version",
"value": {
"string_value": "1.0.0"
}
}
]
}
}
The core idea is that a TracerProvider manages the lifecycle of Tracers. Each Tracer is associated with a specific service name (via resource.WithAttributes) and is used to create Spans. A Span represents a single operation within a trace, like an incoming HTTP request (SPAN_KIND_SERVER) or an outgoing RPC call (SPAN_KIND_CLIENT).
The initTracer function sets up the observability pipeline:
- Exporter: Where the telemetry data goes. Here,
stdouttracesends it to standard output for easy inspection. In production, you’d use an exporter for Jaeger, OTLP, or your preferred backend. - Resource: Metadata about the service generating the telemetry (e.g.,
service.name,service.version). This is attached to every span. - TracerProvider: The central component that configures exporters and resources, and provides
Tracerinstances.otel.SetTracerProvider(tp)makes it globally accessible. - Propagator: This is the magic for distributed tracing. When a service makes a request to another service, it needs to carry tracing context (like
trace_idandspan_id) in the request headers. The propagator serializes this context into headers and deserializes it on the receiving end.propagation.TraceContext{}is the W3C standard for this.
In helloHandler, we first Extract the incoming context. If this request is part of an existing trace, the trace_id and parent_span_id will be there. Then, tracer.Start creates a new span. If a trace_id was extracted, the new span will belong to that trace. If not, a new trace_id is generated. defer span.End() is crucial; it marks the span as complete and sends it to the exporter. The callExternalService function demonstrates creating a child span within the same trace, again by passing the context.
The traceid, spanid, and parentspanid fields in the output JSON are the key to reconstructing the request flow. All spans with the same traceid belong to the same distributed trace. A span whose parentspanid matches another span’s spanid is a child of that span.
The otel.GetTextMapPropagator().Extract(r.Context(), propagation.HeaderCarrier(r.Header)) line is where the distributed magic happens. It looks for specific headers (like traceparent and tracestate from W3C Trace Context) and injects that context into the Go context.Context. When tracer.Start is called with this enriched context, the new span inherits the existing trace ID and is linked as a child to the span that initiated the request.
When you want to send this data to a backend like Jaeger, you’d replace stdouttrace.New() with jaeger.New() (from go.opentelemetry.io/otel/exporters/jaeger) or otlptracegrpc.New() (from go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc). The rest of the instrumentation code remains largely the same because OpenTelemetry provides a vendor-neutral API.
The most common pitfall is forgetting to propagate the context across service boundaries. If you don’t Extract the incoming context and Inject it into outgoing requests, each service will start a new, unrelated trace, and you’ll lose the ability to see the end-to-end flow.
The next step is often integrating metrics and logs using the same OpenTelemetry SDK, allowing you to correlate traces, metrics, and logs within a single observability framework.