The New Relic Trace API lets you send your own distributed traces to New Relic, bypassing the usual agent instrumentation. This is super handy when you’ve got custom services, legacy code, or anything else that doesn’t play nice with standard agents, and you still want to see how requests flow across your entire system.
Here’s how you’d send a custom span using curl:
curl -X POST 'https://trace-api.newrelic.com/trace/v1' \
-H 'Content-Type: application/json' \
-H 'Api-Key: YOUR_API_KEY' \
-d '{
"v": 0,
"d": [
{
"traceId": "my-custom-trace-123",
"spanId": "my-custom-span-456",
"parentSpanId": "my-parent-span-789",
"name": "MyCustomService.processRequest",
"kind": "SERVER",
"startTime": 1678886400000,
"duration": 150,
"attributes": {
"http.method": "POST",
"http.url": "/api/v1/data",
"http.statusCode": 200,
"service.name": "MyCustomApp",
"cloud.provider": "aws",
"aws.region": "us-east-1"
}
}
]
}'
Let’s break down what’s happening here. The core idea is to represent a unit of work (a span) within a larger transaction (a trace). Each span has a unique spanId and belongs to a traceId. If this span was initiated by another service or operation, it will have a parentSpanId. The name is what shows up in New Relic’s UI, and kind tells us if it’s a server-side operation (SERVER) or a client-side call (CLIENT). startTime and duration are in milliseconds since the Unix epoch, and attributes are key-value pairs that provide context – think HTTP methods, URLs, status codes, or cloud provider details.
When New Relic receives these spans, it stitches them together based on traceId and parentSpanId to reconstruct the end-to-end flow of a request. This means you can see not only how long your custom service took to process a request but also how it fits into the bigger picture alongside services instrumented by New Relic agents. You can visualize this as a waterfall chart in the "Distributed Tracing" section of New Relic One.
The attributes are where the real power lies. You can attach almost any relevant metadata. For instance, if you’re processing a message from Kafka, you might include attributes like messaging.system: "kafka", kafka.topic: "user-updates", and kafka.partition: 3. If it’s an AWS Lambda function, you’d add cloud.provider: "aws", cloud.region: "us-west-2", and faas.execution: "some-lambda-invocation-id". New Relic can then use these attributes for filtering, searching, and even creating custom dashboards.
The v field indicates the version of the Trace API payload. Currently, 0 is the standard. The d field is an array containing one or more span objects. You can batch multiple spans in a single API call, which is more efficient than sending each one individually.
A crucial point is how New Relic links these custom spans to existing traces. If a request comes into your custom service from a service already instrumented by a New Relic agent, that agent will have injected W3C Trace Context or New Relic-specific trace headers into the request. Your custom service needs to extract these headers (e.g., traceparent and tracestate for W3C) and use them to populate the traceId and parentSpanId of the spans it generates. This ensures your custom spans are correctly attached to the existing trace. If your custom span is the start of a trace (e.g., an incoming webhook that doesn’t have incoming trace context), you’d omit the parentSpanId and generate a new traceId.
When you’re debugging a slow request and see a gap in your distributed trace, it’s often because a service in the middle isn’t sending spans. Using the Trace API, you can instrument those missing pieces manually. You’d typically have your custom code capture the incoming trace context headers, start a span when processing begins, add relevant attributes, and then end the span when processing finishes, sending it to the Trace API. The kind: "SERVER" attribute is particularly important for incoming requests to distinguish them from outgoing calls.
The startTime and duration fields are critical for accurate performance measurement. Ensure your clock is synchronized and that you’re measuring the actual work done within your service. Incorrect timestamps can lead to misleading visualizations and incorrect aggregations. The duration should be the wall-clock time elapsed for that specific operation.
If you’re sending spans for a client-side operation (e.g., your service making an HTTP request to another service), you’d set kind: "CLIENT". The parentSpanId in this case would be the spanId of the operation that initiated this outgoing request. This allows New Relic to show the full chain of calls, both incoming and outgoing, from your service’s perspective.
The API Key you use should be a "Trace API key" obtained from your New Relic account settings, not a general ingest key. This ensures the data is directed specifically to the distributed tracing ingestion pipeline.
The next thing you’ll want to explore is how to propagate trace context across different protocols, like gRPC or message queues, to maintain end-to-end visibility.