Docs » µAPM PG Traces, Spans, Metrics, and Metadata

µAPM PG Traces, Spans, Metrics, and Metadata 🔗

Site map for µAPM Previous Generation documentation

Deconstructing a trace 🔗

A trace is a collection of spans that share the same trace ID, representing a unique transaction handled by your application and its constituent services. Each span has a name, representing the operation captured by this span, and a service name, representing within which service the operation took place. Additionally, spans may reference another span as their parent, defining the relationships between the operations captured in the trace that were performed to process that transaction.

Span details 🔗

Each trace span contains a lot of information about the method, operation, or block of code that it captures:

  • the operation name
  • the start time of the operation with microsecond precision
  • how long the operation took to execute, also with microsecond precision
  • the logical name of the service on which the operation took place
  • the IP address of the service instance on which the operation took place

Additionally, spans may carry metadata tags that provide additional information and context about the operation captured by the span. This typically includes standard tags like component, http.method, http.url, db.statement, etc., as applicable, but may also include any other custom tags added by a developer through code instrumentation.

For more information about span metadata and tags, see Span metadata and tags.

The initiating span 🔗

For each trace, SignalFx identifies an “initiating” span: the span that represents the first, initial operation at the beginning of the transaction. The initiating span is the earliest span in a trace that does not reference another span as its parent; it’s the earliest of the root spans. Most traces only have one root span, and in most cases this root span will also be the earliest span captured.

This initiating span helps define an identity for a trace, which allows SignalFx to logically group together all the traces that represent the same flow through an endpoint of your application. Those groups are defined by the name of the service in which the trace was initiated (the service name in the initiating span), the name of the initiating operation, and when present, the HTTP method of the initiating operation (as per the http.method or requestMethod tags of the initiating span).

APM identities and SignalFx metrics 🔗

Each unique span and initiating span/endpoint tracked by the SignalFx Smart Gateway is called an APM identity. For example, the illustration below comprises 6 APM identities.

  • Each unique span (A through E) represents an APM identity that emits span metrics. The APM identity for a span is based on its service, operation, kind (if present) and error.
  • The inititating span/endpoint (A) is an APM identity that emits trace metrics. The APM identity for a trace is based on its service, operation, HTTP method (if present), and error.

For each APM identity we receive, we emit metrics with relevant dimensions. Metrics include a count of this identity (how many times we received it), duration metrics, and several quantiles of the distribution of durations: the minimum and maximum, the median (or 50th percentile), the 90th percentile, and the 99th percentile.

The duration of the APM identity for a trace is calculated as the end time of the latest span in the trace minus the start time of its initiating span.

The following illustration summarizes the metrics emitted.


The metrics and dimensions are listed below. All duration values are in nanoseconds.

Span metrics

  • spans.duration.ns.min
  • spans.duration.ns.median
  • spans.duration.ns.max
  • spans.duration.ns.p90
  • spans.duration.ns.p99
  • spans.count

Span metrics dimensions

  • service
  • operation
  • kind (when relevant)
  • error (true or false)
  • httpMethod (when the http.method or httpMethod tag is present)

Trace metrics

  • traces.duration.ns.min
  • traces.duration.ns.median
  • traces.duration.ns.max
  • traces.duration.ns.p90
  • traces.duration.ns.p99
  • traces.count

Trace metrics dimensions (based on the initiating operation of the trace)

  • service
  • operation
  • httpMethod (when relevant)
  • error (true or false)
  • httpMethod (when the http.method or httpMethod tag is present)

The following table provides more information on the dimensions that are added to each unique span or trace.

Dimension Span Source Field Required
service localEndpoint.serviceName Yes
operation name Yes
kind kind No
httpMethod tags named either http.method or requestMethod No
error true if the error field is true or 1, or if tags named http.status_code or httpStatusCode have a 5xx value No

Span metadata and tags 🔗

Beyond their instrinsic properties like name, timestamp, and duration, trace spans carry a lot of additional, free-form metadata to provide information and context about the operations that they represent. This metadata can be used to query and filter traces that contain the desired metadata, or to provide extra information about each operation when inspecting the spans of a trace during troubleshooting.

Span metadata can take one of two forms: tags_(a set of key-value pairs), or annotations_(a collection of timestamped text strings).

Span tags 🔗

Span tags are key-value pairs attached to the span to carry additional context about the operation. Span tags are kept as a map of keys to values; tag keys must therefore be unique within a given span. Both keys and values are text strings and are mostly free-form, but tag keys typically follow predefined ontologies and conventions to ensure that the same information is captured by the same tag names across spans.

Tag naming conventions 🔗

Tags work best when you can construct a simple, dependable, and clear metadata model with them. As such, you should thrive for defining clear tag names that are used throughout your applications to carry the same information. Although there is no common standard or convention around tag names, OpenTracing has laid out some proposals around common tag names and their semantic definitions.

Library and framework instrumentation, as well as automatic instrumentation, will leverage those tag names and their meaning to carry additional information about the operations and requests being traced. Whenever one of those tag names corresponds to the type of information you are capturing in a tag, you should use it. Otherwise, you can simply define your own tag names; when you do, we recommend that you be consistent in their usage.

The host tag 🔗

When following the recommended deployment model, trace spans emitted by your applications are sent to the SignalFx Smart Agent, which is responsible for automatically adding a host tag to every span captured on that host. This allows SignalFx to identify on which piece of infrastructure each trace span was executed, render the corresponding key infrastructure metrics, and link to more complete dashboards for the host. The value of the host tag will typically be the hostname of the underlying instance, or its unique resource identifier.

Special handling for certain tag names 🔗

SignalFx recognizes certain tag names and leverages those tags, when they are present, to augment the trace viewing and exploration experience.

Tag name Example value Representation
host i-005bc51542b23b995 When present, system-level metrics (CPU and memory utilization) about that host will be rendered along with the span’s metadata
http.method POST Visual pill representing the HTTP method of the operation
http.url http://domain.tld/path/example A link to the target URL of the HTTP operation
http.status_code 401 Visual pill representing the status code of the HTTP operation
component java-kafka Monospace font
error true Captured in the list of error spans
db.statement SELECT id, name FROM users Monospace font for the database query statement
db.type mysql Database type used to determine appropriate syntax highlighting of the db.statement
ref.traceId 147dd7529b757eed A link to the referenced trace

How span tags differ from SignalFx tags 🔗

Tags have another meaning in SignalFx, and therefore span tags should not be confused with SignalFx tags. Span tags are free-form key-value pairs that can be attached to spans during instrumentation. Once the span is captured, they are immutable: the span, and its span tags, are not editable.

SignalFx tags, on the other hand, are labels that can be attached to a variety of SignalFx objects, in particular dimensions, metrics, or metric time series. SignalFx tags may also carry additional metadata properties. They are editable, and can be added or removed from other SignalFx objects at any time. For more information, see this section of the Gateway reference documentation.

Limits on span metadata 🔗

There is no limit to the number of span tags or span annotations attached to each span. There is however a limit to the total cummulative size of this metadata: the total length of all tag keys, tag values, and span annotations may not exceed 16kB.