Docs » Using the µAPM User Interface » The View a Trace Page (Drilling Down)

The View a Trace Page (Drilling Down)

The View a Trace page lets you view all spans in a trace and view status information and details about each individual span.

Displaying a single trace

To view a single trace and all its spans, you can do one of the following:

  • If you know the trace ID you want to view, you can hover over µAPM on the navigation bar, select View a Trace, and then enter the trace ID.

  • More commonly, use the Traces page to select a trace to view. Hover over µAPM on the navigation bar and select Traces. From here, you have a couple options:

    • Expand a service in the list at right, then click on a trace ID.
    • Run the Outlier Analyzer, click Top Operations or Top Tags to filter the Traces page, then expand a service and click on a trace ID.

The traces list shows the root (initiating) span at the top and all child spans underneath. Scroll to zoom in and out; click and drag to pan through the spans. (For more information on navigating through a trace, see Panning and zooming in the View a Trace page.)

The following illustration shows an individual trace in a zoomed-out view.

View a Trace page with legend

The following illustration shows a zoomed-in view of the trace.

zoomed in view of a trace

The sidebar to the right contains metadata about the trace itself, a summary of the services involved in this transaction and a map of those services. The Top Contributors section offers insights into which service(s), or which operation(s), are the main drivers of latency in this trace based on how much time they contribute to the overall trace. This can help you quickly identify a problematic service or expensive operation in the path of that transaction. Top Contributors decompose the duration of a trace amongst its constituent spans. A given span is deemed responsible for portions of its duration when none of its descendants are active; when multiple incomparable spans are active during the same period, that duration is divided equally among them. The results are then aggregated by span.

Example 1

The get-data span is active for the first 50ms of the trace and the process-data span is active during the second 50ms. Accordingly, they each contribute 50ms to the latency of this trace.

Top Contributors example 1

Example 2

The get_answer endpoint is active during the entire trace duration, but it is not considered a contributor since a descendant span is always active. Its subtree has the same structure as the previous example, and so the contribution calculation is the same; the retrieve_data span is responsible for the first 50ms of the trace and process_data is responsible for the second 50ms.

Top Contributors example 2

Example 3

This example is similar to the previous example, except a verify span has been prepended and is active during exactly the 10ms preceding the rest of the trace. The attribution among get_answer , retrieve_data, and process_data is the same and verify contributes the remaining 10ms.

Top Contributors example 3

Example 4

In this example, the get_answer endpoint is active during the entire 100ms duration of the trace. During the first 30ms, its descendant retrieve_data is active, and so retrieve_data contributes 30ms to the overall latency. No descendent is active during the next 20ms, so the root span get_answer contributes that 20ms of latency. During the final 50ms of the trace, both process_data and get_params are active, and both are descendants of retrieve_data. Therefore process_data and get_params both contribute half of the final 50ms of latency, i.e., they each contribute 25ms.

Top Contributors example 4

Example 5

This example has the same tree structure as the previous example, but the retrieve_data span has been replaced by another instance of the process_data span. Therefore, the first 30ms is attributed to the process_data operation. Since the results are aggregated by service and operation, process_data contributes 55ms of overall latency (the first 30ms and half of the final 50ms). The contributions of get_answer and get_params are the same as in the previous example.

Top Contributors example 5

Panning and zooming in the View a Trace page

One of the main goals of the View a Trace page is to guide you to the interesting sections of the trace as quickly as possible. This is why when a trace is loaded, you start with a “zoomed out” view of the trace’s structure in the main area.

zoomed out view of a trace

If the trace doesn’t have a lot of spans and fits in the view, it is automatically zoomed to show all its spans when you open the View a Trace page.

view of smaller trace

When you are scrolling to zoom in on a large span, you may “get lost” and not be sure where you are in the tree. To display a more readable view, click the name of the initiating span at right.

if you are lost in the trace

You now see the trace’s spans at your current zoomed-in level, with the initiating span at the top. Details about the span are shown at right. From here, you can click and drag to pan through the spans in the trace.

view initiating span

Span labels appear when you reach maximum zoom, showing the name of the service and the name of the operation that was captured by each span. Parent/child relationships between spans are represented by the lines connecting the spans.

Each sub-tree can be individually collapsed or expanded by clicking the caret to the left of a span with children. You can also use the View Options dropdown to expand all, collapse all, or selectively hide certain traces. For example, you can collapse all the non-RPC spans, leaving only the inter-service interactions shown.

Viewing span information

Hovering over a span displays a tooltip with a quick summary of the span.

contents of span tooltip

To view the full details of a span’s metadata, select a span by clicking on it; this opens a new section for span metadata in the sidebar. All the span’s metadata tags and annotations are displayed there.

span information in sidebar

Each span includes the following:

  • An operation name
  • A start timestamp
  • A finish timestamp
  • A set of zero or more user-defined key:value Span metadata tags. The keys must be strings; the values may be strings, bools, or numeric types.

Note that there are semantic naming conventions for the user-defined Span metadata tags. To ensure that the metadata is correctly parsed, we recommend that you use the standard naming conventions. These conventions allow µAPM to recognize the metadata tags and to leverage SignalFx’s advanced metrics and analytics capabilities. For more information, see OpenTracing semantic conventions.

In the following illustration, we’ve scrolled down to display some charts below the span metadata information.

span charts in sidebar

You can deselect a span by clicking on it again, or by selecting another span.

Selecting an intra-service span

Illustrates the metadata tags that you would typically see with an intra-service span.

intra-service span sidebar information

Selecting an inter-service HTTP call span

Illustrates the metadata tags that you would typically see with an inter-service HTTP call.

inter-service http call span sidebar information

Selecting a 3rd party cloud service span

For 3rd-party cloud services (e.g., refer to the previous inter-service HTTP call span example. You may see a hostname instead of an internal server in the span metadata, such as:


peer.address and peer.hostname are other possible tag names that you might see instead of http.url, where peer.address would have a similar value to http.url (a full URL) and peer.hostname would just be the hostname.

Selecting a database call span

Illustrates the metadata tags that you would typically see with a database query.

database call span sidebar information

Selecting a queue span

Illustrates a span from sbingest to kafka-traces:trace_spans_firehose to spanmonger.

queue span sidebar information