Docs » Using the µAPM User Interface » The View a Trace Page (Drilling Down)

The View a Trace Page (Drilling Down) 🔗

The View a Trace page lets you view all spans in a trace and view status information and details about each individual span.

Displaying a single trace 🔗

To view a single trace and all its spans, you can do one of the following:

  • If you know the trace ID you want to view, you can hover over µAPM on the navigation bar, select View a Trace, and then enter the trace ID.

  • More commonly, use the Traces page to select a trace to view. Hover over µAPM on the navigation bar and select Traces. From here, you have a couple options:

    • Expand a service in the list at right, then click on a trace ID.
    • Expand the Outlier Analyzer, click Top Tags or Top Operations to filter the Traces page, then expand a service and click on a trace ID.

The traces list shows the root (initiating) span at the top and all child spans underneath. Scroll to zoom in and out; click and drag to pan through the spans. (For more information on navigating through a trace, see Panning and zooming in the View a Trace page.)

The following illustration shows an individual trace in a zoomed-out view.

View a Trace page with legend

The following illustration shows a zoomed-in view of the trace.

zoomed in view of a trace

The sidebar to the right contains metadata about the trace itself, a summary of the services involved in this transaction, and a map of those services. The Top Contributors section offers insights into which services or which operations are the main drivers of latency in this trace based on how much time they contribute to the overall trace. This can help you quickly identify a problematic service or expensive operation in the path of that transaction. The Top Contributors section decomposes the duration of a trace amongst its constituent spans. A given span is deemed responsible for portions of its duration when none of its descendants are active; when multiple incomparable spans are active during the same period, that duration is divided equally among them. The results are then grouped by span.

Example 1 🔗

The get-data span is active for the first 50ms of the trace and the process-data span is active during the second 50ms. Accordingly, they each contribute 50ms to the latency of this trace.

Top Contributors example 1

Example 2 🔗

The get_answer endpoint is active during the entire trace duration, but it is not considered a contributor since a descendant span is always active. Its subtree has the same structure as the previous example, and so the contribution calculation is the same; the retrieve_data span is responsible for the first 50ms of the trace and process_data is responsible for the second 50ms.

Top Contributors example 2

Example 3 🔗

This example is similar to the previous example, except a verify span has been prepended and is active during exactly the 10ms preceding the rest of the trace. The attribution among get_answer , retrieve_data, and process_data is the same and verify contributes the remaining 10ms.

Top Contributors example 3

Example 4 🔗

In this example, the get_answer endpoint is active during the entire 100ms duration of the trace. During the first 30ms, its descendant retrieve_data is active, and so retrieve_data contributes 30ms to the overall latency. No descendent is active during the next 20ms, so the root span get_answer contributes that 20ms of latency. During the final 50ms of the trace, both process_data and get_params are active, and both are descendants of retrieve_data. Therefore process_data and get_params both contribute half of the final 50ms of latency, i.e., they each contribute 25ms.

Top Contributors example 4

Example 5 🔗

This example has the same tree structure as the previous example, but the retrieve_data span has been replaced by another instance of the process_data span. Therefore, the first 30ms is attributed to the process_data operation. Since the results are aggregated by service and operation, process_data contributes 55ms of overall latency (the first 30ms and half of the final 50ms). The contributions of get_answer and get_params are the same as in the previous example.

Top Contributors example 5

Panning and zooming in the View a Trace page 🔗

One of the main goals of the View a Trace page is to guide you to the interesting sections of the trace as quickly as possible. This is why when a trace is loaded, you start with a “zoomed out” view of the trace’s structure in the main area.

zoomed out view of a trace

If the trace doesn’t have a lot of spans and fits in the view, it is automatically zoomed to show all its spans when you open the View a Trace page.

view of smaller trace

When you are scrolling to zoom in on a large span, you may “get lost” and not be sure where you are in the tree. To display a more readable view, click the name of the initiating span at right.

if you are lost in the trace

You now see the trace’s spans at your current zoomed-in level, with the initiating span at the top. Details about the span are shown at right. From here, you can click and drag to pan through the spans in the trace.

view initiating span

Span labels appear when you reach maximum zoom, showing the name of the service and the name of the operation that was captured by each span. Parent/child relationships between spans are represented by the lines connecting the spans.

Each sub-tree can be individually collapsed or expanded by clicking the caret to the left of a span with children. You can also use the View Options dropdown menu to expand all, collapse all, or selectively hide certain traces. For example, you can collapse all the non-RPC spans, leaving only the inter-service interactions shown.

To color the spans, select an option in the Color by dropdown menu:

../../_images/color-by.png
  • Service (Default) — Spans are colored according to their service. This allows you to visually tell which services are involved as well what proportion of the trace each service takes.

  • % of Duration — Spans are colored based on how much of the total trace time they consume.

    • Dark green if span duration < 10%
    • Light green if 10% <= span duration < 30%
    • Yellow if 30% <= span duration < 50%
    • Orange if 50% <= span duration < 70%
    • Red if 70% <= span duration
  • Duration Against Historical — Each span is compared to the p90 and p99 metric values over the previous day’s EWMA (exponentially weighted moving average) value. Each span is colored according to the following:

    • Green if span duration <= p90
    • Yellow if p90 < span duration <= p99
    • Red if p99 <= span duration

Viewing span information 🔗

Hovering over a span displays a tooltip with a quick summary of the span.

contents of span tooltip

To view the full details of a span’s metadata, select a span by clicking on it; this opens a new section for span metadata in the sidebar. All the span’s metadata tags and annotations are displayed there.

span information in sidebar

Each span includes the following:

  • An operation name
  • A start timestamp
  • A finish timestamp
  • A set of zero or more user-defined key:value Span metadata tags. The keys must be strings; the values may be strings, bools, or numeric types.

Note that there are semantic naming conventions for the user-defined Span metadata tags. To ensure that the metadata is correctly parsed, we recommend that you use the standard naming conventions. These conventions allow µAPM to recognize the metadata tags and to leverage SignalFx’s advanced metrics and analytics capabilities. For more information, see OpenTracing semantic conventions.

In the following illustration, we’ve scrolled down to display some charts below the span metadata information.

span charts in sidebar

You can deselect a span by clicking on it again, or by selecting another span.

Selecting an intra-service span 🔗

Illustrates the metadata tags that you would typically see with an intra-service span.

intra-service span sidebar information

Selecting an inter-service HTTP call span 🔗

Illustrates the metadata tags that you would typically see with an inter-service HTTP call.

inter-service http call span sidebar information

Selecting a 3rd party cloud service span 🔗

For 3rd-party cloud services (e.g. salesforce.com), refer to the previous inter-service HTTP call span example. You may see a hostname instead of an internal server in the span metadata, such as:

http.url   www.salesforce.com/get/data?id=1234

peer.address and peer.hostname are other possible tag names that you might see instead of http.url, where peer.address would have a similar value to http.url (a full URL) and peer.hostname would just be the hostname.

Selecting an inferred service span 🔗

Illustrates a span from the signalboost service in which we inferred elasticsearch:elasticsearch-traces.

inferred service span sidebar information

Selecting a database call span 🔗

Illustrates the metadata tags that you would typically see with a database query.

database call span sidebar information

Selecting a queue span 🔗

Illustrates a span from sbingest to kafka-traces:trace_spans_firehose to spanmonger.

queue span sidebar information