Docs » The Traces Page

The Traces Page 🔗

Back to User Interface contents

The Traces page lets you “slice and dice” your trace data to view traces for specific services, endpoints, tags, and duration. The Outlier Analyzer makes it easy to quickly filter your traces down to those that are most likely contributing to any problems.

Viewing traces 🔗

To see all the most recent traces, hover over µAPM PG on the navigation bar and select Traces. (On smaller screens, you may need to scroll down to see all the information below the service map.)


You can also get to the Traces page from other locations, with a context based on where you are starting.

  • From the Services page, click a service icon in the map or expand a service in the table, then select View traces from the navigation bar at the top of the service map. The Traces page will be filtered by the specified service.

  • From the Actions menu on a chart, click View traces from this time window. The Traces page will have the same time range as the chart.

  • When viewing an alert message, click View more details if necessary, then click View traces from this time window. The Traces page will open with a time range that encompasses the time when the alert was triggered with the same cluster, service, and endpoint carried forward. If the alert is for an APM error rate detector, the trace filter will default to showing only traces containing error spans for the alerted endpoint.



Which traces are retained by µAPM PG 🔗

While we retain metrics for every trace and span sent to SignalFx, the Smart Gateway determines which traces are shown in the Traces page through the use of our NoSample™ Tail-Based Distributed Tracing mechanism, which prefers interesting or anomalous traces over “normal” traces. Three primary factors influence the likelihood of a trace being kept:

  • its duration (how slow it is compared to traces for the same execution path)
  • whether it contains errors
  • the frequency of execution

The amount of time allowed for a trace to complete (at which point the decision whether to retain the trace is made) is based on the recent history of durations of similar traces.

Learn more about our NoSample™ Tail-Based Distributed Tracing mechanism here.

Filtering the Traces page 🔗

The Traces page provides a set of filter controls to narrow down the view of traces to a more specific set. By default, the Traces page shows traces from the last 15 minutes, up to 10,000 traces. You can adjust the time window to look at traces in any time frame within the past 8 days. You can also specify compound criteria on traces and spans to more easily find traces that are representative of an issue you are investigating. For example, you can choose to see only traces that go through a particular service, endpoint, or operation. For this, from the Service: Endpoint/Operation filter, you first choose a service, then a specific endpoint or operation that is part of the service. For example, specifying the sbingest service and the endpoint /v2/event narrows down the view to traces that go through both of these, as expected.


After a filter is chosen, you can click on the Trace icon beside the pill to specify span criteria, such as duration and tags. Select the appropriate radio button to find traces containing spans matching anywhere in the trace, or traces where the span is the initiating operation.


For example, by selecting the first radio button, all the traces containing an sbingest: /v2/events span will be reported. However, if you select the second radio button, then only if sbingest: /v2/events is the initiating operation will the trace be reported.

Use the Status dropdown to further filter for error, non-error, or all occurrences of a specific Service: Endpoint/Operation span.


Enter a range for trace durations, such as, 2h, 30mins, 500ms, >1m, <30s, 50ms to 5s, etc.


Specify tags to further filter for occurrences of a specific Service: Endpoint/Operation span.


Specify one or multiple tags in the Tags field in span criteria or trace criteria. Adding different values of the same tag would OR the terms together, similar to how filters operate in charts and dashboards. For example, the options shown below will include traces where the value of the host tag is either i00a45a70fa995f577 or i01860744ba0e6058d.


Lastly, you can choose to find all traces that match all of the span filters, or find all traces where any one of the span filters matches the trace.


Using the Latency Distribution chart 🔗

If you click on a bucket or click and drag across an area in the Latency Distribution chart, the page is filtered to display only traces with durations in the same time range as the selection. In this case, we clicked in the area shown below because we wanted to see more details about some of the traces that were above the P99 value.


The traces shown at right are narrowed down to include only those with the specified duration. We can now click one of the traces at right to see it in the View a trace page.


Using the Errors/Requests chart 🔗

If you click on a bucket, or click and drag across an area in the Errors/Request chart, the time range becomes shorter and you can see more information on individual traces. In this case, we clicked in the area shown below because we wanted to see more details about the error shown.


The chart’s time range is narrowed down from the original 15 minutes to several seconds. Depending on the starting time range, you may have to click in the chart more than once to narrow the view down to a single trace.


Clicking on the trace error filters the page to that individual trace. We can now click on the error trace at right to see it in the View a trace page.


Analyzing outliers 🔗

Available in SignalFx Enterprise Edition.

The Outlier Analyzer makes it easy to quickly filter your traces down to those that are most likely contributing to any problems. When you view your trace data, Top Tags and Top Operations are displayed above the list of traces on the right.

  • The Top Tags section displays the tags (key-value pairs) that are more common in traces having the longest durations (above the 90th percentile). The tags at the top appear statistically more often in the long traces than they do in the overall trace set. Hovering over a tag shows where traces with that tag appear in the Latency Distribution chart.
  • The Top Operations section aggregates the durations of the operations, according to the Top Contributors analysis, across the longest traces in the data set (the 100 longest traces or traces above the 90th percentile, whichever is smaller). The list is sorted by % of duration.

Open or close each section individually, or click Expand/Collapse all to expand/collapse both Top Tags and Top Operations simultaneously. Note that Top Tags will load immediately, whereas Top Operations may display a windmill spinner as it loads.


Troubleshooting with Outlier Analyzer 🔗

As you can see in the illustration below, the signalboost operation takes over 62% of the total duration.


Clicking on that operation filters the view by service and operation. In this case, at least one trace for that service and operation has an error.


Expand the view and scroll if necessary until you see the trace with the error. (In a different scenario, there may not be an error, but you could scroll through the traces to find the one with the longest duration.)


Click the trace to analyze it in the View a trace page.


Managing Top Tags 🔗

As you hover over any row in the Top Tags table, an Actions menu icon is displayed at the end of the row. Menu options let you remove certain tags from the results (temporarily or permanently), so that more relevant tags might have a chance to surface.


Select Hide this row to hide a specific row of the Top Tags results.


Select Hide all rows with tag to hide all rows with a given tag name from the Top Tags results, regardless of value.


Select Exclude <tag> from all analyses to hide a tag name from all of the Outlier Analyzer results. This tag is saved on a per-user, per-organization basis so that no future execution of the Outlier Analyzer will include this result.


The exclusions summary is displayed at the bottom of the Top Tags table.


Select Manage Exclusions from the Actions menu (or click Manage in the exclusions summary) to display a list of both local/temporary and global/permanent exclusions.


You can remove or add exclusions from the Manage Exclusions dialog.


Clicking the “+” or Add Exclusion button brings up a filter autocomplete. You are not required to specify a value. If you don’t want to specify a value, select the Any value checkbox or hit Return. The pill will resolve as name:*.


You can also select any value from the dropdown list.


For permanent exclusions (that is, no future execution of the Outlier Analyzer will include this result), it is only possible to exclude the entire tag by name, not any specific value for that tag. Select the Top Tags row that you want to exclude, then click Apply selected filter.


Back to User Interface contents