Docs » Drill down into service performance with indexed tags and MetricSets in SignalFx µAPM » View indexed tags for services with Tag Spotlight

View indexed tags for services with Tag Spotlight 🔗

Important

The original µAPM product, released in 2019, is now called µAPM Previous Generation (µAPM PG). In the documentation, µAPM now refers to the product released on March 31, 2020.

If you are using µAPM Previous Generation (µAPM PG), see µAPM PG Traces, Spans, Metrics, and Metadata.

Use Tag Spotlight to analyze the performance of your services to discover trends that contribute to high latency or error rates with indexed span tags. You can break down every indexed span tag for a particular service to view metrics for it. When you select specific span tag values or a specific time range, you can view representative traces to learn more about an outlying incident.

For every service, Tag Spotlight provides a RED metrics time-series chart that displays the total number of requests, errors, root-cause errors, and latency according to the specified time range in the APM navigation menu. Along with the RED metrics chart, Tag Spotlight also displays the total number of requests, errors, root-cause errors, and latency for every value of an indexed span tag according to the specified time range in the APM navigation menu.

SignalFx uses Troubleshooting MetricSets to display indexed span tag performance for a service. For every indexed span tag value, view metrics for request rate, error, root-cause error rates, and p50, p90, and p99 latency. For more information about Troubleshooting MetricSets, see Drill down into service performance with indexed tags and MetricSets in SignalFx µAPM.

View service performance by indexed span tags with Tag Spotlight 🔗

Access Tag Spotlight from the Troubleshooting tab of the µAPM page to analyze the performance of every indexed span tag value for a service and break down performance for each tag by either requests and errors or latency. Follow these steps to go to Tag Spotlight for a service:

  1. In SignalFx, go to the µAPM page.

  2. Select a service you want to drill down into.

  3. From the filter bar for the service, click Troubleshoot.

  4. Click Spotlight from the service menu bar. You can view the analysis for requests and errors or latency. You can also click Tag Spotlight in the Requests and Errors service card.

  5. View the distribution of all indexed span tags:

    ../../_images/apm-tag-spotlight-errors.png

    The RED metrics time-series chart displays requests, errors, root-cause errors, or latency for the specified time range. Span tag values are available in cards for each indexed span tag. in The default time range is for the last 15 minutes, and the data resolution is 10 seconds.

  6. Click the time-series chart to view representative traces for the selected point.

Example: Find the root cause of an incident with Tag Spotlight 🔗

A service yourService is generating a lot of errors. Follow these steps to learn how you can pinpoint the root cause of an incident with Tag Spotlight.

For this example, you index span tags representing these things:

  • Kubernetes pod name
  • tenant level

This example also uses the Operation span tag, but this is indexed by default.

  1. In SignalFx, go to the µAPM page.
  2. From the Troubleshooting tab, select a service you want to drill down into.
  3. Click Tag Spotlight in the Requests and Errors service card.
  4. Using the RED metrics chart, click and drag the cursor where there’s a spike in errors to view data for only the incident you’re investigating.
  5. In the Operation span tag card, you see that some operation yourOperation has a lot of errors.
  6. Hover over the operation to quickly see RED metrics for the operation.
  7. To drill down further into the performance of the operation, click the yourOperation value in the span tag card. This shows you information about all indexed tags for only requests that include myOperation.
  8. There are multiple tenant values, but you see that users who belong to a single tenant are experiencing the vast majority of errors with the service.
  9. You also see that a particular Kubernetes pod has an error spike that corresponds to the errors the operation is generating.
  10. You infer that the incident is due to an operation running in a specific Kubernetes pod that affects people associated with a particular tenant.
  11. From the RED metrics chart, click the peak error rate to view an exemplar trace for the incident.