Docs » Scenarios for troubleshooting errors and monitoring application performance using Splunk APM » Scenario: Deepu finds the root cause of an error using Tag Spotlight

Scenario: Deepu finds the root cause of an error using Tag Spotlight 🔗

Deepu receives a notification from Kai, the site reliability engineer. The notification says that the high root cause error rate with the /PaymentService/Charge endpoint is preventing customers from shopping on the Buttercup Games website. The notification also includes a link to the endpoint on the Splunk APM service map.

To troubleshoot the issue, Deepu takes the following steps:

  1. Deepu selects the affected service on the service map and opens Tag Spotlight

  2. Deepu reviews Tag Spotlight for tags that are correlated with errors

  3. Deepu notices errors differ based on the version tag

  4. Deepu opens an example trace

  5. Deepu switches to Splunk Log Observer for further troubleshooting

Deepu selects the affected service on the service map and opens Tag Spotlight 🔗

Deepu selects the paymentservice node on the service map and selects the Tag Spotlight view on the sidebar.

This screenshot shows the payment service in the service map and Tag Spotlight side panel.

Deepu reviews Tag Spotlight for tags that are correlated with errors 🔗

Deepu scans through the requests and errors correlated with each indexed tag in the payment service.

This screenshot shows the Tag Spotlight view of the payment service.

Deepu notices errors differ based on the version tag 🔗

Deepu sees that the errors are evenly distributed for all tag values except the version tag. All errors occur in version 350.10, a recent code release for the service. Deepu rolls back to the previous release, version 350.9, to keep the site running while notifying and waiting for the engineers to solve the issue.

This screenshot shows the data based in the indexed version tag in Tag Spotlight.

Deepu opens an example trace 🔗

Deepu narrows the investigation to the code in version 350.10 of the /PaymentService/Charge endpoint and selects the Request/Errors chart to get an example trace to see what the error is.

This screenshot shows an example trace with errors in Tag Spotlight.

Deepu switches to Splunk Log Observer for further troubleshooting 🔗

Because Deepu turned on Related Content in Splunk APM, Deepu can select Logs for trace to switch to Splunk Log Observer for further troubleshooting.

This screenshot shows a trace view with the option to review the logs for the trace.

Summary 🔗

Deepu used Tag Spotlight to quickly isolate a recent code release as the cause of errors that prevented customers from shopping on the Buttercup Games website. Using the links to example traces in Tag Spotlight, they quickly navigate to an example trace and its corresponding logs to further troubleshoot.

Learn more 🔗