Docs » Use cases: Troubleshoot errors and monitor application performance using Splunk APM » Monitor detector service latency for a group of customers

Monitor detector service latency for a group of customers 🔗

Kai, a site reliability engineer at the fictitious Buttercup Games, wants to monitor a latency issue affecting a critical checkout workflow for the cartservice service and /getcart endpoint for a specific set of customers who most frequently have problems with the service. By generating a Monitoring MetricSet with version_id as a custom dimension and filtering it to the customers affected by the issue, Kai can set up a detector to monitor service and endpoint latency by customer. Kai can also create charts and dashboards that show service and endpoint latency for specific customers over time.

Generate a Monitoring MetricSet and filter by span tag 🔗

To generate Monitoring MetricSets by customer:

  1. Kai indexes a span tag to identify each customer: version_id. See Index span tags to generate Troubleshooting MetricSets for the steps Kai takes to generate a custom MMS.

  2. Kai generates a Monitoring MetricSet using version_id as a dimension. See Generate a Monitoring MetricSet with a custom dimension for the steps Kai takes to generate a custom MMS.

  3. Kai sets the scope of the Monitoring MetricSet to the cartservice service, and filters on the tag values for version_id that represent the specific list of customers Kai wants to investigate.

This image shows a sample Monitoring MetricSet configuration with the service selected as cartservice, endpoint as /getcart and filtering by tag values for version_id:

This screenshot shows how to add a custom Monitoring MetricSet for a single service.


Create service latency detectors to track metrics 🔗

Kai can leverage the custom dimensionalized Monitoring MetricSet Kai created to monitor the performance of this critical checkout workflow in the cartservice service. To do this, Kai creates a detector using the same custom indexed tag, version_id, to track error rates associated with the checkout workflow.

  1. There are two ways for Kai to navigate to the Detector Creation page to create a detector using their custom Monitoring MetricSet:

    1. From the left navigation panel, select: Alerts & Detectors > New Detector.

    2. From any APM page, select the plus sign ( + ) on the top bar and select Detector from the drop-down list .

  2. Kai follows the guided Detector Creation flow to create their detector based on the error rate in the service cartservice:GetCart, filtered to the custom dimension of version_id.

  3. Kai can use the Metric Finder to find additional information on the metrics and metadata for their system. For more, see Metric Finder and Metadata Catalog. Apply sf_dimensionalized:true as a Filter to see related metrics as shown in the image below.

This screenshot shows how to filter the MetricFinder for metrics related to custom monitoring MetricSets.

Set up charts, dashboards, and alerts for custom dimensions 🔗

Kai can also create charts and dashboards that leverage the custom dimensions Kai created.

Kai navigates to the built-in APM service endpoint dashboard for cartservice:GetCart using the following steps:

  1. From the APM landing page, select the Dashboards icon in the left navigation bar.

  2. Search APM in the search bar.

  3. Under APM Services in the Built-in Dashboard Groups section, click Service Endpoint.

  4. Select the relevant environment, then select cartservice as the Service, GetCart as the Endpoint, and apply sf_dimensionalized:true as a Filter.

  1. To see the custom dimensions Kai created, Kai applies the filter sf_dimensionalized:true in the filter bar while creating their dashboard, as the following image illustrates:

This screenshot shows how to filter the MetricFinder for metrics related to custom monitoring MetricSets.

See also 🔗

Use the links in the following table to learn more about charts, dashboards, and alerts.

Task

Instructions

Create charts

Create charts in Splunk Observability Cloud

Create dashboards

Create and customize dashboards

Create an alert

Configure detectors and alerts in Splunk APM

Monitor services in APM dashboards

Track service performance using dashboards in Splunk APM

Summary 🔗

Kai created a Monitoring MetricSet with a custom dimension to track a checkout workflow for a subset of customers. To learn more about Monitoring MetricSets, see Generate a Monitoring MetricSet with a custom dimension.