Index span tags to generate Troubleshooting MetricSets 🔗
Thanks to Splunk APM’s full-fidelity tracing, which captures every span from every trace, you can use span tags to break down services and inter-service calls along any characteristic or attribute associated with any given trace. This lets you customize data visualizations and metrics for your monitoring and troubleshooting requirements.
To get additional value from a particular span tag, a Splunk APM administrator can perform an action known as indexing, which enables additional automatic analysis of the indexed span tag. One benefit of this action is to get aggregated metrics, or MetricSets, across all spans that contain a specified indexed tag.
What are MetricSets? 🔗
MetricSets are metric time series you can use to track the performance of specific metrics over time. To help you analyze your service performance, APM provides two kinds of MetricSets: Troubleshooting MetricSets and Monitoring MetricSets. Troubleshooting MetricSets are used for high-cardinality troubleshooting, while Monitoring MetricSets leverage the Infrastructure Monitoring platform for real-time monitoring and alerting.
APM provides a collection of MetricSets by default, and you can index additional span tags to generate additional Troubleshooting MetricSets, and optionally add additional custom dimensionalized Monitoring MetricSets. See Learn about MetricSets in APM for an overview of the types of MetricSets in APM and a list of the MetricSets you receive by default.
Use this topic to learn how to index a span tag and generate additional Troubleshooting MetricSets. Then see Generate a Monitoring MetricSet with a custom dimension to learn how to add custom dimensionalized Monitoring MetricSets.
Prerequisites for indexing new span tags to generate Troubleshooting MetricSets 🔗
You need to be an administrator in your organization to index tags and create Troubleshooting MetricSets.
(Optional) Before you start indexing span tags, see Reference and best practices for span tags and Troubleshooting MetricSets for guidance on choosing span tags to index.
Overview: Troubleshooting MetricSets 🔗
Every Troubleshooting MetricSet generates the following metrics, also known as Request, Error, and Duration (RED) metrics:
Root cause error rate
p50, p90, and p99 latency
These metrics appear when you select a service from the service map in the Troubleshooting view.
The measurement precision of Troubleshooting MetricSets is 10 seconds. Splunk APM reports quantiles from a distribution of metrics for each 10-second reporting window.
Automatically indexed span tags 🔗
Splunk APM automatically indexes and generates Troubleshooting MetricSets for the following six span tags:
For more details about each of these tags, see Span tags APM automatically indexes. You can’t modify or stop APM from indexing these span tags, but you can choose to index additional span tags. Scroll to Index a new span tag in this topic to learn how.
Cardinality contribution of indexed span tags 🔗
When you index a new span tag to generate Troubleshooting MetricSets, Splunk APM runs a cardinality contribution analysis to calculate the potential total cardinality contribution after indexing the span tag. This gives you control of what you index and helps account for any limits you have to stay within. If you try to index a span tag that could increase the total cardinality contribution beyond your limit, you can change the existing cardinality contribution of indexed tags by modifying or removing indexed span tags.
Index a new span tag 🔗
Follow these steps to index a span tag and create a Troubleshooting MetricSet for it.
To open the APM MetricSets page, do one of the following:
In Splunk APM, select APM Configuration and choose APM MetricSets from the drop-down menu. The APM Troubleshooting MetricSets Configuration page opens.
From anywhere in Splunk Observability Cloud, select Settings (gear icon) in the left navigation bar and select APM MetricSets under Data Configuration.
On the APM MetricSets page, select New MetricSet.
Enter the Name of a span tag you want to index.
The Scope determines how APM associates the span tag with services in a trace:
Select Service to associate the span tag with services. This means the value of the span tag could change across services in a given trace. Specify
All Servicesto index the span tag for every service. Select specific services to index the span tag for only those services.
Select Global to associate the span tag with traces. This means the value of the span tag is the same for all services in a given trace.
For more information about span tag scope, see Types of span tags you can index.
(Optional) If your span tag is indexed at the service level, you can also use it as a custom dimension in Monitoring MetricSets. See Generate a Monitoring MetricSet with a custom dimension for instructions.
Select Start Analysis to submit the configuration. When you submit a configuration, Splunk APM runs an analysis of the span tag to calculate the potential cardinality contribution of indexing it and determine whether it would generate Troubleshooting MetricSets that exceed your limit.
Wait a few moments for the cardinality check to run, and then check under Pending MetricSet to view the status of the span tag you’re trying to index. See the table below for possible status options for pending MetricSets and the recommended action for each status.
The application is currently running the cardinality contribution analysis. When this is the status for a span tag you want to index, you can’t create or modify any other span tags.
Wait until the cardinality contribution analysis is complete.
The cardinality contribution analysis is complete: you can index the span tag without any issue.
Select the checkmark in the Actions column to manually index the span tag and start generating Troubleshooting MetricSets that include the span tag.
The cardinality contribution analysis is complete, but you can’t index the span tag because you reached an entitlement or system limit.
Consider pausing or deleting existing Custom MetricSets to open space for another indexed span tag, or reach out to your Splunk Observability Cloud account manage to increase your account limit. See Limits on indexing span tags to learn more about these limits.
If more than one hour passes for a pending MetricSet in a
Readystatus, the status changes to
Rerun the analysis to try indexing the span tag again.
Manage existing indexed span tags and Troubleshooting MetricSets 🔗
After you successfully index a span tag, Splunk APM saves the configuration in the MetricSets Configuration page in your Organization Settings. Visit this page to view the indexing scope of the span tag and its current status.
You can modify the configuration for existing indexed tags, including adding and removing services for specific indexed tags and modifying the scope. You can also pause or stop indexing span tags without deleting their configuration. This is useful when you want to temporarily stop indexing a span tag, but don’t want to remove the configuration.
To review or modify existing indexed span tags, do the following:
Go to Settings > Organization Settings > MetricSets Configuration
Find the indexed span tag you want to view under the Custom MetricSets section of the configuration table.
Refer to Status of configured Troubleshooting MetricSets below to interpret the status of each indexed span tag.
Make any desired changes using the buttons in the Actions column:
Use the pencil button to edit the scope of an indexed span tag.
Use the pause button to pause generating MetricSets for a given span tag.
Use the trash button to delete a MetricSet configuration.
Status of configured Troubleshooting MetricSets 🔗
Once configured, custom Troubleshooting MetricSets can have the statuses listed in the following table:
The application is indexing the span tag and generating Troubleshooting MetricSets for it.
You or another administrator paused indexing for the span tag. The application isn’t generating Troubleshooting MetricSets for the span tag, and you can’t view any data you previously indexed for it.
The application stopped indexing the span tag and is no longer generating Troubleshooting MetricSets for the span tag. You can’t view any data you previously indexed for this span tag.