Docs » Analyze services with span tags and MetricSets in Splunk APM » Generate a Monitoring MetricSet with a custom dimension

Generate a Monitoring MetricSet with a custom dimension 🔗

Generate Monitoring MetricSets with custom dimensions to glean meaningful insights about your services in real time. Splunk APM automatically provides a set of Monitoring MetricSets by default, but you can configure additional MetricSets using indexed span tags as custom dimensions. You can use your Monitoring MetricSets to create custom charts, dashboards, and detectors to keep track of what matters most to your team.

Prerequisites 🔗

How custom Monitoring MetricSets add value to your organization 🔗

APM generates a collection of Monitoring MetricSets by default; see Metrics and dimensions of Monitoring MetricSets for that list. By creating Monitoring MetricSets with custom dimensions, you can filter and aggregate the generated metrics by a specific indexed span tag such as customer.id, version, or cloud.provider.

You can create custom Monitoring MetricSets at the service level and optionally at the endpoint (span) level. When you create a custom dimension for a service-level MMS, APM generates MMS that include the 6 service-level metrics listed in Service metrics and dimensions, with your chosen indexed span tag as a custom dimension. If you choose to add endpoint-level metrics as well, APM generates MMS that include the 6 span-level metrics listed in Span (endpoint-level) metrics and dimensions, with your chosen indexed span tag as a custom dimension.

When you create a Monitoring MetricSet with a custom dimension, you can use this custom dimension to create charts, dashboards, and alerts by leveraging the Infrastructure Monitoring platform.

To learn more about a specific use case for custom Monitoring MetricSets: Monitor SLO detector service latency for a group of customers.

How to generate a Monitoring MetricSet with a custom dimension 🔗

When you generate a Monitoring MetricSet, the MetricSet configuration moves to the pending state where APM reviews the cardinality of your data. Before you generate a Monitoring MetricSet, first consider:

  • How you want to allocate the cardinality of your data to make your Monitoring MetricSets most efficient.

  • If the cardinality check fails, the new MetricSet exceeds the cardinality limit for your organization. See Troubleshoot cardinality in Monitoring MetricSets for guidance on editing your MetricSet configuration to optimize your overall cardinality.

  • If the cardinality of your MetricSet is within your limits then you can enable the MetricSet.

Choose how you want to configure your Monitoring MetricSet 🔗

There are three options for the scope of the Monitoring MetricSet you generate, including generating a service-level MMS only, generating endpoint-level MMS for specific endpoints within a given service, or generating both service-level MMS and endpoint-level MMS for all endpoints within a service.

Choosing the right option for your needs will help you allocate your organization’s cardinality entitlement efficiently. See Guidance on how to reduce cardinality to learn more. You can also filter by specific tag values to further reduce the cardinality of your Monitoring MetricSet. All custom MMS include service level metrics, but you will have the option to additionally generate endpoint level metrics.

Option

Guidance

service-level MMS and endpoint-level MMS for all endpoints within a service

This option creates a custom dimensionalized MMS for each of the selected service(s), as well as an MMS for each endpoint in each selected service. This is the highest cardinality option.

service-level MMS and endpoint-level MMS for specific endpoints

This option creates custom dimensionalized MMS for each of the selected service(s) and an MMS for specific endpoints you select. In the box labeled Endpoint Filters, you can provide a list of endpoints or a regular expression pattern to generate MMS for specific endpoints you’re interested to minimize unecessary cardinality.

Service-level MMS only

This option creates an MMS for each of the selected service(s) and no endpoint-level MMS. This is the lowest cardinality option.

Scope of Monitoring MetricsSets 🔗

You can create custom MMS for endpoints (metrics starting with span*) and services (metrics starting with service.request.*), but not for workflows (metrics starting with workflow*) or traces (metrics starting with trace*) at this time. Custom MMS aren’t supported for global tags.

Add a Monitoring MetricSet 🔗

Follow these steps to create a Monitoring MetricSet.

  1. Navigate to the APM MetricSets configuration page. There are two ways to get there:

    1. From the left navigation panel, select: APM > APM Configurations > APM MetricSets.

    2. From the APM landing page, click APM Configuration and select APM MetricSets.

  2. Once in the MetricSets Configuation page, you have two options:

    1. If you have already indexed the span tag you are interested in, it appears in the list of MetricSets and is already generating Troubleshooting MetricSets. Click the edit icon for that span tag to open the Edit MetricSet modal and add a Monitoring MetricSet to your configuration using the following steps.

    2. If you haven’t already indexed the tag, follow steps 1-6 in Index a new span tag, then continue with the following steps in the Add MetricSet modal.

  3. In the Service field, enter the service or services for which you want to create a Monitoring MetricSet. Note that you can only create custom dimensionalized MMS for service-level indexed span tags. You can’t create custom MMS for globally indexed span tags.

  4. In the Add MetricSet or Edit MetricSet modal, select the checkbox for Also Create Monitoring MetricSet.

    This image shows the MetricSet creation modal.
  5. Choose how you want to add tag data to your Monitoring MetricSet from the dropdown. See Choose how you want to configure your Monitoring MetricSet above for more details.
    1. Service and all endpoint MMS: Create an MMS for each of the selected service(s), as well as an MMS for each endpoint in each selected service.

    2. Service and specific endpoint MMS: Create an MMS for each of the selected service(s) and an MMS for specific endpoints you select. To add Endpoint Filters, provide a list of endpoints or a regular expression pattern to generate MMS for specific endpoints you’re interested in.

    3. Service MMS only: Create an MMS for each of the selected service(s) and no endpoint-level MMS.

  6. (Optional) In the field labeled Filter by tag values, add tag filters to narrow the scope of your MMS to include only data associated with the tag values you enter. This can help further reduce cardinality in your MetricSets.

  7. Select Start Analysis and wait for the cardinality check to run. Then see Review the cardinality of your MetricSet below to assess the new MetricSet’s cardinality contribution and enable the MetricSet if you’re happy with it.

Review the cardinality of your MetricSet 🔗

When the cardinality check completes, you can see the estimated total cardinality.

  • If you are happy with the cardinality, click the check mark under Actions to enable your Monitoring MetricSet. The cardinality check expires after one hour.

  • If the check fail to run, run the check again.

  • If the check runs and returns a failure, your cardinality is too high to create the new MMS. See Troubleshoot cardinality in Monitoring MetricSets to adjust accordingly.

  • Once you accept or discard the pending MetricSet, you can continue editing your other MetricSets.

The following example shows a cardinality check.

This image shows the cardinality check for a Monitoring MetricSet.

Set up charts, dashboards, and alerts for custom dimensions 🔗

You can create charts, dashboards, and alerts based on your custom Monitoring MetricSet by leveraging the Infrastructure Monitoring platform.

To use the custom dimensionalized Monitoring MetricSets you have created, apply the filter sf_dimensionalized:true. This filters out the metrics generated by the default Monitoring Metricset. To filter your metrics even more, use the new dimension you have created which is the tag name.

Task

Instructions

Create charts

Create charts in Splunk Observability Cloud

Create dashboards

Create and customize dashboards

Create an alert

Configure detectors and alerts in Splunk APM

Monitor services in APM dashboards

Track service performance using dashboards in Splunk APM