Docs » Key concepts in Splunk APM

Key concepts in Splunk APM ๐Ÿ”—

Splunk APM is an Application Performance Monitoring solution that collects every span from every instrumented application to give you a complete picture of the interactions among the microservices that make up your distributed system.

Concept

Description

Service

A unit of software that connects to other services to make up a complete application.

Inferred service

A remote service that is not instrumented in Splunk APM, but which Splunk APM can identify based on information in spans that make calls to the remote service.

Service map

A visualization of your instrumented and inferred services and their relationships.

Identity

A unique set of indexed span tags that corresponds to a Splunk APM object.

Operation

A specific action performed by a service.

Endpoint

A special type of operation where the root spanโ€™s span.kind = SERVER. A given service typically has one or more endpoints associated with it.

Environment

A distinct deployment of your application that doesnโ€™t interact directly with other deployments of the same application.

Trace

A collection of operations, known as spans, that represents a unique transaction an application handles.

Trace Search

Search through all traces from all instrumented services to find the exact trace youโ€™re looking for.

Trace View

View the span waterfall chart for a specific trace, and search for spans within that trace.

Business Workflow

A set of correlated traces that track a transaction or user flow of particular interest.

Span

A single operation within a system of applications and services.

Span tag

A piece of metadata attached to a span that provides more information about the operation the span represents.

Indexed span tag

A span tag for which Splunk generates Troubleshooting MetricSets.

Tag Spotlight

A top-down view of your services based on indexed span tags.

MetricSet

A set of metric time series capturing the values of key indicators over time, such as request rate, error rate and durations, calculated based on your traces and spans in Splunk APM.

Troubleshooting MetricSets (TMS)

Metric time series used for high-cardinality troubleshooting of identities in APM and for historical comparison among spans and workflows.

Monitoring MetricSets (MMS)

Metric time series that power Splunk APMโ€™s real-time monitoring capabilities, including charts, dashboards, and detectors.

Cardinality

The number of distinct values in a dataset.

Services ๐Ÿ”—

Services are the key components of the systems you can monitor with Splunk APM.

Service ๐Ÿ”—

A small, flexible, and autonomous unit of software that connects to other services to make up a complete application. A service typically represents a collection of API endpoints and operations that work together with other servicesโ€™ endpoints in a distributed and dynamic architecture to deliver the full functionality of an application.

โ€œServiceโ€ is an umbrella term that encompasses container services (e.g. Docker, Kubernetes), microservices, and even calls to serverless functions. By instrumenting each of the services that make up your application, you can collect spans that represent operations within services and traces that represent collections of operations across services, to analyze and monitor this activity in Splunk APM.

Instrumented service ๐Ÿ”—

An instrumented service is one you have instrumented using an agent such as OpenTelemetry or Smart Agent so that it sends its spans to Splunk APM. See Instrument back-end applications to send spans to Splunk APM to learn more about instrumenting services.

Inferred service ๐Ÿ”—

A remote service that is not instrumented in Splunk APM, but which Splunk APM can identify based on information in spans that make calls to the remote service. Inferred services often include external service providers, pub/subs, Remote Procedure Calls (RPCs), and databases. To learn more, see Analyze the performance of inferred services.

Service map ๐Ÿ”—

A visualization of your instrumented and inferred services and their relationships. The service map is dynamically generated based on your selections in the time range, environment, workflow, service, and tag filters. See View dependencies among your services in the service map to learn more about using the service map in APM, or see Investigate the root cause of an error with Splunk APM service map for a dedicated use case.

Identity ๐Ÿ”—

A unique set of indexed span tags that corresponds to a Splunk APM object. An identity can represent a service, endpoint, operation, edge, or workflow, and is always related to at least one service. For more information, see Manage services, spans, and traces in Splunk APM.

Operation ๐Ÿ”—

A specific action performed by a service. Each operation in an instrumented service is captured in an individual spans.

Endpoint ๐Ÿ”—

A special type of operation where the root spanโ€™s span.kind = SERVER. A given service typically has one or more endpoints associated with it.

Environment ๐Ÿ”—

The term โ€œenvironmentโ€ refers to the deployment environment, which is a distinct deployment in Splunk APM that doesnโ€™t interact directly with other deployments of the same application. Separate deployment environments are often used for different stages of the development process, such as development, staging, and production. For more information, see Set up deployment environments in Splunk APM.

Traces and spans ๐Ÿ”—

Spans and traces form the backbone of application monitoring in Splunk APM. The following image illustrates the relationship between traces and spans:

This image shows a trace represented by a series of multicolored bars labeled with the letters A, B, C, D, and E. Each lettered bar represents a single span. The spans are organized to visually represent a hierarchical relationship in which span A is the parent span and the subsequent spans are its children.

Trace ๐Ÿ”—

A collection of related operations, known as spans, that represents a unique transaction an application handles. For more information, see Manage services, spans, and traces in Splunk APM.

Trace View ๐Ÿ”—

In Trace view, you can view the span waterfall chart for a specific trace, and search for spans within that trace.

To learn more, see:

Business Workflow ๐Ÿ”—

Using Business Workflows, you can correlate a set of related traces that track a transaction or user flow of particular interest.

To learn more, see:

Span ๐Ÿ”—

A single operation within a system of applications and services. Spans include span tags, which provide metadata such as the location and duration of the operations they represent. A group of related spans makes up a trace. For more information, see Manage services, spans, and traces in Splunk APM.

Span tag ๐Ÿ”—

A piece of metadata attached to a span that provides more information about the operation the span represents. Examples of span tags include service.name and http.operation. You can add span tags to spans during instrumentation or in the Splunk Distribution of OpenTelemetry Collector. Span tags are also known as โ€œattributesโ€ in the OpenTelemetry context.

For more information, see Analyze services with span tags and MetricSets in Splunk APM.

Indexed span tag ๐Ÿ”—

When you index a span tag, you indicate to Splunk APM that you are particularly interested in this tag and would like to generate additional analytics for it. Indexing a span tag generates Troubleshooting MetricSets for that tag. When you index a service-level span tag, you also have the option to generate custom dimensionalized Monitoring MetricSets using that span tag as a dimension.

To learn how to index a span tag, see Index span tags to generate Troubleshooting MetricSets.

Tag Spotlight ๐Ÿ”—

The Tag Spotlight view in Splunk APM offers a top-down view of your services based on indexed span tags.

To learn more, see:

MetricSets ๐Ÿ”—

MetricSets are the central type of metric data that power Splunk APM.

A MetricSet is a set of metric time series capturing the values of key indicators over time, such as request rate, error rate and durations, calculated based on your traces and spans in Splunk APM. Generate MetricSets by indexing span tags of interest.

There are two categories of MetricSet in APM: Troubleshooting MetricSets (TMS), used for high-cardinality troubleshooting, and Monitoring MetricSets (MMS), used for real-time monitoring. For more information, see Learn about MetricSets in APM.

Troubleshooting MetricSets ๐Ÿ”—

Metric time series used for high-cardinality troubleshooting of identities in APM and for historical comparison among spans and workflows. Splunk APM generates Troubleshooting MetricSets based on indexed span tags.

To learn more, see Troubleshooting MetricSets .

Monitoring MetricSets ๐Ÿ”—

Metric time series used to monitor and alert on the performance of your services in real time. MMS power the real-time APM Landing Page and the dashboard view, and are the metrics that detectors monitor and use to generate alerts. MMS use the same functionality as metric time series in Infrastructure Monitoring to monitor and alert on the performance of applications and services.

For more information about MMS, see Monitoring MetricSets.

Cardinality ๐Ÿ”—

The number of distinct values in a dataset. Low cardinality data has a small number of distinct values. High cardinality data has a large number of distinct values, and requires more computation and storage to analyze and store.

See Troubleshoot cardinality in Monitoring MetricSets to learn more about working with high cardinality data.