Docs » Get started with Splunk APM » Deploy an OpenTelemetry Collector for Splunk APM

Deploy an OpenTelemetry Collector for Splunk APM 🔗


The original µAPM product, released in 2019, is now called µAPM Previous Generation (µAPM PG). Wherever you see the reference, µAPM now refers to the product released on March 31, 2020.

If you’re using µAPM Previous Generation (µAPM PG), see Overview of SignalFx Microservices APM Previous Generation (µAPM PG).

After installing the SignalFx Smart Agent on each instance, you can optionally deploy the Splunk distribution of OpenTelemetry Collector in each datacenter/region/cluster where traced applications run. In general, the OpenTelemetry Collector should receive data from the Smart Agent.

Sending trace spans through the OpenTelemetry Collector gives the OpenTelemetry Collector an opportunity to modify span attributes prior to being exported to the configured backend. In addition, the collector reduces the number of devices that require internet access. For more information, see Attribute Processor on GitHub.

For more information about deploying an OpenTelemetry Collector, see Getting started on the OpenTelemetry website.

After you deploy an OpenTelemetry Collector, configure a Smart Agent to forward data to it over the SAPM protocol. You can deploy multiple OpenTelemetry Collectors behind a load balancer for a high-availability configuration. In this deployment model, configure the Smart Agent to point to an OpenTelemetry Collector VIP.


Splunk APM requires OpenTelemetry Collector version 0.3.0 or above.

How the OpenTelemetry Collector works 🔗

The OpenTelemetry Collector uses pipelines to receive, process, and export trace data with components conveniently known as receivers, processors, and exporters. Set up pipelines with services. You can also add extensions that provide an OpenTelemetry Collector with additional functionality, such as diagnostics and health checks. The OpenTelemetry Collector has two versions: a core version and a contributions version. The core version provides receivers, processors, and exporters for general use. The contributions version provides receivers, processors, and exporters for specific vendors and use cases.

The Splunk distribution of OpenTelemetry Collector includes everything you need to send spans and traces to Splunk APM.

APM uses these contributions versions for receivers and exporters to send data to an OpenTelemetry Collector and to receive data from an OpenTelemetry Collector:

Component Name Description
Receiver sapm Component that sets the endpoint for receiving trace data with the SAPM format.
Receiver signalfx Component that sets the endpoint for receiving metrics data with the metric data format.
Exporter sapm Component that forwards data to APM with the SAPM format.
Exporter signalfx Component that forwards data to APM with the metric data format.

OpenTelemetry provides information about some Collector components on GitHub:

For more information about each component, including services, see Configuration on the OpenTelemetry site.

Plan your OpenTelemetry Collector deployment 🔗

The OpenTelemetry Collector can be scaled up or out as needed. A single Collector is generally capable of over 10,000 spans per second per CPU core. Try to leverage a ratio of 1:2 for CPU:memory and to allocate at least a CPU core per Collector. You can also deploy multiple Collectors. Each Collector runs independently, so sizing increases linearly with the number of Collectors you deploy.

Migrate from OpenTracing to OpenTelemetry 🔗

Splunk APM is migrating from OpenTracing semantics to OpenTelemetry semantics for span tag names. Some span tag names are changing. If you indexed any span tags that are migrating to new names, you just have to be aware of the new name for each affected span tag.

Starting on October 19, 2020, APM includes span tag names for both OpenTracing and OpenTelemetry.

On October 28, 2020, APM makes only OpenTelemetry span tag names available, and automatically remaps any OpenTracing span tag names to OpenTelemetry span tag names.

These are the span tag names that change with the migration to OpenTelemetry semantics:

OpenTracing OpenTelemetry
db.type db.system
message_bus.destination messaging.destination
topic messaging.destination peer.service peer.service
peer.ipv4 net.peer.ip
peer.ipv6 net.peer.ip
peer.port net.peer.port
sfx.error.kind exception.type
error.type exception.type
sfx.error.stack exception.stacktrace
error.stack exception.stacktrace
sfx.error.message exception.message
error.message exception.message
sfx.error.object exception.object
error.object exception.object
signalfx.tracing.version otel.library.version
kubernetes_pod_uid k8s.pod.uid

Splunk is also migrating automatic instrumentation tools to OpenTelemetry. For information about migrating from a SignalFx instrumentation library to a Splunk distribution of OpenTelemetry instrumentation, see the steps for each supported language:

Deploy an OpenTelemetry Collector from a binary or as a Docker image 🔗

Follow these steps to deploy an OpenTelemetry Collector from a binary or as a Docker image. Use the contributions version of the OpenTelemetry Collector.

  1. Download the latest release of the OpenTelemetry Collector contributions version on GitHub.

  2. Create a configuration file that defines components for the OpenTelemetry Collector. For more information about creating a configuration file, see the example collector.yaml file on GitHub.

  3. Deploy an Open Telemetry Collector.

    Deploy from a Docker container (replace 0.13.0 with the latest stable version number if necessary):

    $ docker run -p 13133:13133 -p 14250:14250 -p 14268:14268 -p 4317:4317 \
        -p 6060:6060 -p 7276:7276 -p 8888:8888 \ -p 9411:9411 -p 9943:9943 \
        -e SPLUNK_CONFIG=/etc/collector.yaml \
        -v "${PWD}/collector.yaml":/etc/collector.yaml:ro \
        --name otelcontribcol otel/opentelemetry-collector-contrib:0.13.0

    Deploy from a binary:

    $ otelcontribcol --config collector.yaml --mem-ballast-size-mib=683

Deploy an OpenTelemetry Collector in Kubernetes 🔗

To deploy the OpenTelemetry Collector in Kubernetes, create a configuration file that defines a ConfigMap, Service, and Deployment for the cluster. For more information about creating a configuration file, see the example signalfx-k8s.yaml file on GitHub.