Docs » Get AlwaysOn Profiling data into Splunk APM

Get AlwaysOn Profiling data into Splunk APM 🔗

Prerequisites 🔗

To get data into Splunk AlwaysOn Profiling, you need the following:

If the version of your Splunk OTel Collector is lower than 0.44.0, see Check the OpenTelemetry Collector configuration.

Get profiling data in 🔗

Follow these instructions to get profiling data into Splunk APM using AlwaysOn Profiling:

  1. Instrument your application or service.

  2. Enable AlwaysOn Profiling.

  3. Check that Observability Cloud is receiving profiling data.

Instrument your application or service 🔗

AlwaysOn Profiling requires APM tracing data to correlate stack traces to your application requests. To instrument your application for Splunk APM, follow the steps for the appropriate programming language:

Language

Available instrumentation

Documentation

Java

Splunk Distribution of OpenTelemetry Java v1.14.2 or higher

Instrument a Java application for Splunk Observability Cloud

Node.js

Splunk Distribution of OpenTelemetry JS

Instrument a Node application for Splunk Observability Cloud

.NET

SignalFx Instrumentation for .NET

Instrument a .NET application for Splunk Observability Cloud

Note

See Data retention in Application Performance Monitoring (APM) for information on Profiling data retention.

Enable AlwaysOn Profiling 🔗

After you’ve instrumented your service for Observability Cloud and checked that APM data is getting into Splunk APM, enable AlwaysOn Profiling.

To enable AlwaysOn Profiling, follow the steps for the appropriate programming language:

  • To use CPU profiling, enable the splunk.profiler.enabled system property, or set the SPLUNK_PROFILER_ENABLED environment variable to true.

  • Enable Memory profiling by setting the splunk.profiler.memory.enabled system property or the SPLUNK_PROFILER_MEMORY_ENABLED environment variable to true. To enable memory profiling, the splunk.profiler.enabled property must be set to true.

  • Make sure that the splunk.profiler.logs-endpoint system property or the SPLUNK_PROFILER_LOGS_ENDPOINT environment variable point to http://localhost:4317.

  • Port 9943 is the default port for the SignalFX receiver in the collector distribution. If you change this port in your collector config, you need to pass the custom port to the JVM.

The following example shows how to enable the profiler using the system property:

java -javaagent:./splunk-otel-javaagent.jar \
-Dsplunk.profiler.enabled=true \
-Dsplunk.profiler.memory.enabled=true \
-Dotel.exporter.otlp.endpoint=http(s)://collector:4317 \
-Dsplunk.metrics.endpoint=http(s)://collector:9943
-jar <your_application>.jar

For more configuration options, including setting a separate endpoint for profiling data, see Java settings for AlwaysOn Profiling.

Check that Observability Cloud is receiving profiling data 🔗

After you set up and enable AlwaysOn Profiling, check that profiling data is coming in:

  1. Log in to Splunk Observability Cloud.

  2. In the left navigation menu, select APM.

  3. In Splunk APM, select AlwaysOn Profiling.

  4. Select a service, and switch from the CPU view to the Memory view.

  5. If your service runs in multiple instances, select the instance that you’re interested in by selecting the host, container and process ID.

  6. If you’ve enabled Memory profiling, explore memory metrics. See Memory profiling metrics.

  7. You can also browse all stack traces coming from your application in the flame graph. See Understand and use the flame graph for more information about the flame graph. See also Browse stack traces linked to spans in Splunk APM to learn how to locate and browse call stacks.