Docs » Troubleshoot AlwaysOn Profiling

Troubleshoot AlwaysOn Profiling 🔗

If you have instrumented an application but are not seeing profiling data in Splunk APM, use the following guidelines to troubleshoot AlwaysOn Profiling:

No profiling data in Splunk Observability Cloud 🔗

If profiling data does not appear in Observability Cloud, do the following:

Check that you’ve instrumented your application 🔗

You can extract AlwaysOn Profiling data only if you’ve instrumented your application or service for Splunk APM. If the APM instrumentation is not loading or isn’t working, the profiler cannot work.

To solve this, check that you’ve instrumented your application and that the application is sending trace data to APM. See Instrument your application or service.

Check the OpenTelemetry Collector configuration 🔗

If the Splunk Distribution of OpenTelemetry Collector isn’t configured to send logs to Observability Cloud using the Splunk HEC exporter, it drops profiling data. To solve this issue, edit the configuration file for the Splunk Distribution of OpenTelemetry Collector and make sure that a profiling pipeline exists with an OTLP gRPC receiver and a Splunk HEC exporter.

The following example shows you how to configure a pipeline in the agent-config.yaml file. Set the SPLUNK_ACCESS_TOKEN environment variable to a valid access token. See Create and manage organization access tokens using Splunk Observability Cloud.

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  splunk_hec:
    token: "${SPLUNK_ACCESS_TOKEN}"
    endpoint: "https://ingest.${SPLUNK_REALM}.signalfx.com/v1/log"
  logging/info:
    loglevel: info

service:
  pipelines:
    logs/profiling:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [logging/info, splunk_hec]

AlwaysOn Profiling requires the Splunk HTTP Event Collector (HEC) exporter to send profiling data to Splunk Observability Cloud. The exporter is configured automatically for the Splunk OTel Collector version 0.44.0 and higher. If you’re using a version of the Collector lower than 0.44.0, you might have to edit the configuration manually.

Note

AlwaysOn Profiling requires the Splunk Distribution of OpenTelemetry Collector version 0.34 or higher.

Check that you’ve enabled AlwaysOn Profiling 🔗

Depending on the programming language, you can enable AlwaysOn Profiling by setting a system property, a function argument, or an environment variable. System properties and function arguments always take precedence. If the profiler is not enabled, Observability Cloud can’t receive profiling data.

To solve this issue, check that you’ve enabled the profiler. See Enable AlwaysOn Profiling.

No call stacks available for a span 🔗

Span might lack call stacks if the duration of the span is shorter than the snapshot interval for capturing call stacks. For example, the default snapshot interval for Java instrumentation is 10 seconds, so spans shorter than 10 seconds might not contain call stacks. To set a shorter interval, set the SPLUNK_PROFILER_CALL_STACK_INTERVAL environment variable to a value lower than 10000 milliseconds.

Another cause for call stacks not appearing is when HTTP requests follow an async/await pattern. When the processing thread was executing in the scope of a span from another trace when the snapshot was recorded.

AlwaysOn Profiling is not accessible in Observability Cloud 🔗

If you’re sending profiling data to Observability Cloud but can’t see AlwaysOn Profiling in Splunk APM, your organization might be lacking the profiler entitlement.

To solve this issue, reach out to Splunk Support to request they enable the AlwaysOn Profiling feature.

Instrumentation-specific troubleshooting 🔗

Some profiler issues might be specific to the APM instrumentation. See the following instructions to troubleshoot instrumentation-specific issues: