Docs » Instrument back-end applications to send spans to Splunk APM » Instrument Java applications for Splunk Observability Cloud » Metrics and attributes collected by the Splunk OTel Java agent

Metrics and attributes collected by the Splunk OTel Java agent πŸ”—

The agent of the Splunk Distribution of OpenTelemetry Java collects the following application metrics data and attributes in addition to all that the upstream OpenTelemetry agent collects.

To learn about the different metric types, see Identify metric types.

Enable metrics collection πŸ”—

To collect Java application and Java Virtual Machine metrics, see Enable metrics collection.

Note

Application metrics collection is an experimental feature subject to future changes.

Application metrics πŸ”—

The agent of the Splunk Distribution of OpenTelemetry Java collects the following application metrics.

Default metric dimensions πŸ”—

The following dimensions are automatically added to all metrics exported by the agent:

Dimension

Description

deployment.environment

Value of the deployment.environment resource attribute, if present.

runtime

Value of the process.runtime.name resource attribute, for example OpenJDK Runtime Environment.

process.pid

The Java process identifier (PID).

container.id

Identifier of the container, if applicable.

host.name

Name of the host.

service

Value of the service.name resource attribute.

Supported libraries πŸ”—

The agent collects the following metrics through the following libraries:

Library/Framework

Instrumentation

Supported versions

JVM metrics

jvm-metrics-splunk

Java runtimes version 8 and higher

Apache DBCP2 connection pool metrics

commons-dbcp2-splunk

Version 2.0 and higher

c3p0 connection pool metrics

c3p0-splunk

Version 0.9.5 and higher

HikariCP connection pool metrics

hikaricp-splunk

Version 3.0 and higher

Oracle Universal Connection Pool metrics (UCP)

oracle-ucp-splunk

Version 11.2.0.4 and higher

Tomcat JDBC connection pool metrics

tomcat-jdbc-splunk

Version 8.5 and higher

Vibur DBCP connection pool metrics

vibur-dbcp-splunk

Version 20.0 and higher

Tomcat thread pool metrics

tomcat

Version 8.5 and higher

WebSphere Liberty thread pool metrics

liberty

Version 20.0.0.12

WebLogic thread pool metrics

weblogic

Versions 12.x and 14.x

JVM metrics πŸ”—

The Splunk OTel Java agent collects the following Java Virtual Machine (JVM) metrics when metric collection is enabled:

ClassLoader metrics πŸ”—

The agent collects the following ClassLoader metrics:

Metric

Type

Description

runtime.jvm.classes.loaded

Gauge

Number of loaded classes.

runtime.jvm.classes.unloaded

Counter

Total number of unloaded classes since the process started.

Garbage collection metrics πŸ”—

The agent collects the following garbage collection (GC) metrics:

Metric

Type

Description

runtime.jvm.gc.concurrent.phase.time

Timer

Time spent in concurrent phase, in milliseconds.

runtime.jvm.gc.live.data.size

Gauge

Size of long-lived heap memory pool after reclamation, in bytes.

runtime.jvm.gc.max.data.size

Gauge

Maximum size of long-lived heap memory pool, in bytes.

runtime.jvm.gc.memory.allocated

Counter

Increase in the size of the young heap memory pool after one garbage collection and before the next.

runtime.jvm.gc.memory.promoted

Counter

Count of positive increases in the size of the old generation memory pool from before to after garbage collection.

runtime.jvm.gc.pause

Timer

Time spent in garbage collection pause, in seconds. It produces multiple aggregations, such as runtime.jvm.gc.pause.avg, runtime.jvm.gc.pause.count, runtime.jvm.gc.pause.max, or runtime.jvm.gc.pause.totalTime.

Heap preassure metrics πŸ”—

The agent collects the following heap pressure metrics:

Metric

Type

Description

runtime.jvm.gc.overhead

Gauge

An approximation of the percentage of CPU time used by GC activities over the last lookback period or since monitoring began, whichever is shorter, in the range [0..1].

runtime.jvm.memory.usage.after.gc

Gauge

The percentage of long-lived heap pool used after the last GC event, in the range [0..1].

Memory metrics πŸ”—

The agent collects the following memory metrics:

Metric

Type

Description

process.runtime.jvm.memory.allocated

Counter

Total number of bytes allocated by JVM threads since the previous data point was emitted.
  • Use the rate per second rollup.

  • Requires to enable memory profiling, or to use the splunk.metrics.experimental.enabled flag.

process.runtime.jvm.memory.reclaimed

Counter

Total number of bytes reclaimed by the GC since the previous data point was emitted. Notes:
  • This metric might be inaccurate for concurrent garbage collectors such as Shenandoah or ZGC.

  • Use the rate per second rollup.

  • Requires to enable memory profiling,or to use the splunk.metrics.experimental.enabled flag.

runtime.jvm.buffer.count

Gauge

An estimate of the number of buffers in the pool.

runtime.jvm.buffer.memory.used

Gauge

An estimate of the memory that the JVM is using for this buffer pool, in bytes.

runtime.jvm.buffer.total.capacity

Gauge

An estimate of the total capacity of the buffers in this pool, in bytes.

runtime.jvm.memory.committed

Gauge

Amount of memory available to the JVM, in bytes.

runtime.jvm.memory.max

Gauge

Maximum amount of memory available for memory management, in bytes.

runtime.jvm.memory.used

Gauge

Amount of used memory, in bytes.

All memory pool metrics share the following tags:

Tag

Value

area

Either heap or nonheap

id

Name of the memory pool. For example, Perm Gen

Thread metrics πŸ”—

The agent collects the following thread metrics:

Metric

Type

Description

runtime.jvm.threads.daemon

Gauge

Number of live daemon threads.

runtime.jvm.threads.live

Gauge

Number of live threads, including both daemon and nondaemon threads.

runtime.jvm.threads.peak

Gauge

Peak live thread count since the JVM started or peak was reset.

runtime.jvm.threads.states

Gauge

Number of threads per state as a metric tag.

Connection pool metrics πŸ”—

The Splunk Distribution of OpenTelemetry Java instruments several Java Database Connectivity (JDBC) connection pool implementations:

  • Apache DBCP2

  • c3p0

  • HikariCP

  • Oracle Universal Connection Pool (UCP)

  • Tomcat JDBC

  • Vibur DBCP

  • WebSphere Liberty

  • WebLogic thread pools

Each of the connection pools reports a subset of the following metrics:

Metric

Type

Description

db.pool.connections

Gauge

Number of open connections.

db.pool.connections.active

Gauge

Number of open connections that are in use.

db.pool.connections.idle

Gauge

Number of open connections that are idle.

db.pool.connections.idle.max

Gauge

Maximum number of idle open connections allowed.

db.pool.connections.idle.min

Gauge

Minimum number of idle open connections allowed.

db.pool.connections.max

Gauge

Maximum number of open connections allowed.

db.pool.connections.pending_threads

Gauge

Number of threads that are waiting for an open connection.

db.pool.connections.timeouts

Counter

Number of connection timeouts that have happened since the application started.

db.pool.connections.create_time

Timer

Time it took to create a new connection.

db.pool.connections.wait_time

Timer

Time it took to get an open connection from the pool.

db.pool.connections.use_time

Timer

Time between borrowing a connection and returning it to the pool.

All connection pool metrics share the following tags:

Tag

Value

pool.name

Name of the connection pool. Spring bean name if Spring is used, JMX object name otherwise.

pool.type

Type or implementation of the connection pool. For example, c3p0, dbcp2, or hikari.

Thread pool metrics πŸ”—

The Splunk Distribution of OpenTelemetry Java instruments the following thread pool implementations:

  • Tomcat connector thread pools

  • WebSphere Liberty web request thread pool

  • Weblogic thread pools

Each of the supported connection pools reports a subset of the following metrics:

Metric

Type

Description

executor.threads

Timer

Number of threads in the pool.

executor.threads.active

Timer

Number of threads that are executing code.

executor.threads.idle

Timer

Number of threads that aren’t executing code.

executor.threads.core

Timer

Core thread pool size, expressed as the number of threads that are always kept in the pool.

executor.threads.max

Timer

Maximum number of threads in the pool.

executor.tasks.submitted

Counter

Total number of tasks submitted to the executor.

executor.tasks.completed

Counter

Total number of tasks completed by the executor.

All thread pool metrics have the following tags:

Tag

Value

executor.name

Name of the thread pool.

executor.type

Type/implementation of the connection pool. For example, tomcat, liberty, or weblogic.

WebEngine attributes πŸ”—

The Splunk Distribution of OpenTelemetry Java captures data about the application server and adds the following attributes to SERVER spans:

Span attribute

Description

webengine.name

Name of the applications server. For example, tomcat.

webengine.version

Version of the application server.

For a list of supported application servers, see the OpenTelemetry documentation at https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md#application-servers.

Send custom Java application metrics πŸ”—

The Splunk Distribution of OpenTelemetry Java agent detects if the instrumented application is using Micrometer and injects a special MeterRegistry implementation that lets the agent collect user-defined meters.

Follow these steps to enable custom application metrics:

Add the micrometer-core dependency πŸ”—

To export custom metrics through the Java agent, add a dependency on the micrometer-core library with version 1.5 and higher:

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-core</artifactId>
  <version>1.7.5</version>
</dependency>

Register each custom meter πŸ”—

You must register each custom meter in the global Metrics.globalRegistry instance provided by the Micrometer library. You can use one of meter factory methods provided by the Metrics class, or use meter builders and reference the Metrics.globalRegistry directly, as in the following example:

class MyClass {
Counter myCounter = Metrics.counter("my_custom_counter");
  Timer myTimer = Timer.builder("my_custom_timer").register(Metrics.globalRegistry);

  int foo() {
    myCounter.increment();
    return myTimer.record(this::fooImpl);
  }

  private int fooImpl() {
     // ...
  }
}

For more information on the Micrometer API, see the Micrometer official documentation.