Expvar (Go) 🔗

Description 🔗

The Splunk Distribution of OpenTelemetry Collector provides this integration as the expvar monitor using the Smart Agent receiver.

The expvar monitor is a Smart Agent monitor that scrapes metrics from variables exposed in JSON format at an HTTP endpoint by expvar. The monitor uses configured paths to get metric and dimension values from fetched JSON objects.

The Metrics section in this document shows metrics derived from expvar variable memstats. The memstat variable is exposed by default. These memstat metrics are referred to as standard or default metrics. The configuration examples shown are excerpts limited to the monitor configuration section of the Smart Agent configuration file agent.yml.

Installation 🔗

This monitor is available in the Smart Agent Receiver, which is part of the Splunk Distribution of OpenTelemetry Collector.

To install this integration:

Deploy the Splunk Distribution of OpenTelemetry Collector to your host or container platform.
Configure the monitor, as described in the next section.

Configuration 🔗

The Splunk Distribution of OpenTelemetry Collector allows embedding a Smart Agent monitor configuration in an associated Smart Agent Receiver instance.

Note: Providing an expvar monitor entry in your Collector or Smart Agent (deprecated) configuration is required for its use. Use the appropriate form for your agent type.

Splunk Distribution of OpenTelemetry Collector 🔗

To activate this monitor in the OpenTelemetry Collector, add the following to your agent configuration:

receivers:
  smartagent/expvar:
    type: expvar
    ... # Additional config

To complete the monitor activation, you must also include the smartagent/expvar receiver item in a metrics pipeline. To do this, add the receiver item to the service > pipelines > metrics > receivers section of your configuration file.

See configuration examples for specific use cases that show how the Splunk Distribution of OpenTelemetry Collector can integrate and complement existing environments.

Smart Agent 🔗

To activate this monitor in the Smart Agent, add the following to your agent configuration:

monitors:  # All monitor config goes under this key
 - type: expvar
   ...  # Additional config

See Smart Agent example configuration for an autogenerated example of a YAML configuration file, with default values where applicable.

Configuration settings 🔗

The following tables show the configuration options for this monitor:

Option	Required	Type	Description
`host`	yes	`string`	Host of the expvar endpoint
`port`	yes	`integer`	Port of the expvar endpoint
`useHTTPS`	no	`bool`	If `true`, the agent will connect to the host using HTTPS instead of plain HTTP. (default: `false`)
`skipVerify`	no	`bool`	If useHTTPS is `true` and this option is also `true`, the host TLS cert will not be verified. (default: `false`)
`path`	no	`string`	Path to the expvar endpoint, usually `/debug/vars` (the default). (default: `/debug/vars`)
`enhancedMetrics`	no	`bool`	If `true`, sends metrics memstats.alloc, memstats.by_size.size, memstats.by_size.mallocs and memstats.by_size.frees (default: `false`)
`metrics`	no	`list of objects (see below)`	Metrics configurations

The nested metrics configuration object has the following fields:

Option	Required	Type	Description
`name`	no	`string`	Metric name
`JSONPath`	yes	`string`	JSON path of the metric value
`type`	yes	`string`	SignalFx metric type. Possible values are “gauge” or “cumulative”
`dimensions`	no	`list of objects (see below)`	Metric dimensions
`pathSeparator`	no	`string`	Path separator character of metric value in JSON object (default: `.`)

The nested dimensions configuration object has the following fields:

Option	Required	Type	Description
`name`	yes	`string`	Dimension name
`JSONPath`	no	`string`	JSON path of the dimension value
`value`	no	`string`	Dimension value

Example configurations 🔗

The following example shows the minimum required expvar monitor configuration for exporting the default metrics from endpoint http://172.17.0.3:8000/debug/vars, where /debug/vars is the default path:

monitors:
- type: expvar
  host: 172.17.0.3
  path: /debug/vars
  port: 8000

You can add the extra dimension metric_source with a meaningful value to facilitate filtering in Splunk Observability Cloud:

monitors:
- type: expvar
  host: 172.17.0.3
  path: /debug/vars
  port: 8000
  extraDimensions:
    metric_source: expvar

The following example shows part of a JSON payload containing the exposed variable requestsPerSecond containing requests per second metric information:

{
  ...
  "requestsPerSecond": 919,
  ...
}

If the payload is emanating from endpoint http://172.17.0.4:6000/appmetrics, the monitor can be configured to scrape the JSONPath value requestsPerSecond. The metric name is optional. If not provided, the default JSONPath value requests_per_second is used instead.

monitors:
- type: expvar
  host: 172.17.0.4
  path: /debug/vars
  port: 6000
  metrics:
    - name: requests.sec
      JSONPath: requestsPerSecond
      type: gauge
  extraDimensions:
    metric_source: expvar-aws

The expvar monitor can be configured to extract metric values from complex JSON objects such as the one shown in the following example. Suppose the memstats variable shown in the example is exposed at endpoint http://172.17.0.5:5000/debug/vars and you want to extract the cumulative Mallocs values.

{
  ...
  "memstats": {
                ...
                "GCCPUFraction": 0.0000032707490586459204,
                "BySize": [
                  {
                      "Size": 32,
                      "Mallocs": 35387,
                      "Frees": 35021
                  },
                  {
                      "Size": 48,
                      "Mallocs": 35387,
                      "Frees": 63283
                  }
                ]
                "HeapAlloc": 2138088,
                ...
              }
  ...
}

To fetch the first cumulative Mallocs value in the BySize array, configure the monitor as shown in the following example. The configured JSONPath contains character delimited keys of metric values in the JSON object. The path must be defined fully, terminating on primitive values or array of primitive values. The path should not terminate on embedded objects. If you don’t provide a metric name for this configuration, the metric name defaults to memstats.by_size.mallocs. A dimension named memstats_by_size_index containing the array index 0 is also created.

monitors:
- type: expvar
  host: 172.12.0.5
  path: /debug/vars
  port: 5000
  metrics:
    - JSONPath: memstats.BySize.0.Mallocs
      type: cumulative
  extraDimensions:
    metric_source: expvar

The default path separator character is ., so there is no need to specify unless you have a different path separtor character. The following example is the same configuration using / as the path separator character:

monitors:
- type: expvar
  host: 172.12.0.5
  path: /debug/vars
  port: 5000
  metrics:
    - JSONPath: memstats/BySize/0/Mallocs
      pathSeparator: /
      type: cumulative
  extraDimensions:
    metric_source: expvar

To fetch all Mallocs values or a combination thereof, configure JSONPath with regular expression. The following configuration configures the monitor to fetch all Mallocs values (35387 and 35387). Two data points for metric memstats.by_size.mallocs containing the values will be fetched. The data points will have dimension memstats_by_size_index containing their respective array index. Note that the escape character \ is used to escape character . of regex .* in order take . literally as opposed to a path separator character.

monitors:
- type: expvar
  host: 172.12.0.5
  path: /debug/vars
  port: 5000
  metrics:
    - JSONPath: memstats.BySize.\\.*.Mallocs
      type: cumulative
  extraDimensions:
    metric_source: expvar

The following configuration fetches all BySize values:

monitors:
- type: expvar
  host: 172.12.0.5
  path: /debug/vars
  port: 5000
  metrics:
    - JSONPath: memstats.BySize.\\.*.\\.*
      type: cumulative
  extraDimensions:
    metric_source: expvar

The following configuration also fetches all BySize values:

monitors:
- type: expvar
  host: 172.12.0.5
  path: /debug/vars
  port: 5000
  metrics:
    - JSONPath: memstats.BySize.\\d+.\\.*
      type: cumulative
  extraDimensions:
    metric_source: expvar

Custom dimensions can be added to metrics, as shown in the following example. The dimension name is required if a dimension value is provided. The dimension name is optional when the JSONPath for the dimension is provided.

monitors:
- type: expvar
  host: 172.12.0.5
  path: /debug/vars
  port: 5000
  metrics:
    - JSONPath: memstats.BySize.\\.*.Mallocs
      type: cumulative
      - dimensions:
        name: physical_memory
        value: 4GiB
      - name: app_mem
        value: "10 MiB"
  extraDimensions:
    metric_source: expvar

The dimension JSONPathcan be configured, as shown in the following example. If the dimension name is not provided, then it is constructed from snake casing the JSONPath. The dimension JSONPath must be shorter than the metric JSONPath and start at same root. For the following configuration, dimensions memory_stats and memstats_by_size will contain values BySize and 0 respectively.

monitors:
- type: expvar
  host: 172.12.0.5
  path: /debug/vars
  port: 5000
  metrics:
    - JSONPath: memstats.BySize.0.Mallocs
      type: cumulative
      - dimensions:
        name: memory_stats
        JSONPath: memstats
      - dimensions:
        JSONPath: memstats/BySize
  extraDimensions:
    metric_source: expvar

Note: Do not configure the monitor for memstats metrics because they are standard metrics provided by default. Memstats is used in this configuration as an example.

Metrics 🔗

The following metrics are available for this integration: This monitor will also emit by default any metrics that are not listed in the following table.

Get help 🔗

If you are not able to see your data in Splunk Observability Cloud, try these tips:

Submit a case in the Splunk Support Portal
- Available to Splunk Observability Cloud customers
Call Splunk Customer Support
- Available to Splunk Observability Cloud customers
Ask a question and get answers through community support at Splunk Answers
- Available to Splunk Observability Cloud customers and free trial users
Join the Splunk #observability user group Slack channel to communicate with customers, partners, and Splunk employees worldwide
- Available to Splunk Observability Cloud customers and free trial users
- To learn how to join, see Get Started with Splunk Community - Chat groups

To learn about even more support options, see Splunk Customer Success.

Related Topics