Hadoop 🔗

Description 🔗

The Splunk Distribution of OpenTelemetry Collector provides this integration as the Hadoop monitor type for the Smart Agent Receiver.

This monitor collects metrics about a Hadoop 2.0+ cluster using the Hadoop Python plugin. If a remote JMX port is exposed in the Hadoop cluster, then you can also configure the hadoopjmx monitor to collect additional metrics about the Hadoop cluster.

Note

This monitor is not available on Windows as collectd plugins are only supported in Linux and Kubernetes.

The Hadoop monitor collects metrics from the Resource Manager REST API for the following components:

Cluster Metrics
Cluster Scheduler
Cluster Applications
Cluster Nodes
MapReduce Jobs

Installation 🔗

This monitor is available in the Smart Agent Receiver, which is part of the Splunk Distribution of OpenTelemetry Collector.

To install this integration:

Deploy the Splunk Distribution of OpenTelemetry Collector to your host or container platform.
Configure the monitor, as described in the next section.

Configuration 🔗

The Splunk Distribution of OpenTelemetry Collector allows embedding a Smart Agent monitor configuration in an associated Smart Agent Receiver instance.

Note: Providing a Hadoop monitor entry in your Collector or Smart Agent (deprecated) configuration is required for its use. Use the appropriate form for your agent type.

Splunk Distribution of OpenTelemetry Collector 🔗

To activate this monitor in the Splunk Distribution of OpenTelemetry Collector, add the following to your agent configuration:

receivers:
  smartagent/hadoop:
    type: collectd/hadoop
    ...  # Additional config

To complete the monitor activation, you must also include the smartagent/hadoop receiver item in a metrics pipeline. To do this, add the receiver item to the service > pipelines > metrics > receivers section of your configuration file.

See configuration examples for specific use cases that show how the Splunk Distribution of OpenTelemetry Collector can integrate and complement existing environments.

Smart Agent 🔗

To activate this monitor in the Smart Agent, add the following to your agent configuration:

monitors:  # All monitor config goes under this key
 - type: collectd/hadoop
   ...  # Additional config

See Smart Agent example configuration for an autogenerated example of a YAML configuration file, with default values where applicable.

Configuration settings 🔗

The following table shows the configuration options for this monitor:

Option	Required	Type	Description
`pythonBinary`	no	`string`	Path to a python binary that should be used to execute the Python code. If not set, a built-in runtime will be used. Can include arguments to the binary as well.
`host`	yes	`string`	Resource Manager Hostname
`port`	yes	`integer`	Resource Manager Port
`verbose`	no	`bool`	Log verbose information about the plugin (default: `false`)

Metrics 🔗

The following metrics are available for this integration:

Get help 🔗

If you are not able to see your data in Splunk Observability Cloud, try these tips:

Submit a case in the Splunk Support Portal
- Available to Splunk Observability Cloud customers
Call Splunk Customer Support
- Available to Splunk Observability Cloud customers
Ask a question and get answers through community support at Splunk Answers
- Available to Splunk Observability Cloud customers and free trial users
Join the Splunk #observability user group Slack channel to communicate with customers, partners, and Splunk employees worldwide
- Available to Splunk Observability Cloud customers and free trial users
- To learn how to join, see Get Started with Splunk Community - Chat groups

To learn about even more support options, see Splunk Customer Success.

Related Topics