Docs » Integrations Guide » Integrations Reference » Apache Hadoop

../../_images/integrations_hadoop.png Apache Hadoop πŸ”—

DESCRIPTION πŸ”—

This integration primarily consists of the Smart Agent monitor collectd/hadoop. Below is an overview of that monitor.

Smart Agent Monitor πŸ”—

Collects metrics about a Hadoop 2.0+ cluster using the collectd Hadoop Python plugin. If a remote JMX port is exposed in the hadoop cluster, then you may also configure the collectd/hadoopjmx monitor to collect additional metrics about the hadoop cluster.

The collectd/hadoop monitor will collect metrics from the Resource Manager REST API for the following:

  • Cluster Metrics
  • Cluster Scheduler
  • Cluster Applications
  • Cluster Nodes
  • MapReduce Jobs

INSTALLATION πŸ”—

This integration is part of the SignalFx Smart Agent as the collectd/hadoop monitor. You should first deploy the Smart Agent to the same host as the service you want to monitor, and then continue with the configuration instructions below.

CONFIGURATION πŸ”—

To activate this monitor in the Smart Agent, add the following to your agent config:

monitors:  # All monitor config goes under this key
 - type: collectd/hadoop
   ...  # Additional config

For a list of monitor options that are common to all monitors, see Common Configuration.

Config option Required Type Description
pythonBinary no string Path to a python binary that should be used to execute the Python code. If not set, a built-in runtime will be used. Can include arguments to the binary as well.
host yes string Resource Manager Hostname
port yes integer Resource Manager Port
verbose no bool Log verbose information about the plugin (default: false)

USAGE πŸ”—

SignalFx provides several built-in dashboards for Hadoop YARN, HDFS, and MapReduce. Examples are shown below.

  • Hadoop YARN Resource Manager and Application metrics

    ../../_images/yarn_resource_manager.png ../../_images/yarn_application.png

  • Hadoop HDFS HDFS Overview, NameNode, and DataNode metrics

    ../../_images/hdfs_overview.png ../../_images/hdfs_namenode.png ../../_images/hdfs_datanode.png

  • Hadoop MapReduce MapReduce applications, jobs, and resource usage

    ../../_images/mapreduce_apps.png ../../_images/mapreduce_jobs.png ../../_images/mapreduce_usage.png

METRICS πŸ”—

Metric Name Description Type
counter.hadoop.cluster.metrics.total_mb cumulative
counter.hadoop.cluster.metrics.total_nodes cumulative
counter.hadoop.cluster.metrics.total_virtual_cores cumulative
gauge.hadoop.cluster.metrics.active_nodes gauge
gauge.hadoop.cluster.metrics.allocated_mb gauge
gauge.hadoop.cluster.metrics.allocated_virtual_cores gauge
gauge.hadoop.cluster.metrics.apps_completed gauge
gauge.hadoop.cluster.metrics.apps_failed gauge
gauge.hadoop.cluster.metrics.apps_killed gauge
gauge.hadoop.cluster.metrics.apps_pending gauge
gauge.hadoop.cluster.metrics.apps_running gauge
gauge.hadoop.cluster.metrics.apps_submitted gauge
gauge.hadoop.cluster.metrics.available_mb gauge
gauge.hadoop.cluster.metrics.available_virtual_cores gauge
gauge.hadoop.cluster.metrics.containers_allocated gauge
gauge.hadoop.cluster.metrics.containers_pending gauge
gauge.hadoop.cluster.metrics.containers_reserved gauge
gauge.hadoop.cluster.metrics.decommissioned_nodes gauge
gauge.hadoop.cluster.metrics.lost_nodes gauge
gauge.hadoop.cluster.metrics.rebooted_nodes gauge
gauge.hadoop.cluster.metrics.reserved_mb gauge
gauge.hadoop.cluster.metrics.reserved_virtual_cores gauge
gauge.hadoop.cluster.metrics.total_mb gauge
gauge.hadoop.cluster.metrics.total_virtual_cores gauge
gauge.hadoop.cluster.metrics.unhealthy_nodes gauge
gauge.hadoop.mapreduce.job.elapsedTime gauge
gauge.hadoop.mapreduce.job.failedMapAttempts gauge
gauge.hadoop.mapreduce.job.failedReduceAttempts gauge
gauge.hadoop.mapreduce.job.mapsTotal gauge
gauge.hadoop.mapreduce.job.successfulMapAttempts gauge
gauge.hadoop.mapreduce.job.successfulReduceAttempts gauge
gauge.hadoop.resource.manager.apps.allocatedMB gauge
gauge.hadoop.resource.manager.apps.allocatedVCores gauge
gauge.hadoop.resource.manager.apps.clusterUsagePercentage gauge
gauge.hadoop.resource.manager.apps.memorySeconds gauge
gauge.hadoop.resource.manager.apps.priority gauge
gauge.hadoop.resource.manager.apps.progress gauge
gauge.hadoop.resource.manager.apps.queueUsagePercentage gauge
gauge.hadoop.resource.manager.apps.runningContainers gauge
gauge.hadoop.resource.manager.apps.vcoreSeconds gauge
gauge.hadoop.resource.manager.nodes.availMemoryMB gauge
gauge.hadoop.resource.manager.nodes.availableVirtualCores gauge
gauge.hadoop.resource.manager.nodes.numContainers gauge
gauge.hadoop.resource.manager.nodes.usedMemoryMB gauge
gauge.hadoop.resource.manager.nodes.usedVirtualCores gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.absoluteCapacity gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.absoluteMaxCapacity gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.absoluteUsedCapacity gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.allocatedContainers gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.capacity gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.maxApplications gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.maxApplicationsPerUser gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.maxCapacity gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.numActiveApplications gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.numApplications gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.numContainers gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.numPendingApplications gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.pendingContainers gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.reservedContainers gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.usedCapacity gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.userLimit gauge
gauge.hadoop.resource.manager.scheduler.leaf.queue.userLimitFactor gauge
gauge.hadoop.resource.manager.scheduler.root.queue.capacity gauge
gauge.hadoop.resource.manager.scheduler.root.queue.maxCapacity gauge
gauge.hadoop.resource.manager.scheduler.root.queue.usedCapacity gauge
hadoop.cluster.metrics.active_nodes gauge
hadoop.cluster.metrics.allocated_mb gauge
hadoop.cluster.metrics.allocated_virtual_cores gauge
hadoop.cluster.metrics.apps_completed gauge
hadoop.cluster.metrics.apps_failed gauge
hadoop.cluster.metrics.apps_killed gauge
hadoop.cluster.metrics.apps_pending gauge
hadoop.cluster.metrics.apps_running gauge
hadoop.cluster.metrics.apps_submitted gauge
hadoop.cluster.metrics.available_mb gauge
hadoop.cluster.metrics.available_virtual_cores gauge
hadoop.cluster.metrics.containers_allocated gauge
hadoop.cluster.metrics.containers_pending gauge
hadoop.cluster.metrics.containers_reserved gauge
hadoop.cluster.metrics.decommissioned_nodes gauge
hadoop.cluster.metrics.lost_nodes gauge
hadoop.cluster.metrics.rebooted_nodes gauge
hadoop.cluster.metrics.reserved_mb gauge
hadoop.cluster.metrics.reserved_virtual_cores gauge
hadoop.cluster.metrics.total_mb counter
hadoop.cluster.metrics.total_nodes counter
hadoop.cluster.metrics.total_virtual_cores counter
hadoop.cluster.metrics.unhealthy_nodes gauge
hadoop.mapreduce.job.elapsedTime gauge
hadoop.mapreduce.job.failedMapAttempts gauge
hadoop.mapreduce.job.failedReduceAttempts gauge
hadoop.mapreduce.job.killedMapAttempts gauge
hadoop.mapreduce.job.killedReduceAttempts gauge
hadoop.mapreduce.job.mapsCompleted gauge
hadoop.mapreduce.job.mapsPending gauge
hadoop.mapreduce.job.mapsRunning gauge
hadoop.mapreduce.job.mapsTotal gauge
hadoop.mapreduce.job.newMapAttempts gauge
hadoop.mapreduce.job.newReduceAttempts gauge
hadoop.mapreduce.job.reducesCompleted gauge
hadoop.mapreduce.job.reducesPending gauge
hadoop.mapreduce.job.reducesTotal gauge
hadoop.mapreduce.job.runningMapAttempts gauge
hadoop.mapreduce.job.runningReduceAttempts gauge
hadoop.mapreduce.job.successfulMapAttempts gauge
hadoop.mapreduce.job.successfulReduceAttempts gauge
hadoop.resource.manager.apps.allocatedMB gauge
hadoop.resource.manager.apps.allocatedVCores gauge
hadoop.resource.manager.apps.clusterUsagePercentage gauge
hadoop.resource.manager.apps.memorySeconds gauge
hadoop.resource.manager.apps.numAMContainerPreempted gauge
hadoop.resource.manager.apps.numNonAMContainerPreempted gauge
hadoop.resource.manager.apps.preemptedResourceMB gauge
hadoop.resource.manager.apps.preemptedResourceVCores gauge
hadoop.resource.manager.apps.priority gauge
hadoop.resource.manager.apps.progress gauge
hadoop.resource.manager.apps.queueUsagePercentage gauge
hadoop.resource.manager.apps.runningContainers gauge
hadoop.resource.manager.apps.vcoreSeconds gauge
hadoop.resource.manager.node.nodeCPUUsage gauge
hadoop.resource.manager.node.nodePhysicalMemoryMB gauge
hadoop.resource.manager.node.nodeVirtualMemoryMB gauge
hadoop.resource.manager.nodes.availMemoryMB gauge
hadoop.resource.manager.nodes.availableVirtualCores gauge
hadoop.resource.manager.nodes.numContainers gauge
hadoop.resource.manager.nodes.usedMemoryMB gauge
hadoop.resource.manager.nodes.usedVirtualCores gauge
hadoop.resource.manager.scheduler.fifo.availNodeCapacity gauge
hadoop.resource.manager.scheduler.fifo.capacity gauge
hadoop.resource.manager.scheduler.fifo.maxQueueMemoryCapacity gauge
hadoop.resource.manager.scheduler.fifo.minQueueMemoryCapacity gauge
hadoop.resource.manager.scheduler.fifo.numContainers gauge
hadoop.resource.manager.scheduler.fifo.numNodes gauge
hadoop.resource.manager.scheduler.fifo.totalNodeCapacity gauge
hadoop.resource.manager.scheduler.fifo.usedCapacity gauge
hadoop.resource.manager.scheduler.fifo.usedNodeCapacity gauge
hadoop.resource.manager.scheduler.leaf.queue.absoluteCapacity gauge
hadoop.resource.manager.scheduler.leaf.queue.absoluteMaxCapacity gauge
hadoop.resource.manager.scheduler.leaf.queue.absoluteUsedCapacity gauge
hadoop.resource.manager.scheduler.leaf.queue.allocatedContainers gauge
hadoop.resource.manager.scheduler.leaf.queue.capacity gauge
hadoop.resource.manager.scheduler.leaf.queue.maxActiveApplications gauge
hadoop.resource.manager.scheduler.leaf.queue.maxActiveApplicationsPerUser gauge
hadoop.resource.manager.scheduler.leaf.queue.maxApplications gauge
hadoop.resource.manager.scheduler.leaf.queue.maxApplicationsPerUser gauge
hadoop.resource.manager.scheduler.leaf.queue.maxCapacity gauge
hadoop.resource.manager.scheduler.leaf.queue.numActiveApplications gauge
hadoop.resource.manager.scheduler.leaf.queue.numApplications gauge
hadoop.resource.manager.scheduler.leaf.queue.numContainers gauge
hadoop.resource.manager.scheduler.leaf.queue.numPendingApplications gauge
hadoop.resource.manager.scheduler.leaf.queue.pendingContainers gauge
hadoop.resource.manager.scheduler.leaf.queue.reservedContainers gauge
hadoop.resource.manager.scheduler.leaf.queue.usedCapacity gauge
hadoop.resource.manager.scheduler.leaf.queue.userLimit gauge
hadoop.resource.manager.scheduler.leaf.queue.userLimitFactor gauge
hadoop.resource.manager.scheduler.queue.resource.memory gauge
hadoop.resource.manager.scheduler.queue.resource.vCores gauge
hadoop.resource.manager.scheduler.queue.users.numActiveApplications gauge
hadoop.resource.manager.scheduler.queue.users.numPendingApplications gauge
hadoop.resource.manager.scheduler.root.queue.capacity gauge
hadoop.resource.manager.scheduler.root.queue.maxCapacity gauge
hadoop.resource.manager.scheduler.root.queue.usedCapacity gauge

counter.hadoop.cluster.metrics.total_mb πŸ”—

cumulative

counter.hadoop.cluster.metrics.total_nodes πŸ”—

cumulative

counter.hadoop.cluster.metrics.total_virtual_cores πŸ”—

cumulative

gauge.hadoop.cluster.metrics.active_nodes πŸ”—

gauge

gauge.hadoop.cluster.metrics.allocated_mb πŸ”—

gauge

gauge.hadoop.cluster.metrics.allocated_virtual_cores πŸ”—

gauge

gauge.hadoop.cluster.metrics.apps_completed πŸ”—

gauge

gauge.hadoop.cluster.metrics.apps_failed πŸ”—

gauge

gauge.hadoop.cluster.metrics.apps_killed πŸ”—

gauge

gauge.hadoop.cluster.metrics.apps_pending πŸ”—

gauge

gauge.hadoop.cluster.metrics.apps_running πŸ”—

gauge

gauge.hadoop.cluster.metrics.apps_submitted πŸ”—

gauge

gauge.hadoop.cluster.metrics.available_mb πŸ”—

gauge

gauge.hadoop.cluster.metrics.available_virtual_cores πŸ”—

gauge

gauge.hadoop.cluster.metrics.containers_allocated πŸ”—

gauge

gauge.hadoop.cluster.metrics.containers_pending πŸ”—

gauge

gauge.hadoop.cluster.metrics.containers_reserved πŸ”—

gauge

gauge.hadoop.cluster.metrics.decommissioned_nodes πŸ”—

gauge

gauge.hadoop.cluster.metrics.lost_nodes πŸ”—

gauge

gauge.hadoop.cluster.metrics.rebooted_nodes πŸ”—

gauge

gauge.hadoop.cluster.metrics.reserved_mb πŸ”—

gauge

gauge.hadoop.cluster.metrics.reserved_virtual_cores πŸ”—

gauge

gauge.hadoop.cluster.metrics.total_mb πŸ”—

gauge

gauge.hadoop.cluster.metrics.total_virtual_cores πŸ”—

gauge

gauge.hadoop.cluster.metrics.unhealthy_nodes πŸ”—

gauge

gauge.hadoop.mapreduce.job.elapsedTime πŸ”—

gauge

gauge.hadoop.mapreduce.job.failedMapAttempts πŸ”—

gauge

gauge.hadoop.mapreduce.job.failedReduceAttempts πŸ”—

gauge

gauge.hadoop.mapreduce.job.mapsTotal πŸ”—

gauge

gauge.hadoop.mapreduce.job.successfulMapAttempts πŸ”—

gauge

gauge.hadoop.mapreduce.job.successfulReduceAttempts πŸ”—

gauge

gauge.hadoop.resource.manager.apps.allocatedMB πŸ”—

gauge

gauge.hadoop.resource.manager.apps.allocatedVCores πŸ”—

gauge

gauge.hadoop.resource.manager.apps.clusterUsagePercentage πŸ”—

gauge

gauge.hadoop.resource.manager.apps.memorySeconds πŸ”—

gauge

gauge.hadoop.resource.manager.apps.priority πŸ”—

gauge

gauge.hadoop.resource.manager.apps.progress πŸ”—

gauge

gauge.hadoop.resource.manager.apps.queueUsagePercentage πŸ”—

gauge

gauge.hadoop.resource.manager.apps.runningContainers πŸ”—

gauge

gauge.hadoop.resource.manager.apps.vcoreSeconds πŸ”—

gauge

gauge.hadoop.resource.manager.nodes.availMemoryMB πŸ”—

gauge

gauge.hadoop.resource.manager.nodes.availableVirtualCores πŸ”—

gauge

gauge.hadoop.resource.manager.nodes.numContainers πŸ”—

gauge

gauge.hadoop.resource.manager.nodes.usedMemoryMB πŸ”—

gauge

gauge.hadoop.resource.manager.nodes.usedVirtualCores πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.absoluteCapacity πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.absoluteMaxCapacity πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.absoluteUsedCapacity πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.allocatedContainers πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.capacity πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.maxApplications πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.maxApplicationsPerUser πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.maxCapacity πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.numActiveApplications πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.numApplications πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.numContainers πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.numPendingApplications πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.pendingContainers πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.reservedContainers πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.usedCapacity πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.userLimit πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.leaf.queue.userLimitFactor πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.root.queue.capacity πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.root.queue.maxCapacity πŸ”—

gauge

gauge.hadoop.resource.manager.scheduler.root.queue.usedCapacity πŸ”—

gauge

hadoop.cluster.metrics.active_nodes πŸ”—

gauge

hadoop.cluster.metrics.allocated_mb πŸ”—

gauge

hadoop.cluster.metrics.allocated_virtual_cores πŸ”—

gauge

hadoop.cluster.metrics.apps_completed πŸ”—

gauge

hadoop.cluster.metrics.apps_failed πŸ”—

gauge

hadoop.cluster.metrics.apps_killed πŸ”—

gauge

hadoop.cluster.metrics.apps_pending πŸ”—

gauge

hadoop.cluster.metrics.apps_running πŸ”—

gauge

hadoop.cluster.metrics.apps_submitted πŸ”—

gauge

hadoop.cluster.metrics.available_mb πŸ”—

gauge

hadoop.cluster.metrics.available_virtual_cores πŸ”—

gauge

hadoop.cluster.metrics.containers_allocated πŸ”—

gauge

hadoop.cluster.metrics.containers_pending πŸ”—

gauge

hadoop.cluster.metrics.containers_reserved πŸ”—

gauge

hadoop.cluster.metrics.decommissioned_nodes πŸ”—

gauge

hadoop.cluster.metrics.lost_nodes πŸ”—

gauge

hadoop.cluster.metrics.rebooted_nodes πŸ”—

gauge

hadoop.cluster.metrics.reserved_mb πŸ”—

gauge

hadoop.cluster.metrics.reserved_virtual_cores πŸ”—

gauge

hadoop.cluster.metrics.total_mb πŸ”—

counter

hadoop.cluster.metrics.total_nodes πŸ”—

counter

hadoop.cluster.metrics.total_virtual_cores πŸ”—

counter

hadoop.cluster.metrics.unhealthy_nodes πŸ”—

gauge

hadoop.mapreduce.job.elapsedTime πŸ”—

gauge

hadoop.mapreduce.job.failedMapAttempts πŸ”—

gauge

hadoop.mapreduce.job.failedReduceAttempts πŸ”—

gauge

hadoop.mapreduce.job.killedMapAttempts πŸ”—

gauge

hadoop.mapreduce.job.killedReduceAttempts πŸ”—

gauge

hadoop.mapreduce.job.mapsCompleted πŸ”—

gauge

hadoop.mapreduce.job.mapsPending πŸ”—

gauge

hadoop.mapreduce.job.mapsRunning πŸ”—

gauge

hadoop.mapreduce.job.mapsTotal πŸ”—

gauge

hadoop.mapreduce.job.newMapAttempts πŸ”—

gauge

hadoop.mapreduce.job.newReduceAttempts πŸ”—

gauge

hadoop.mapreduce.job.reducesCompleted πŸ”—

gauge

hadoop.mapreduce.job.reducesPending πŸ”—

gauge

hadoop.mapreduce.job.reducesTotal πŸ”—

gauge

hadoop.mapreduce.job.runningMapAttempts πŸ”—

gauge

hadoop.mapreduce.job.runningReduceAttempts πŸ”—

gauge

hadoop.mapreduce.job.successfulMapAttempts πŸ”—

gauge

hadoop.mapreduce.job.successfulReduceAttempts πŸ”—

gauge

hadoop.resource.manager.apps.allocatedMB πŸ”—

gauge

hadoop.resource.manager.apps.allocatedVCores πŸ”—

gauge

hadoop.resource.manager.apps.clusterUsagePercentage πŸ”—

gauge

hadoop.resource.manager.apps.memorySeconds πŸ”—

gauge

hadoop.resource.manager.apps.numAMContainerPreempted πŸ”—

gauge

hadoop.resource.manager.apps.numNonAMContainerPreempted πŸ”—

gauge

hadoop.resource.manager.apps.preemptedResourceMB πŸ”—

gauge

hadoop.resource.manager.apps.preemptedResourceVCores πŸ”—

gauge

hadoop.resource.manager.apps.priority πŸ”—

gauge

hadoop.resource.manager.apps.progress πŸ”—

gauge

hadoop.resource.manager.apps.queueUsagePercentage πŸ”—

gauge

hadoop.resource.manager.apps.runningContainers πŸ”—

gauge

hadoop.resource.manager.apps.vcoreSeconds πŸ”—

gauge

hadoop.resource.manager.node.nodeCPUUsage πŸ”—

gauge

hadoop.resource.manager.node.nodePhysicalMemoryMB πŸ”—

gauge

hadoop.resource.manager.node.nodeVirtualMemoryMB πŸ”—

gauge

hadoop.resource.manager.nodes.availMemoryMB πŸ”—

gauge

hadoop.resource.manager.nodes.availableVirtualCores πŸ”—

gauge

hadoop.resource.manager.nodes.numContainers πŸ”—

gauge

hadoop.resource.manager.nodes.usedMemoryMB πŸ”—

gauge

hadoop.resource.manager.nodes.usedVirtualCores πŸ”—

gauge

hadoop.resource.manager.scheduler.fifo.availNodeCapacity πŸ”—

gauge

hadoop.resource.manager.scheduler.fifo.capacity πŸ”—

gauge

hadoop.resource.manager.scheduler.fifo.maxQueueMemoryCapacity πŸ”—

gauge

hadoop.resource.manager.scheduler.fifo.minQueueMemoryCapacity πŸ”—

gauge

hadoop.resource.manager.scheduler.fifo.numContainers πŸ”—

gauge

hadoop.resource.manager.scheduler.fifo.numNodes πŸ”—

gauge

hadoop.resource.manager.scheduler.fifo.totalNodeCapacity πŸ”—

gauge

hadoop.resource.manager.scheduler.fifo.usedCapacity πŸ”—

gauge

hadoop.resource.manager.scheduler.fifo.usedNodeCapacity πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.absoluteCapacity πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.absoluteMaxCapacity πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.absoluteUsedCapacity πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.allocatedContainers πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.capacity πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.maxActiveApplications πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.maxActiveApplicationsPerUser πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.maxApplications πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.maxApplicationsPerUser πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.maxCapacity πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.numActiveApplications πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.numApplications πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.numContainers πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.numPendingApplications πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.pendingContainers πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.reservedContainers πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.usedCapacity πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.userLimit πŸ”—

gauge

hadoop.resource.manager.scheduler.leaf.queue.userLimitFactor πŸ”—

gauge

hadoop.resource.manager.scheduler.queue.resource.memory πŸ”—

gauge

hadoop.resource.manager.scheduler.queue.resource.vCores πŸ”—

gauge

hadoop.resource.manager.scheduler.queue.users.numActiveApplications πŸ”—

gauge

hadoop.resource.manager.scheduler.queue.users.numPendingApplications πŸ”—

gauge

hadoop.resource.manager.scheduler.root.queue.capacity πŸ”—

gauge

hadoop.resource.manager.scheduler.root.queue.maxCapacity πŸ”—

gauge

hadoop.resource.manager.scheduler.root.queue.usedCapacity πŸ”—

gauge

These are the metrics available for this monitor. Metrics that are categorized as container/host (default) are in bold and italics in the list below.

  • counter.hadoop.cluster.metrics.total_mb (cumulative)
  • counter.hadoop.cluster.metrics.total_nodes (cumulative)
  • counter.hadoop.cluster.metrics.total_virtual_cores (cumulative)
  • gauge.hadoop.cluster.metrics.active_nodes (gauge)
  • gauge.hadoop.cluster.metrics.allocated_mb (gauge)
  • gauge.hadoop.cluster.metrics.allocated_virtual_cores (gauge)
  • gauge.hadoop.cluster.metrics.apps_completed (gauge)
  • gauge.hadoop.cluster.metrics.apps_failed (gauge)
  • gauge.hadoop.cluster.metrics.apps_killed (gauge)
  • gauge.hadoop.cluster.metrics.apps_pending (gauge)
  • gauge.hadoop.cluster.metrics.apps_running (gauge)
  • gauge.hadoop.cluster.metrics.apps_submitted (gauge)
  • gauge.hadoop.cluster.metrics.available_mb (gauge)
  • gauge.hadoop.cluster.metrics.available_virtual_cores (gauge)
  • gauge.hadoop.cluster.metrics.containers_allocated (gauge)
  • gauge.hadoop.cluster.metrics.containers_pending (gauge)
  • gauge.hadoop.cluster.metrics.containers_reserved (gauge)
  • gauge.hadoop.cluster.metrics.decommissioned_nodes (gauge)
  • gauge.hadoop.cluster.metrics.lost_nodes (gauge)
  • gauge.hadoop.cluster.metrics.rebooted_nodes (gauge)
  • gauge.hadoop.cluster.metrics.reserved_mb (gauge)
  • gauge.hadoop.cluster.metrics.reserved_virtual_cores (gauge)
  • gauge.hadoop.cluster.metrics.total_mb (gauge)
  • gauge.hadoop.cluster.metrics.total_virtual_cores (gauge)
  • gauge.hadoop.cluster.metrics.unhealthy_nodes (gauge)
  • gauge.hadoop.mapreduce.job.elapsedTime (gauge)
  • gauge.hadoop.mapreduce.job.failedMapAttempts (gauge)
  • gauge.hadoop.mapreduce.job.failedReduceAttempts (gauge)
  • gauge.hadoop.mapreduce.job.mapsTotal (gauge)
  • gauge.hadoop.mapreduce.job.successfulMapAttempts (gauge)
  • gauge.hadoop.mapreduce.job.successfulReduceAttempts (gauge)
  • gauge.hadoop.resource.manager.apps.allocatedMB (gauge)
  • gauge.hadoop.resource.manager.apps.allocatedVCores (gauge)
  • gauge.hadoop.resource.manager.apps.clusterUsagePercentage (gauge)
  • gauge.hadoop.resource.manager.apps.memorySeconds (gauge)
  • gauge.hadoop.resource.manager.apps.priority (gauge)
  • gauge.hadoop.resource.manager.apps.progress (gauge)
  • gauge.hadoop.resource.manager.apps.queueUsagePercentage (gauge)
  • gauge.hadoop.resource.manager.apps.runningContainers (gauge)
  • gauge.hadoop.resource.manager.apps.vcoreSeconds (gauge)
  • gauge.hadoop.resource.manager.nodes.availMemoryMB (gauge)
  • gauge.hadoop.resource.manager.nodes.availableVirtualCores (gauge)
  • gauge.hadoop.resource.manager.nodes.numContainers (gauge)
  • gauge.hadoop.resource.manager.nodes.usedMemoryMB (gauge)
  • gauge.hadoop.resource.manager.nodes.usedVirtualCores (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.absoluteCapacity (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.absoluteMaxCapacity (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.absoluteUsedCapacity (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.allocatedContainers (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.capacity (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.maxApplications (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.maxApplicationsPerUser (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.maxCapacity (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.numActiveApplications (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.numApplications (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.numContainers (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.numPendingApplications (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.pendingContainers (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.reservedContainers (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.usedCapacity (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.userLimit (gauge)
  • gauge.hadoop.resource.manager.scheduler.leaf.queue.userLimitFactor (gauge)
  • gauge.hadoop.resource.manager.scheduler.root.queue.capacity (gauge)
  • gauge.hadoop.resource.manager.scheduler.root.queue.maxCapacity (gauge)
  • gauge.hadoop.resource.manager.scheduler.root.queue.usedCapacity (gauge)

Group applications πŸ”—

All of the following metrics are part of the applications metric group. All of the non-default metrics below can be turned on by adding applications to the monitor config option extraGroups:

  • hadoop.resource.manager.apps.allocatedMB (gauge)
  • hadoop.resource.manager.apps.allocatedVCores (gauge)
  • hadoop.resource.manager.apps.clusterUsagePercentage (gauge)
  • hadoop.resource.manager.apps.memorySeconds (gauge)
  • hadoop.resource.manager.apps.numAMContainerPreempted (gauge)
  • hadoop.resource.manager.apps.numNonAMContainerPreempted (gauge)
  • hadoop.resource.manager.apps.preemptedResourceMB (gauge)
  • hadoop.resource.manager.apps.preemptedResourceVCores (gauge)
  • hadoop.resource.manager.apps.priority (gauge)
  • hadoop.resource.manager.apps.progress (gauge)
  • hadoop.resource.manager.apps.queueUsagePercentage (gauge)
  • hadoop.resource.manager.apps.runningContainers (gauge)
  • hadoop.resource.manager.apps.vcoreSeconds (gauge)

Group cluster πŸ”—

All of the following metrics are part of the cluster metric group. All of the non-default metrics below can be turned on by adding cluster to the monitor config option extraGroups:

  • hadoop.cluster.metrics.active_nodes (gauge)
  • hadoop.cluster.metrics.allocated_mb (gauge)
  • hadoop.cluster.metrics.allocated_virtual_cores (gauge)
  • hadoop.cluster.metrics.apps_completed (gauge)
  • hadoop.cluster.metrics.apps_failed (gauge)
  • hadoop.cluster.metrics.apps_killed (gauge)
  • hadoop.cluster.metrics.apps_pending (gauge)
  • hadoop.cluster.metrics.apps_running (gauge)
  • hadoop.cluster.metrics.apps_submitted (gauge)
  • hadoop.cluster.metrics.available_mb (gauge)
  • hadoop.cluster.metrics.available_virtual_cores (gauge)
  • hadoop.cluster.metrics.containers_allocated (gauge)
  • hadoop.cluster.metrics.containers_pending (gauge)
  • hadoop.cluster.metrics.containers_reserved (gauge)
  • hadoop.cluster.metrics.decommissioned_nodes (gauge)
  • hadoop.cluster.metrics.lost_nodes (gauge)
  • hadoop.cluster.metrics.rebooted_nodes (gauge)
  • hadoop.cluster.metrics.reserved_mb (gauge)
  • hadoop.cluster.metrics.reserved_virtual_cores (gauge)
  • hadoop.cluster.metrics.total_mb (counter)
  • hadoop.cluster.metrics.total_nodes (counter)
  • hadoop.cluster.metrics.total_virtual_cores (counter)
  • hadoop.cluster.metrics.unhealthy_nodes (gauge)

Group fifo-scheduler πŸ”—

All of the following metrics are part of the fifo-scheduler metric group. All of the non-default metrics below can be turned on by adding fifo-scheduler to the monitor config option extraGroups:

  • hadoop.resource.manager.scheduler.fifo.availNodeCapacity (gauge)
  • hadoop.resource.manager.scheduler.fifo.capacity (gauge)
  • hadoop.resource.manager.scheduler.fifo.maxQueueMemoryCapacity (gauge)
  • hadoop.resource.manager.scheduler.fifo.minQueueMemoryCapacity (gauge)
  • hadoop.resource.manager.scheduler.fifo.numContainers (gauge)
  • hadoop.resource.manager.scheduler.fifo.numNodes (gauge)
  • hadoop.resource.manager.scheduler.fifo.totalNodeCapacity (gauge)
  • hadoop.resource.manager.scheduler.fifo.usedCapacity (gauge)
  • hadoop.resource.manager.scheduler.fifo.usedNodeCapacity (gauge)

Group leaf-queue πŸ”—

All of the following metrics are part of the leaf-queue metric group. All of the non-default metrics below can be turned on by adding leaf-queue to the monitor config option extraGroups:

  • hadoop.resource.manager.scheduler.leaf.queue.absoluteCapacity (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.absoluteMaxCapacity (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.absoluteUsedCapacity (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.allocatedContainers (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.capacity (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.maxActiveApplications (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.maxActiveApplicationsPerUser (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.maxApplications (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.maxApplicationsPerUser (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.maxCapacity (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.numActiveApplications (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.numApplications (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.numContainers (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.numPendingApplications (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.pendingContainers (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.reservedContainers (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.usedCapacity (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.userLimit (gauge)
  • hadoop.resource.manager.scheduler.leaf.queue.userLimitFactor (gauge)

Group mapreduce-jobs πŸ”—

All of the following metrics are part of the mapreduce-jobs metric group. All of the non-default metrics below can be turned on by adding mapreduce-jobs to the monitor config option extraGroups:

  • hadoop.mapreduce.job.elapsedTime (gauge)
  • hadoop.mapreduce.job.failedMapAttempts (gauge)
  • hadoop.mapreduce.job.failedReduceAttempts (gauge)
  • hadoop.mapreduce.job.killedMapAttempts (gauge)
  • hadoop.mapreduce.job.killedReduceAttempts (gauge)
  • hadoop.mapreduce.job.mapsCompleted (gauge)
  • hadoop.mapreduce.job.mapsPending (gauge)
  • hadoop.mapreduce.job.mapsRunning (gauge)
  • hadoop.mapreduce.job.mapsTotal (gauge)
  • hadoop.mapreduce.job.newMapAttempts (gauge)
  • hadoop.mapreduce.job.newReduceAttempts (gauge)
  • hadoop.mapreduce.job.reducesCompleted (gauge)
  • hadoop.mapreduce.job.reducesPending (gauge)
  • hadoop.mapreduce.job.reducesTotal (gauge)
  • hadoop.mapreduce.job.runningMapAttempts (gauge)
  • hadoop.mapreduce.job.runningReduceAttempts (gauge)
  • hadoop.mapreduce.job.successfulMapAttempts (gauge)
  • hadoop.mapreduce.job.successfulReduceAttempts (gauge)

Group node-resources πŸ”—

All of the following metrics are part of the node-resources metric group. All of the non-default metrics below can be turned on by adding node-resources to the monitor config option extraGroups:

  • hadoop.resource.manager.node.nodeCPUUsage (gauge)
  • hadoop.resource.manager.node.nodePhysicalMemoryMB (gauge)
  • hadoop.resource.manager.node.nodeVirtualMemoryMB (gauge)

Group nodes πŸ”—

All of the following metrics are part of the nodes metric group. All of the non-default metrics below can be turned on by adding nodes to the monitor config option extraGroups:

  • hadoop.resource.manager.nodes.availMemoryMB (gauge)
  • hadoop.resource.manager.nodes.availableVirtualCores (gauge)
  • hadoop.resource.manager.nodes.numContainers (gauge)
  • hadoop.resource.manager.nodes.usedMemoryMB (gauge)
  • hadoop.resource.manager.nodes.usedVirtualCores (gauge)

Group queue-users πŸ”—

All of the following metrics are part of the queue-users metric group. All of the non-default metrics below can be turned on by adding queue-users to the monitor config option extraGroups:

  • hadoop.resource.manager.scheduler.queue.users.numActiveApplications (gauge)
  • hadoop.resource.manager.scheduler.queue.users.numPendingApplications (gauge)

Group resource-objects πŸ”—

All of the following metrics are part of the resource-objects metric group. All of the non-default metrics below can be turned on by adding resource-objects to the monitor config option extraGroups:

  • hadoop.resource.manager.scheduler.queue.resource.memory (gauge)
  • hadoop.resource.manager.scheduler.queue.resource.vCores (gauge)

Group root-queue πŸ”—

All of the following metrics are part of the root-queue metric group. All of the non-default metrics below can be turned on by adding root-queue to the monitor config option extraGroups:

  • hadoop.resource.manager.scheduler.root.queue.capacity (gauge)
  • hadoop.resource.manager.scheduler.root.queue.maxCapacity (gauge)
  • hadoop.resource.manager.scheduler.root.queue.usedCapacity (gauge)

Non-default metrics (version 4.7.0+) πŸ”—

The following information applies to the agent version 4.7.0+ that has enableBuiltInFiltering: true set on the top level of the agent config.

To emit metrics that are not default, you can add those metrics in the generic monitor-level extraMetrics config option. Metrics that are derived from specific configuration options that do not appear in the above list of metrics do not need to be added to extraMetrics.

To see a list of metrics that will be emitted you can run agent-status monitors after configuring this monitor in a running agent instance.

Legacy non-default metrics (version < 4.7.0) πŸ”—

The following information only applies to agent version older than 4.7.0. If you have a newer agent and have set enableBuiltInFiltering: true at the top level of your agent config, see the section above. See upgrade instructions in Old-style whitelist filtering.

If you have a reference to the whitelist.json in your agent’s top-level metricsToExclude config option, and you want to emit metrics that are not in that whitelist, then you need to add an item to the top-level metricsToInclude config option to override that whitelist (see Inclusion filtering. Or you can just copy the whitelist.json, modify it, and reference that in metricsToExclude.