Docs » Integrations Guide » Integrations Reference » MongoDB

image0 MongoDB

Metadata associated with the MongoDB collectd plugin can be found here. The relevant code for the plugin can be found here.

DESCRIPTION

Use the mongodb collectd plugin to collect metrics from MongoDB nodes.

This plugin captures the following metrics about MongoDB generally:

  • memory
  • network input/output bytes count
  • heap usage
  • db connections
  • operations count
  • active client connections
  • queued operations

The plugin also captures the following DB-specific metrics:

  • db size
  • db counters

Documentation for MongoDB can be found here.

FEATURES

Built-in dashboards

  • MongoDB Hosts: Overview of data from all MongoDB hosts.

image1

  • MongoDB Host: Focus on a single MongoDB host.

image2

  • MongoDB Cluster: Overview of a MongoDB cluster.

REQUIREMENTS AND DEPENDENCIES

Version information

Software Version
collectd 4.9 or later
Python 2.4 or later
MongoDB 2.6 or later
PyMongo 3.0 or later
Python plugin for collectd (included with SignalFx collectd agent)

INSTALLATION

If you are using the new Smart Agent, see the docs for thecollectd/mongodbmonitor for more information. The configuration documentation below may be helpful as well, but consult the Smart Agent repo’s docs for the exact schema.

  1. Install pip and pymongo.

    • RHEL/CentOS and Amazon Linux

      yum install -y epel-release
      yum install -y python-pip
      sudo pip install pymongo==3.0.3
      
    • Ubuntu and Debian:

      apt-get install -y python-pip python-dev build-essential
      sudo pip install pymongo==3.0.3
      
  2. If you want to use SSL/TLS to connect to Mongo, install the PyMongo TLS
    dependencies as well:
    sudo pip install pymongo[tls]
    
  3. Download the collectd-mongodb Python module.

  4. Download SignalFx’s sample configuration file for this plugin to /etc/collectd/managed_config.

  5. Modify the sample configuration file as described in Configuration, below.

  6. Restart collectd.

CONFIGURATION

Using the example configuration file 10-mongodb.conf as a guide, provide values for the configuration options listed below that make sense for your environment and allow you to connect to the MongoDB instance to be monitored.

configuration option definition default value
ModulePath Path on disk where collectd can find this module. “/usr/share/collectd/mongodb-collectd-plugin”
Host Host IP “127.0.0.1”
Port Port number for IP connection “27017”
User Valid mongodb user “”
Password Associated password for valid user “password”
Database Name(s) of database(s) that you would like metrics from. Note: the first database in this list must be “admin”, as it is used to perform a serverStatus() command. “admin” “db-prod” “db-dev”
Interval How frequently to send metrics in seconds collectd Interval setting
SendCollectionMetrics Whether to send collection level metrics or not false
SendCollectionTopMetrics Whether to send collection level top (timing) metrics or not false
CollectionMetricsIntervalMultiplier How frequently to send collection level metrics as a multiple of the configured plugin interval (e.g. if the Interval is 15 and the multiplier is 4, collection level metrics will be fetched every minute) 6
UseTLS Set this to true if you want to connect to Mongo using TLS/x509/SSL. false
CACerts Path to a CA cert that will be used to verify the certificate that Mongo presents (not needed if not using TLS or if Mongo’s cert is signed by a globally trusted issuer already installed in the default location on your OS) “”
TLSClientCert Path to a client certificate (not needed unless your Mongo instance requires x509 client verification) “”
User (same as above ``User`` field but required for TLS client auth) The username associated with the client cert (this is the subject field of the client cert, formatted according to RFC2253 and in the same order that was specified when creating the user in the Mongo ``$external`` database). You can get this value by running the following command: openssl x509 -in <pathToClient PEM> -inform PEM -subject -nameopt RFC2253. “”
TLSClientKey Path to a client certificate key (not needed unless your Mongo instance requires x509 client verification, or if your client cert above has the key included) “”
TLSClientKeyPassphrase Passphrase for the TLSClientKey above (requires Python 2.7.9+) “”

Note: Monitoring multiple instances

Each MongoDB instance to be monitored is specified in a <Module> block within the configuration file. By default, the sample configuration file 10-mongodb.conf contains only one such block. To monitor an additional instance of MongoDB, add another <Module> block immediately below the first, and configure it according to that instance’s parameters.

Note: Creating a MongoDB user for collectd

If you’re monitoring a secured MongoDB deployment, it is a good practice to create a MongoDB user with minimal read-only roles, as follows:

db.createUser( {
  user: "collectd",
  pwd: "collectd",
  roles: [ { role: "readAnyDatabase", db: "admin" }, { role: "clusterMonitor", db: "admin" } ]
});

USAGE

Below are screen captures of dashboards created for this plugin by SignalFx, illustrating the metrics emitted by this plugin.

For general reference on how to monitor MongoDB performance, see Analyzing MongoDBPerformance.

Monitoring MongoDB clusters

Writes to MongoDB require the use of the global lock. If lock utilization is high, operations can begin to slow down. This can be a symptom of database issues such as poorly configured or absent indexes, or a schema design that needs improvement. It can also indicate the failure of a disk. Monitor the number of readers and writers waiting for the lock in gauge.globalLock.currentQueue.total.

image3

This lock has little utilization and few queued readers and writers.

MongoDB flushes data changes from memory to disk on a timed interval, by default every 60 seconds. If background flushes begin taking longer than usual, it can indicate that the disk doesn’t have enough I/O capacity to handle the load (read more below). It could also reflect a large number of writes occurring at once – check counter.opcounters.insert and counter.opcounters.update.

Monitor average background flush time and the most recent background flush time in gauge.backgroundFlushing.average_ms and gauge.backgroundFlushing.last_ms respectively.

image4

Average background flush time on this cluster is around 40ms, well within healthy parameters.

When analyzing the performance of a MongoDB cluster, it’s also important to verify that the load is balanced across each instance. The cluster dashboard included in this repository contains many list charts of individual MongoDB instances ordered by important metrics like requests per second (counter.network.numRequests) and number of connections to MongoDB (gauge.connections.current). This can help you compare load between instances. Load imbalance can arise in a sharded cluster if MongoDB is unable to balance chunks equally between the shards, for example if lock utilization is high.

image5

All the listed instances show about the same requests per second and number of connections. Their load is balanced.

Monitoring MongoDB hosts

On an individual instance level, it’s important to monitor system statistics like memory usage, page faults, and disk I/O utilization.

MongoDB uses memory-mapped files to store data, so it is important to compare the amount of memory that MongoDB has allocated to the amount of system memory. This plugin reports resident memory usage in gauge.mem.resident and mapped memory usage in gauge.mem.mapped. If either of these quantities exceed the amount of system memory (reported by the memory plugin for collectd), the system may be at or over capacity.

image6

This MongoDB instance is not using a large amount of resident memory, and has non-mapped memory available to the process (calculated as gauge.mem.virtual - gauge.mem.mapped.)

This plugin reports page faults in counter.extra_info.page_faults. Page faults indicate that reads or writes are occurring to data files that are not currently in memory. This is different from an OS page fault. Sudden increases in MongoDB page faults can indicate that a large read operation is taking place. Steadily high numbers of page faults indicate that MongoDB is reading more often from disk than is optimal.

image7

This MongoDB instance has a low rate of page faults. This means that most of the data MongoDB needs to access is in memory, and doesn’t need to be fetched from disk.

You can monitor disk I/O utilization for your MongoDB host using the disk_ops.write and disk_ops.read metrics emitted by the disk plugin for collectd, which is included and enabled by default in most packages of collectd. Click here to learn more about the collectd-disk plugin.

METRICS

Below is a list of all metrics.

Metric Name Brief Type
counter.backgroundFlushing.flushes Number of times the database has been flushed gauge
counter.collection.commandsCount Number of commands issued for a collection counter
counter.collection.commandsTime Time spent in microseconds processing commands issued for a collection counter
counter.collection.getmoreCount Number of getMore requests issued for a collection counter
counter.collection.getmoreTime Time spent in microseconds processing getMore requests for a collection counter
counter.collection.index.accesses.ops Number of times an index has been used (only on Mongo 3.2+) counter
counter.collection.insertCount Number of inserts issued for a collection counter
counter.collection.insertTime Time spent in microseconds processing insert requests for a collection counter
counter.collection.queriesCount Number of queries issued for a collection counter
counter.collection.queriesTime Time spent in microseconds processing query requests for a collection counter
counter.collection.readLockCount Number of read locks issued for a collection counter
counter.collection.readLockTime Time spent in microseconds processing read locks for a collection counter
counter.collection.removeCount Number of remove requests issued for a collection counter
counter.collection.removeTime Time spent in microseconds processing remove requests for a collection counter
counter.collection.totalCount Total number of operations issued for a collection counter
counter.collection.totalTime Time spent in microseconds processing all operations for a collection counter
counter.collection.updateCount Number of update requests issued for a collection counter
counter.collection.updateTime Time spent in microseconds processing update requests for a collection counter
counter.collection.writeLockCount Number of write locks issued for a collection counter
counter.collection.writeLockTime Time spent in microseconds processing write locks for a collection counter
counter.extra_info.page_faults Mongod page faults gauge
counter.network.bytesIn Network bytes received by the database server gauge
counter.network.bytesOut Network bytes sent by the database server gauge
counter.network.numRequests Requests received by the server cumulative_counter
counter.opcounters.delete Number of deletes per second cumulative counter
counter.opcounters.insert Number of inserts per second cumulative counter
counter.opcounters.query Number of queries per second cumulative counter
counter.opcounters.update Number of updates per second cumulative counter
dimension.plugin_instance Port number of the MongoDB instance  
gauge.backgroundFlushing.average_ms Average time (ms) to write data to disk gauge
gauge.backgroundFlushing.last_ms Most recent time (ms) spent writing data to disk gauge
gauge.collection.avgObjSize Mean object/document size of a collection gauge
gauge.collection.count Number of objects/documents in a collection gauge
gauge.collection.indexSize Size of a particular index on a collection gauge
gauge.collection.max Maximum number of documents in a capped collection gauge
gauge.collection.maxSize Maximum disk usage of a capped collection gauge
gauge.collection.size Size of a collection in bytes, not including indexes gauge
gauge.collection.storageSize Size of the collection on disk in bytes, never decreases. gauge
gauge.collections Number of collections gauge
gauge.connections.available Number of available incoming connections gauge
gauge.connections.current Number of current client connections gauge
gauge.dataSize Total size of data, in bytes gauge
gauge.extra_info.heap_usage_bytes Heap size used by the mongod process, in bytes gauge
gauge.globalLock.activeClients.readers Number of active client connections performing reads gauge
gauge.globalLock.activeClients.total Total number of active client connections gauge
gauge.globalLock.activeClients.writers Number of active client connections performing writes gauge
gauge.globalLock.currentQueue.readers Read operations currently in queue gauge
gauge.globalLock.currentQueue.total Total operations currently in queue gauge
gauge.globalLock.currentQueue.writers Write operations currently in queue gauge
gauge.indexSize Total size of indexes, in bytes gauge
gauge.indexes Number of indexes across all collections gauge
gauge.mem.mapped Mongodb mapped memory usage, in MB gauge
gauge.mem.resident Mongodb resident memory usage, in MB gauge
gauge.mem.virtual Mongodb virtual memory usage, in MB gauge
gauge.objects Number of documents across all collections gauge
gauge.storageSize Total bytes allocated to collections for document storage gauge
gauge.uptime Uptime of this server in milliseconds counter

counter.backgroundFlushing.flushes

gauge

Number of times the database has been flushed.

counter.collection.commandsCount

counter

Number of commands issued for a collection

counter.collection.commandsTime

counter

Time spent in microseconds processing commands issued for a collection

counter.collection.getmoreCount

counter

Number of getMore requests issued for a collection

counter.collection.getmoreTime

counter

Time spent in microseconds processing getMore requests for a collection

counter.collection.index.accesses.ops

counter

Number of times an index has been used (only on Mongo 3.2+)

counter.collection.insertCount

counter

Number of inserts issued for a collection

counter.collection.insertTime

counter

Time spent in microseconds processing insert requests for a collection

counter.collection.queriesCount

counter

Number of queries issued for a collection

counter.collection.queriesTime

counter

Time spent in microseconds processing query requests for a collection

counter.collection.readLockCount

counter

Number of read locks issued for a collection

counter.collection.readLockTime

counter

Time spent in microseconds processing read locks for a collection

counter.collection.removeCount

counter

Number of remove requests issued for a collection

counter.collection.removeTime

counter

Time spent in microseconds processing remove requets for a collection

counter.collection.totalCount

counter

Total number of operations issued for a collection

counter.collection.totalTime

counter

Time spent in microseconds processing all operations for a collection

counter.collection.updateCount

counter

Number of update requests issued for a collection

counter.collection.updateTime

counter

Time spent in microseconds processing update requets for a collection

counter.collection.writeLockCount

counter

Number of write locks issued for a collection

counter.collection.writeLockTime

counter

Time spent in microseconds processing write locks for a collection

counter.extra_info.page_faults

gauge

Total number of page faults that require disk operations.

counter.network.bytesIn

gauge

Amount of network traffic in bytes received by the database server.

counter.network.bytesOut

gauge

Amount of network traffic in bytes sent by the database server.

counter.network.numRequests

cumulative_counter

Total number of unique requests received by the server.

counter.opcounters.delete

cumulative counter

Total number of delete operations per second.

counter.opcounters.insert

cumulative counter

Total number of insert operations per second.

counter.opcounters.query

cumulative counter

Total number of query operations per second.

counter.opcounters.update

cumulative counter

Total number of update operations per second.

dimension.plugin_instance

None

This dimension indicates the port number of the MongoDB instance being monitored.

gauge.backgroundFlushing.average_ms

gauge

Average time (ms) that the mongod process spent writing data to disk.

gauge.backgroundFlushing.last_ms

gauge

Time(ms) that the mongod process last spent writing data to disk.

gauge.collection.avgObjSize

gauge

Mean object/document size of a collection

gauge.collection.count

gauge

Number of objects/documents in a collection

gauge.collection.indexSize

gauge

Size of a particular index on a collection

gauge.collection.max

gauge

Maximum number of documents in a capped collection

gauge.collection.maxSize

gauge

Maximum disk usage of a capped collection

gauge.collection.size

gauge

Size of a collection in bytes, not including indexes

gauge.collection.storageSize

gauge

Size of the collection on disk in bytes, never decreases.

gauge.collections

gauge

Number of collections in the database.

gauge.connections.available

gauge

Number of unused available incoming connections to the database.

gauge.connections.current

gauge

Number of current connections to the database server from clients, including shell commands.

gauge.dataSize

gauge

The total size in bytes of the data held in the database.

gauge.extra_info.heap_usage_bytes

gauge

The total size in bytes of heap space used by the database process.

gauge.globalLock.activeClients.readers

gauge

Number of active client connections performing read operation.

gauge.globalLock.activeClients.total

gauge

Number of active client connections to the database.

gauge.globalLock.activeClients.writers

gauge

Number of active client connections performing write operation.

gauge.globalLock.currentQueue.readers

gauge

Number of operations queued for the read lock.

gauge.globalLock.currentQueue.total

gauge

Number of operations queued for the lock.

gauge.globalLock.currentQueue.writers

gauge

Number of operations queued for the write lock.

gauge.indexSize

gauge

The total size in bytes of all indexes created on the database.

gauge.indexes

gauge

Total number of indexes across all collections in the database.

gauge.mem.mapped

gauge

Amount of mapped memory (MB) used by the database.

gauge.mem.resident

gauge

Amount of RAM (MB) currently used by the database process.

gauge.mem.virtual

gauge

Virtual memory (MB) currently used by the mongod process.

gauge.objects

gauge

Number of documents in the database across all collections.

gauge.storageSize

gauge

The total amount of space in bytes allocated to collections in the database for document storage.

gauge.uptime

counter

The number of milliseconds since this server was last restarted.