Docs » µAPM Deployment Guide » Monitoring the SignalFx Smart Gateway

Monitoring the SignalFx Smart Gateway 🔗

Monitoring your Smart Gateway is paramount to ensure its correct operation and that you have deployed the appropriate Smart Gateway capacity to handle your workload.

Configuring the SignalFx Smart Agent to monitor the Smart Gateway 🔗

For dashboards to fully populate with all the data you need to monitor the health and resource utilization of your Smart Gateway, you need to deploy the SignalFx Smart Agent on your Smart Gateway instances. This makes sure that the relevant metrics are being reported to SignalFx.

When configuring the Smart Agent on your gateway instances, pay attention to the following settings:

  • Make sure the hostname used by the agent matches the ServerName used by the Smart Gateway.
  • Enable the collectd/signalfx-metadata monitor, and configure it to report its metrics with two additional dimensions: a source: gateway dimension, and a cluster dimension whose value matches the ClusterName used by the Smart Gateway.

Finally, configure the following monitors to provide the appropriate metrics for the Smart Gateway monitoring dashboards:

hostname: REPLACE-WITH-SERVER-NAME
signalFxAccessToken: YOUR_SIGNALFX_API_TOKEN
ingestUrl: https://ingest.YOUR_SIGNALFX_REALM.signalfx.com

monitors:
  - type: host-metadata
  - type: collectd/cpu
  - type: collectd/df
  - type: collectd/disk
  - type: collectd/interface
  - type: collectd/load
  - type: collectd/memory
  - type: collectd/vmem
  - type: collectd/signalfx-metadata
    extraDimensions:
      source: gateway
      cluster: REPLACE-WITH-CLUSTER-NAME

Metrics emitted by the SignalFx Smart Gateway 🔗

In addition to the metrics listed in APM identities and SignalFx metrics, we also emit the metrics listed below. All metrics sent by the Smart Gateway have a host dimension matching the configured ServerName, as well as the source:gateway dimension.

Metric Name Additional Dimensions Description
gateway.commit samplerCommit: the sampler’s commit SHA, gatewayCommit: the gateway’s commit SHA Gauge with a constant value of 1 for the SHAs of the components that make up the Smart Gateway.
gateway.processedTraces none Cumulative counter of all traces processed by this gateway or cluster
gateway.processedSpans none Cumulative counter of all spans processed by this gateway or cluster
gateway.sentTraces none Cumulative counter of all traces that were selected by the Smart Gateway and sent to SignalFx
gateway.sentSpans none Cumulative counter of all spans that were selected by the Smart Gateway and sent to SignalFx
dropped_spans reason: the reason the span was dropped Cumulative counter of all spans dropped by the Smart Gateway
dropped_traces reason: the reason the trace was dropped Cumulative counter of all traces dropped by the Smart Gateway

Built-in Smart Gateway dashboards 🔗

SignalFx provides three built-in, curated dashboards for the Smart Gateway. They will appear in your organization soon after you deploy the Smart Gateway.

../../_images/smart-gateway-db-group.png
  • Cluster(s) — cluster-level view of the utilization and activity of the Smart Gateway clusters
  • Internals — detailed metrics about the internal state of the SignalFx Smart Gateway (ringbuffer sizes, caches, etc.)
  • Instance — instance-level view of the utilization and activity of the Smart Gateway, filtered by host

To open a dashboard, click the Smart Gateway dashboard group name on the Dashboards page and then select the dashboard to display.

../../_images/clusters-dashboard.png

By default, these dashboards show a time range of the past 15 minutes. You can adjust the time range as on any dashboard.

Important

As with all built-in dashboards, you cannot edit the dashboards described above. However, you can select Save as from a dashboard’s Actions menu, place an editable copy in another dashboard group, then edit the dashboard to meet your organization’s specific requirements.

Important

The standard recommended deployment model for APM involves running the SignalFx Smart Agent on each host that runs traced applications. For the Internals and Instance dashboards, you must select a Smart Gateway host (or wildcard “*” for all hosts), or no data will appear.

Smart Gateway detector templates 🔗

As discussed in Creating a detector from a built-in template, creating detectors from built-in templates provides an easy way to set alert conditions in SignalFx. Many of the charts in the Smart Gateway dashboards described above have associated templates you can use as a starting point for a related detector.

The following table summarizes the available templates. Templates are shown on the Detector menu (bell icon) for a chart only if it relates to that chart.

Template name Trigger summary Notes
APM Dead Smart Gateway Detector One of your Smart Gateway instances goes down  
APM Identities Detector Sudden change in number of APM identities Identities should usually be stable, without sudden numbers of new services coming up or going down.
APM Multi-Version Detector There is a version mismatch in your deployed Smart Gateway versions Use the Smart Gateway Version chart on the Cluster(s) dashboard to see which versions of the Smart Gateway are deployed.
APM Smart Gateway Memory Utilization Detector Major (orange) alert when memory.utilization is above 75 for several minutes; Critical (red) alert when memory.utilization is above 90 several seconds. These values map to the colors configured in the Gateway Saturation chart in the Cluster(s) dashboard
APM Zero TAPM Detector Your Smart Gateway cluster stops receiving traces, possibly due to a problem upstream This condition is reflected in the data table of the TAPM per cluster chart in the Cluster(s) dashboard.
APM Smart Gateway Inconsistent Cluster Size Detector Your Smart Gateway instances are reporting inconsistent cluster sizes for more than five minutes. This condition indicates that some nodes are having difficulty reaching other nodes in the cluster. To resolve the issue, restart any nodes that are reporting the incorrect cluster size.