Docs » Integrations Guide » Integrations Reference » HAProxy

image0 HAProxy

Metadata associated with the HAProxy collectd plugin can be found here. The relevant code for the plugin can be found here.

DESCRIPTION

Use the collectd-haproxy collectd plugin to collect metrics about HaProxy.

REQUIREMENTS AND DEPENDENCIES

Software Version
collectd 4.9 or later
haproxy 1.5 or later

INSTALLATION

If you are using the new Smart Agent, see the docs for thecollectd/haproxymonitor for more information. The configuration documentation below may be helpful as well, but consult the Smart Agent repo’s docs for the exact schema.

  1. Download the collectd-haproxy-plugin git repo to /usr/share/collectd/collectd-haproxy

  2. Download SignalFx’s sample configuration file for this plugin to /etc/collectd/managed_config.

  3. Modify the sample configuration file as described in Configuration, below.

  4. SELINUX ONLY Create a SELinux policy package using the supplied type enforcement file. Enter the commands below to create and install the policy package.

    $ cd /usr/share/collectd/collectd-haproxy/selinux
    $ checkmodule -M -m -o collectd-haproxy.mod collectd-haproxy.te
    checkmodule:  loading policy configuration from collectd-haproxy.te
    checkmodule:  policy configuration loaded
    checkmodule:  writing binary representation (version 17) to collectd-haproxy.mod
    $ semodule_package -o collectd-haproxy.pp -m collectd-haproxy.mod
    $ sudo semodule -i collectd-haproxy.pp
    $ sudo reboot
    
  5. Restart collectd.

CONFIGURATION

Using the example configuration file 10-haproxy.conf as a guide, provide values for the configuration options listed below that make sense for your environment and allow you to connect to the HAProxy instance to be monitored.

Configuration Option Definition Example Value
Socket Location of the HAProxy socket file. The default socket is /var/run/haproxy.sock Socket “/var/run/haproxy.sock”
ProxyMonitor A list of all the pxname(s) or svname(s) that you want to monitor ProxyMonitor “http-in”ProxyMonitor “server1”ProxyMonitor “backend”
Interval Interval between collectd to get HAProxy metrics. 10
EnhancedMetrics Enable sending all metrics, not just the default metrics. Defaults to false EnhancedMetrics ‘True’
ExcludeMetric Do not send a specific metric. Metric names can be found in the docs repo ExcludeMetric ‘response_1xx’

The location of the HAProxy socket file is defined in the HAProxy config file, as in the following example:

global
    daemon
    stats socket /var/run/haproxy.sock
    stats timeout 2m

Note: it is possible to use a tcp socket for stats in HAProxy. Users will first need to define in their collectd-haproxy plugin config file the tcp address for the socket, for example localhost:9000, and then in the haproxy.cfg file change the stats socket to listen on the same address

global
    daemon
    stats socket localhost:9000
    stats timeout 2m

For a more restricted tcp socket, a backend server can be defined to listen to stats on localhost. A frontend proxy can use the backend server on a different port, with ACLs to restrict access. See below for example.

global
    daemon
    stats socket localhost:9000
    stats timeout 2m

backend stats-backend
    mode tcp
    server stats-localhost localhost:9000

frontend stats-frontend
    bind *:9001
    default_backend stats-backend
    acl ...
    acl ...

USAGE

Default metrics

  • HAProxy Overview
    • Connection Rate - Total number of connections per second being made.
    • Requests per Second - Number of incoming requests per second.
    • Idle Percent - HAProxy runs on an event loop and waits for new events using poll(). Idle_pct is the ratio of polling time versus total time. Near 100% means load is low, near 0% means the load is very high.
    • Current Sessions - Current number of active sessions in the system. A session is an end-to-end connection.
    • Sessions Rate - Chart showing the rate at which sessions are being created.
    • Top Servers Selected - List of which servers are being selected the most by the load balancer algorithm.
    • Highest Bytes Out - List of which servers are outputting the most data.
    • Average Session Time - List of the average session time over the last 1024 requests
  • HAProxy Frontend
    • Request Rate - The rate of requests being made to the frontend, useful to monitor spikes in traffic.
    • Session Rate - Number of sessions per second being created, also useful to monitor spikes in traffic.
    • Response 2xx, 4xx, 5xx - The ratio between 2xx and 4xx/5xx responses is ideally very low. Watch for spikes in 4xx or 5xx as they may show a misconfiguration or other potential errors.
    • Request Errors - Rate of error requests being made in the system. These may stem from early client termination, read errors from the client, client timeouts, or other various bad requests from the client.
    • Denied Requests - Rate of requests denied because of security concerns. This could be denied requests because they do not satisfy an ACL configuration, or some other deny.
    • Bytes Out - Number of bytes sent out.
    • Bytes In - Number of bytes sent in.
  • HAProxy Backend
    • Current Queue Size - Number of items in the queued requests. If HAProxy hits the configured max number of connections, HAProxy will queue new incoming requests until a server becomes available.
    • Average Response Time - Lists average response time in seconds over the past 1024 requests. Backend must be using mode http, otherwise the metric will report 0.
    • Average Queue Time - The average time a request spends in the queue. This value should be low to keep the system latency low.
    • Top Servers Selected - List of which servers are being selected the most by the load balancer algorithm.
    • Response Errors - The backend error response rate. This metric can be correlated with denied responses and frontend requests to gain insight on potential errors.
    • Server Retries and Redispatches - Server retries occur when a server is not reached the first time. The request will be redispatched to another server if the retry limit is hit.
    • Connection Errors - The rate of connection errors includes both failed backend requests and general backend errors. Correlate with response errors and response codes to track down an issue.
    • Denied Responses - The rate of denied responses by the backend. Most denied responses will come from an ACL and can be correlated with 5xx responses.

METRICS

Enhanced Metrics

Non-default metrics can be enabled in the plugin configuration file, by setting EnhancedMetrics to “True”. Any metric can be excluded from being sent by adding ExcludeMetric “metric_name” in the plugin configuration file. Metric names are found in the docs.

Below is a list of all metrics.

Metric Name Brief Type
counter.connection_total Cumulative number of connections (frontend). This corresponds to HAProxy’s “conn_tot” metric. counter
counter.server_selected_total Number of times a server was selected, either for new sessions or when re-dispatching. This corresponds to HAProxy’s “lbtot” metric. counter
derive.bytes_in Corresponds to HAProxy’s bin metric - Bytes in cumulative counter
derive.bytes_out Corresponds to HAProxy’s bout metric - Bytes out cumulative counter
derive.cli_abrt Corresponds to HAProxy’s cli_abrt metric - Number of data transfers aborted by the client cumulative counter
derive.comp_byp Corresponds to HAProxy’s comp_byp metric - Number of bytes that bypassed the HTTP compressor (CPU/BW limit) cumulative counter
derive.comp_in Corresponds to HAProxy’s comp_in metric - Number of HTTP response bytes fed to the compressor cumulative counter
derive.comp_out Corresponds to HAProxy’s comp_out metric - Number of HTTP response bytes emitted by the compressor cumulative counter
derive.comp_rsp Corresponds to HAProxy’s comp_rsp metric - Number of HTTP responses that were compressed cumulative counter
derive.compress_bps_in Corresponds to HAProxy’s CompressBpsIn metric. cumulative counter
derive.compress_bps_out Corresponds to HAProxy’s CompressBpsOut metric. cumulative counter
derive.connections Corresponds to HAProxy’s CumConns metric. Cumulative number of connections. cumulative counter
derive.denied_request Corresponds to HAProxy’s dreq metric - Requests denied because of security concerns. cumulative counter
derive.denied_response Corresponds to HAProxy’s dresp metric - Responses denied because of security concerns. cumulative counter
derive.downtime Corresponds to HAProxy’s downtime metric - Total downtime (in seconds). The value for the backend cumulative counter
derive.error_connection Corresponds to HAProxy’s econ metric - Number of requests that encountered an error trying to cumulative counter
derive.error_request Corresponds to HAProxy’s ereq metric - Request errors. cumulative counter
derive.error_response Corresponds to HAProxy’s eresp metric - Response errors. srv_abrt will be counted here also. Responses denied because of security concerns. cumulative counter
derive.failed_checks Corresponds to HAProxy’s chkfail metric - Number of failed checks. (Only counts checks failed when cumulative counter
derive.redispatched Corresponds to HAProxy’s wredis metric - Number of times a request was redispatched to another cumulative counter
derive.request_total Corresponds to HAProxy’s req_tot metric - Total number of HTTP requests received cumulative counter
derive.requests Corresponds to HAProxy’s CumReq metric. cumulative counter
derive.response_1xx Corresponds to HAProxy’s hrsp_1xx metric - Http responses with 1xx code cumulative counter
derive.response_2xx Corresponds to HAProxy’s hrsp_2xx metric - Http responses with 2xx code cumulative counter
derive.response_3xx Corresponds to HAProxy’s hrsp_3xx metric - Http responses with 3xx code cumulative counter
derive.response_4xx Corresponds to HAProxy’s hrsp_4xx metric - Http responses with 4xx code cumulative counter
derive.response_5xx Corresponds to HAProxy’s hrsp_5xx metric - Http responses with 5xx code cumulative counter
derive.response_other Corresponds to HAProxy’s hrsp_other metric - Http responses with other codes (protocol error) cumulative counter
derive.retries Corresponds to HAProxy’s wretr metric - Number of times a connection to a server was retried. cumulative counter
derive.session_total Corresponds to HAProxy’s stot metric - Cumulative number of connections cumulative counter
derive.srv_abrt Corresponds to HAProxy’s srv_abrt metric - Number of data transfers aborted by the server cumulative counter
derive.ssl_cache_lookups Corresponds to HAProxy’s SslCacheLookups metric. cumulative counter
derive.ssl_cache_misses Corresponds to HAProxy’s SslCacheMisses metric. cumulative counter
derive.ssl_connections Corresponds to HAProxy’s CumSslConns metric. cumulative counter
derive.uptime_seconds Corresponds to HAProxy’s Uptime_sec metric. cumulative counter
gauge.active_servers Number of active servers. This corresponds to HAProxy’s “act” metric. gauge
gauge.backup_servers Number of backup servers. This corresponds to HAProxy’s “bck” metric. gauge
gauge.check_duration Time in ms took to finish to last health check. This corresponds to HAProxy’s “check_duration” metric. gauge
gauge.connection_rate Number of connections over the last elapsed second (frontend). This corresponds to HAProxy’s “conn_rate” metric. gauge
gauge.connection_rate_max Highest known connection rate. This corresponds to HAProxy’s “conn_rate_max” metric. gauge
gauge.current_connections Current number of connections. Corresponds to HAProxy’s CurrConns metric. gauge
gauge.current_ssl_connections Corresponds to HAProxy’s CurrSslConns metric. gauge
gauge.denied_tcp_connections Requests denied by ‘tcp-request connection’ rules. This corresponds to HAProxy’s “dcon” metric. gauge
gauge.denied_tcp_sessions Requests denied by ‘tcp-request session’ rules. This corresponds to HAProxy’s “dses” metric. gauge
gauge.idle_pct Corresponds to HAProxy’s “Idle_pct” metric. Ratio of system polling time versus total time. gauge
gauge.intercepted_requests Cumulative number of intercepted requests, corresponds to HAProxys metric ‘intercepted’ gauge
gauge.last_session Number of seconds since last session was assigned to server/backend. This corresponds to HAProxy’s “lastsess” metric. gauge
gauge.max_connection_rate Corresponds to HAProxy’s MaxConnRate metric. gauge
gauge.max_connections Corresponds to HAProxy’s MaxConn metric. gauge
gauge.max_pipes Corresponds to HAProxy’s MaxPipes metric. gauge
gauge.max_session_rate Corresponds to HAProxy’s MaxSessRate metric. gauge
gauge.max_ssl_connections Corresponds to HAProxy’s MaxSslConns metric. gauge
gauge.pipes_free Corresponds to HAProxy’s PipesFree metric. gauge
gauge.pipes_used Corresponds to HAProxy’s PipesUsed metric. gauge
gauge.queue_current Corresponds to HAProxy’s qcur metric - Current queued requests. For the backend this reports the gauge
gauge.queue_limit Configured max queue for the server, 0 being no limit. Corresponds to HAProxy’s “qlimit” metric. gauge
gauge.queue_max Max number of queued requests, queue_current, corresponds to HAProxy’s ‘qmax’ metric. gauge
gauge.request_rate Corresponds to HAProxy’s req_rate metric - HTTP requests per second over last elapsed second gauge
gauge.request_rate_max Max number of HTTP requests per second observed. Corresponds to HAProxy’s “req_rate_max” metric. gauge
gauge.run_queue Corresponds to HAProxy’s Run_queue metric. gauge
gauge.session_current Corresponds to HAProxy’s scur metric - Current sessions gauge
gauge.session_rate Corresponds to HAProxy’s rate metric - Number of sessions per second over last elapsed second gauge
gauge.session_rate_limit Configured limit on number of new sessions per second. Corresponds to HAProxy’s “rate_lim” metric. gauge
gauge.session_rate_max Max number of new sessions per second gauge
gauge.session_time_average Average total session time in ms over the last 1024 requests. Corresponds to HAProxy’s “ttime” metric. gauge
gauge.ssl_backend_key_rate Corresponds to HAProxy’s SslBackendKeyRate metric. gauge
gauge.ssl_frontend_key_rate Corresponds to HAProxy’s SslFrontendKeyRate metric. gauge
gauge.ssl_rate Corresponds to HAProxy’s SslRate metric. gauge
gauge.tasks Corresponds to HAProxy’s Tasks metric. gauge
gauge.throttle Corresponds to HAProxy’s throttle metric - Current throttle percentage for the server, when gauge
gauge.zlib_mem_usage Corresponds to HAProxy’s ZlibMemUsage metric. gauge

counter.connection_total

counter

Cumulative number of connections (frontend). This corresponds to HAProxy’s “conn_tot” metric.

counter.server_selected_total

counter

Number of times a server was selected, either for new sessions or when re-dispatching. This corresponds to HAProxy’s “lbtot” metric.

derive.bytes_in

cumulative counter

Corresponds to HAProxy’s bin metric - Bytes in

derive.bytes_out

cumulative counter

Corresponds to HAProxy’s bout metric - Bytes out

derive.cli_abrt

cumulative counter

Corresponds to HAProxy’s cli_abrt metric - Number of data transfers aborted by the client

derive.comp_byp

cumulative counter

Corresponds to HAProxy’s comp_byp metric - Number of bytes that bypassed the HTTP compressor
(CPU/BW limit)

derive.comp_in

cumulative counter

Corresponds to HAProxy’s comp_in metric - Number of HTTP response bytes fed to the compressor

derive.comp_out

cumulative counter

Corresponds to HAProxy’s comp_out metric - Number of HTTP response bytes emitted by the compressor

derive.comp_rsp

cumulative counter

Corresponds to HAProxy’s comp_rsp metric - Number of HTTP responses that were compressed

derive.compress_bps_in

cumulative counter

Corresponds to HAProxy’s CompressBpsIn metric.

derive.compress_bps_out

cumulative counter

Corresponds to HAProxy’s CompressBpsOut metric.

derive.connections

cumulative counter

Corresponds to HAProxy’s CumConns metric. Cumulative number of connections.

derive.denied_request

cumulative counter

Corresponds to HAProxy’s dreq metric - Requests denied because of security concerns.
- For tcp this is because of a matched tcp-request content rule.
- For http this is because of a matched http-request or tarpit rule.

derive.denied_response

cumulative counter

Corresponds to HAProxy’s dresp metric - Responses denied because of security concerns.
- For http this is because of a matched http-request rule, or
“option checkcache”.

derive.downtime

cumulative counter

Corresponds to HAProxy’s downtime metric - Total downtime (in seconds). The value for the backend
is the downtime for the whole backend, not the sum of the server downtime.

derive.error_connection

cumulative counter

Corresponds to HAProxy’s econ metric - Number of requests that encountered an error trying to
connect to a backend server. The backend stat is the sum of the stat
for all servers of that backend, plus any connection errors not
associated with a particular server (such as the backend having no
active servers).

derive.error_request

cumulative counter

Corresponds to HAProxy’s ereq metric - Request errors. Some of the possible causes are:
- early termination from the client, before the request has been sent.
- read error from the client
- client timeout
- client closed connection
- various bad requests from the client.
- request was tarpitted.

derive.error_response

cumulative counter

Corresponds to HAProxy’s eresp metric - Response errors. srv_abrt will be counted here also.
Some other errors are:
- write error on the client socket (won’t be counted for the server stat)
- failure applying filters to the response.

derive.failed_checks

cumulative counter

Corresponds to HAProxy’s chkfail metric - Number of failed checks. (Only counts checks failed when
the server is up.)

derive.redispatched

cumulative counter

Corresponds to HAProxy’s wredis metric - Number of times a request was redispatched to another
server. The server value counts the number of times that server was
switched away from.

derive.request_total

cumulative counter

Corresponds to HAProxy’s req_tot metric - Total number of HTTP requests received

derive.requests

cumulative counter

Corresponds to HAProxy’s CumReq metric.

derive.response_1xx

cumulative counter

Corresponds to HAProxy’s hrsp_1xx metric - Http responses with 1xx code

derive.response_2xx

cumulative counter

Corresponds to HAProxy’s hrsp_2xx metric - Http responses with 2xx code

derive.response_3xx

cumulative counter

Corresponds to HAProxy’s hrsp_3xx metric - Http responses with 3xx code

derive.response_4xx

cumulative counter

Corresponds to HAProxy’s hrsp_4xx metric - Http responses with 4xx code

derive.response_5xx

cumulative counter

Corresponds to HAProxy’s hrsp_5xx metric - Http responses with 5xx code

derive.response_other

cumulative counter

Corresponds to HAProxy’s hrsp_other metric - Http responses with other codes (protocol error)

derive.retries

cumulative counter

Corresponds to HAProxy’s wretr metric - Number of times a connection to a server was retried.

derive.session_total

cumulative counter

Corresponds to HAProxy’s stot metric - Cumulative number of connections

derive.srv_abrt

cumulative counter

Corresponds to HAProxy’s srv_abrt metric - Number of data transfers aborted by the server
(inc. in eresp)

derive.ssl_cache_lookups

cumulative counter

Corresponds to HAProxy’s SslCacheLookups metric.

derive.ssl_cache_misses

cumulative counter

Corresponds to HAProxy’s SslCacheMisses metric.

derive.ssl_connections

cumulative counter

Corresponds to HAProxy’s CumSslConns metric.

derive.uptime_seconds

cumulative counter

Corresponds to HAProxy’s Uptime_sec metric.

gauge.active_servers

gauge

Number of active servers. This corresponds to HAProxy’s “act” metric.

gauge.backup_servers

gauge

Number of backup servers. This corresponds to HAProxy’s “bck” metric.

gauge.check_duration

gauge

Time in ms took to finish to last health check. This corresponds to HAProxy’s “check_duration” metric.

gauge.connection_rate

gauge

Number of connections over the last elapsed second (frontend). This corresponds to HAProxy’s “conn_rate” metric.

gauge.connection_rate_max

gauge

Highest known connection rate. This corresponds to HAProxy’s “conn_rate_max” metric.

gauge.current_connections

gauge

Current number of connections. Corresponds to HAProxy’s “CurrConns” metric.

gauge.current_ssl_connections

gauge

Corresponds to HAProxy’s CurrSslConns metric.

gauge.denied_tcp_connections

gauge

Requests denied by ‘tcp-request connection’ rules. This corresponds to HAProxy’s “dcon” metric.

gauge.denied_tcp_sessions

gauge

Requests denied by ‘tcp-request session’ rules. This corresponds to HAProxy’s “dses” metric.

gauge.idle_pct

gauge

Corresponds to HAProxy’s “Idle_pct” metric. Ratio of system polling time versus total time.

gauge.intercepted_requests

gauge

Cumulative number of intercepted requests, corresponds to HAProxys metric ‘intercepted’

gauge.last_session

gauge

Number of seconds since last session was assigned to server/backend. This corresponds to HAProxy’s “lastsess” metric.

gauge.max_connection_rate

gauge

Corresponds to HAProxy’s MaxConnRate metric.

gauge.max_connections

gauge

Corresponds to HAProxy’s MaxConn metric.

gauge.max_pipes

gauge

Corresponds to HAProxy’s MaxPipes metric.

gauge.max_session_rate

gauge

Corresponds to HAProxy’s MaxSessRate metric.

gauge.max_ssl_connections

gauge

Corresponds to HAProxy’s MaxSslConns metric.

gauge.pipes_free

gauge

Corresponds to HAProxy’s PipesFree metric.

gauge.pipes_used

gauge

Corresponds to HAProxy’s PipesUsed metric.

gauge.queue_current

gauge

Corresponds to HAProxy’s qcur metric - Current queued requests. For the backend this reports the
number queued without a server assigned.

gauge.queue_limit

gauge

Configured max queue for the server, 0 being no limit. Corresponds to HAProxy’s “qlimit” metric.

gauge.queue_max

gauge

Max number of queued requests, queue_current, corresponds to HAProxy’s ‘qmax’ metric.

gauge.request_rate

gauge

Corresponds to HAProxy’s req_rate metric - HTTP requests per second over last elapsed second

gauge.request_rate_max

gauge

Max number of HTTP requests per second observed. Corresponds to HAProxy’s “req_rate_max” metric.

gauge.run_queue

gauge

Corresponds to HAProxy’s Run_queue metric.

gauge.session_current

gauge

Corresponds to HAProxy’s scur metric - Current sessions

gauge.session_rate

gauge

Corresponds to HAProxy’s rate metric - Number of sessions per second over last elapsed second

gauge.session_rate_limit

gauge

Configured limit on number of new sessions per second. Corresponds to HAProxy’s “rate_lim” metric.

gauge.session_rate_max

gauge

Max number of new sessions per second

gauge.session_time_average

gauge

Average total session time in ms over the last 1024 requests. Corresponds to HAProxy’s “ttime” metric.

gauge.ssl_backend_key_rate

gauge

Corresponds to HAProxy’s SslBackendKeyRate metric.

gauge.ssl_frontend_key_rate

gauge

Corresponds to HAProxy’s SslFrontendKeyRate metric.

gauge.ssl_rate

gauge

Corresponds to HAProxy’s SslRate metric.

gauge.tasks

gauge

Corresponds to HAProxy’s Tasks metric.

gauge.throttle

gauge

Corresponds to HAProxy’s throttle metric - Current throttle percentage for the server, when
slowstart is active, or no value if not in slowstart.

gauge.zlib_mem_usage

gauge

Corresponds to HAProxy’s ZlibMemUsage metric.