Docs » Deploying the SignalFx Smart Agent

Deploying the SignalFx Smart Agent 🔗

Back to Deployment Guide contents

The standard recommended deployment model for APM PG involves running the SignalFx Smart Agent on each host that runs traced applications. This ensures that a set of known, system-level metrics are reported to SignalFx to provide information about the host. Additionally, the traced applications on that host must report their trace spans to the Smart Agent, which will forward them to the Smart Gateway.

Sending trace spans through the Smart Agent gives the Smart Agent an opportunity to add certain tags to the spans that identify the host on which the spans were generated (e.g. the host tag that is the hostname, and cloud-provider specific tags such as AWSUniqueId if running on EC2). While this could be done in the tracer configuration as global tags, it is better to let the Smart Agent do it for consistency with the host metadata of metrics.

The Smart Agent will also generate internally-used metric time series called that our backend uses to match up the local service name of spans to the host(s) on which they are running. This metric is included in all subscriptions and does not count towards your custom metric limit.


If you do not already have the Smart Agent version 4.7.3 or above running on your target hosts, see the information starting in Smart Agent Quick Install to install the agent on those hosts using the method of your choice, then continue with the steps below.

Once you have the agent running on your hosts, there are a few configuration options you need to set in your agent.yaml config file to enable trace forwarding and the host correlation datapoints as shown in this configuration example. Additionally, make sure that your Smart Agent is installed and configured to send metrics to your SignalFx realm.

Note about realms

A realm is a self-contained deployment of Splunk Infrastructure Monitoring or APM in which your organization is hosted. Different realms have different API endpoints. For example, the endpoint for sending data in the us1 realm is, while the endpoint for sending data in the eu0 realm is

Where you see a placeholder that refers to realms, such as <REALM> or <YOUR_REALM>, replace it with the actual name of your realm. You can find this name on your profile page in the user interface. If you don’t include the realm name when specifying an endpoint, Infrastructure Monitoring defaults to the us0 realm.

# Your SignalFx realm

# The host's hostname

# The host's environment/cluster name. This should match the
# ClusterName configured in your Smart Gateway.

# Your Smart Gateway's listener endpoint.
# When not specified this is automatically derived from the "ingestUrl",
# "signalFxRealm" and "writer.traceExportFormat" settings. If specified, the
# HTTP path and port should match the export format being used.
# Check "Trace export formats" for more information.

    # The format that traces should be exported in. Choices are "zipkin" and "sapm".
    # Defaults to "zipkin".
    traceExportFormat: zipkin

  # Enable host metadata and basic system-level summary metrics
  - type: host-metadata
  - type: collectd/cpu
  - type: collectd/memory
  - type: collectd/interface
  - type: collectd/df
  - type: collectd/disk
  - type: collectd/signalfx-metadata
  # Enable the trace span listener server
  - type: trace-forwarder
    # This can be changed if needed; always HTTP only.
    # Use if your app runs in a container
    # and won't talk to your Smart Agent over lo0.

Once you have the agent configured properly, the only thing left to do is configure your application’s tracer to send to the listenAddress specified in the trace-forwarder monitor configuration. The trace-forwarder monitor supports all of the span formats that our Smart Gateway supports (see APM PG supported span formats). To configure your tracer, see µAPM PG Instrumentation Guide. Note that all SignalFx-provided instrumentation libraries default to sending to a locally-running Smart Agent listening for trace spans on port 9080 as shown in the configuration example above.

In the case of a slow downstream trace endpoint, the Smart Agent will buffer spans up to a default of 100,000 spans before dropping data to prevent unbounded memory consumption. The agent does not do any sampling, as that is handled in the Smart Gateway.

Note that in this minimal configuration, the footprint and impact of the Smart Agent on the host system is minimal. The CPU utilization of the Smart Agent will be directly related to the volume of trace spans that pass through it; it can comfortably handle over 100,000 spans/sec. The memory utilization of the Smart Agent should remain minimal during normal operation, but it may increase if the SignalFx Smart Gateway is unavailable and the Smart Agent needs to buffer trace spans.

Trace export formats 🔗

The SignalFx Smart Agent supports exporting traces in multiple formats. To specify the format, use can be specified using the writer.traceExportFormat setting. Supported formats are zipkin and sapm.

Note that the Smart Agent exports traces in each format to a different HTTP path. When traceEndpointUrl is not set, the Smart Agent determines it’s value from ingestUrl, signalFxRealm and writer.traceExportFormat options. By default, traces are exported to /v1/trace when using the Zipkin format and to /v2/trace when using SAPM. If you choose to specify a custom value for traceEndpointUrl, confirm that traceEndpointUrl uses the correct HTTP path for the chosen format.

# Endpoint should use "/v1/trace" for zipkin and "/v2/trace" for sapm.

    # Can be either "zipkin" or "sapm".
    # Defaults to "zipkin"
    traceExportFormat: sapm

Deploying on Kubernetes 🔗

When deploying the Smart Agent in a Kubernetes cluster, follow the normal instructions at Kubernetes Setup. This will deploy one Smart Agent instance per node in your cluster. You should send spans from your application pods to the Smart Agent instance running on the same node. The Smart Agent’s DaemonSet is configured to run on the host network stack such that you can reach the trace forwarder endpoint at the node’s primary IP address.

The status.hostIP Downward API field in Kubernetes workload resources provides a means of dynamically setting an environment variable in your pod that tells it which node it is running on. Here is a basic Deployment resource that has the host IP address injected as the envvar SIGNALFX_AGENT_HOST:

apiVersion: extensions/v1beta1
kind: Deployment
    app: myapp
  name: myapp
  replicas: 1
      app: myapp
        app: myapp
      - env:
        - name: SIGNALFX_AGENT_HOST
              apiVersion: v1
              fieldPath: status.hostIP
        image: my-image
        name: myapp

You will have to configure your tracer to make use of that SIGNALFX_AGENT_HOST environment variable (see the documentation for each language and tracer).

Back to Deployment Guide contents