Docs » µAPM Deployment Guide » Deploying the SignalFx Smart Gateway

Deploying the SignalFx Smart Gateway

The Smart Gateway is a key component in your deployment of SignalFx Microservices APM: it receives all the distributed traces from your instrumented applications, generates metrics for each unique span and trace path, and selects the interesting, erroneous or outlier traces to forward to SignalFx. It is designed to run within your environment, close to your application, and operate reliably and at scale.

The Smart Gateway is an extension of SignalFx’s Gateway (formely known as the Metric Proxy) with support for Microservices APM’s NoSample™ Tail-Based Distributed Tracing features. It is available for download as a statically-linked Linux x86_64 binary that you can deploy and run in your environment. While most applications only require a single Smart Gateway instance, high-availability and scale can be achieved by running multiple Smart Gateway instances as a cluster.

Instance sizing

Many factors are involved in the resource utilization of the Smart Gateway, ranging from how detailed your tracing instrumentation is to, of course, the volume of transactions being captured and analyzed by the Smart Gateway. Generally, we recommend current generation instances like the c5-class AWS instances, offering high speed networking, fast CPU cores, and plenty of system memory.

Recommended instance sizing to run the Smart Gateway are as follows, based on your expected volume of trace spans per minute (SPM). If you’re unsure which one to use, err on the side of larger instances and monitor your Smart Gateway instances to evaluate your actual resource utilization.

SPM AWS EC2 Type
Up to 6M c5.18xlarge
Up to 3M c5.9xlarge
Up to 1.5M c5.4xlarge
Up to 750k c5.2xlarge

For volume applications, scale and high-availability, we recommend running multiple Smart Gateway instances, configured to operate as a coordinated cluster behind a load balancer.

Deploying the Smart Gateway

Note about realms

A realm is a self-contained deployment of SignalFx in which your organization is hosted. Different realms have different API endpoints (e.g. the endpoint for sending data is ingest.us1.signalfx.com for the us1 realm, and ingest.eu0.signalfx.com for the eu0 realm).

Various statements in the instructions below include a YOUR_SIGNALFX_REALM placeholder that you should replace with the actual name of your realm. This realm name is shown on your profile page in SignalFx. If you do not include the realm name when specifying an endpoint, SignalFx will interpret it as pointing to the us0 realm.

The Smart Gateway is available as a statically-linked Linux x86_64 binary. If you know your organization’s access token and realm, you can download the binary from SignalFx using the following command:

$ curl -qs -H"X-SF-Token:YOUR_ACCESS_TOKEN" \
    https://api.YOUR_SIGNALFX_REALM.signalfx.com/v2/smart-gateway/download \
    | gunzip > smart-gateway

Otherwise, head over to your organization’s Integrations page, click on the SignalFx Smart Gateway tile, and follow the link to download the Smart Gateway binary.

Building a Docker image

If you intend to deploy your Smart Gateway as a Docker image for easier orchestration, you can use the following Dockerfile to build a functioning Docker image. This step is optional; you can distribute and orchestrate the Smart Gateway whichever way best fits your current infrastructure and environment.

FROM scratch
ADD smart-gateway /
ADD https://raw.githubusercontent.com/signalfx/gateway/master/ca-bundle.crt /etc/pki/tls/certs/ca-bundle.crt
VOLUME /var/lib/gateway
CMD ["/smart-gateway", "--configfile", "/var/lib/gateway/etc/gateway.conf"]

Assuming both the smart-gateway binary and this Dockerfile are in your current working directory, you can build the corresponding Docker image with:

$ docker build -t signalfx-smart-gateway .

Smart Gateway configuration

The Smart Gateway, like the SignalFx Gateway it extends, reads its configuration from a single JSON file. By default, the Smart Gateway will look for this configuration file at /etc/gateway.conf; to specify a different location, use the command line flag --configfile. The configuration is composed of three main elements: top-level configuration of the Smart Gateway, listeners (ListenFrom) and forwarders (ForwardTo); for a complete reference of those sections, refer to this section in the SignalFx Gateway’s configuration reference.

The following instructions enable our NoSample™ Tail-Based Distributed Tracing feature, and “transform” the SignalFx Gateway into the SignalFx Smart Gateway. To this effect, there are four key configuration elements to pay attention to:

  • the ServerName of this Smart Gateway instance. This should match the hostname of the underlying instance as reported by the SignalFx Smart Agent to enable monitoring of the Smart Gateway;
  • the ClusterName of the cluster this Smart Gateway is a part of. This is typically an environment name, like qa or prod. It is required even for single-instance deployments;
  • a signalfx listener must be configured for the Smart Gateway to receive traces;
  • a signalfx forwarder with a TraceSample section and persistent BackupLocation must be configured to enable smart trace sampling and send the selected traces to SignalFx.

When put together, your Smart Gateway configuration should look as follows (replace YOUR_SIGNALFX_REALM by the name of the SignalFx realm your organization is hosted in and YOUR_SIGNALFX_API_TOKEN with your organization token):

{
  "ServerName": "smart-gateway-1",
  "ClusterName": "prod",
  "StatsDelay": "10s",
  "LogDir": "/var/log/gateway",
  "ListenFrom": [
    {
      "Type": "signalfx",
      "ListenAddr": "0.0.0.0:8080"
    }
  ],
  "ForwardTo": [
    {
      "Type": "signalfx",
      "URL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/datapoint",
      "EventURL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/event",
      "TraceURL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v1/trace",
      "DefaultAuthToken": "YOUR_SIGNALFX_API_TOKEN",
      "Name": "smart-gateway-forwarder",
      "TraceSample": {
        "BackupLocation": "/var/lib/gateway/data"
      }
    }
  ]
}

Note that the ServerName of your SignalFx Smart Gateway must be unique within a given Smart Gateway cluster.

Dockerized Smart Gateway configuration

If you are packaging your Smart Gateway in a Docker image as described above, you must ensure that the configured BackupLocation persists across container restarts. This can be achieved by placing your BackupLocation on a bind-mounted volume like /var/lib/gateway in the example above. Similarly, you might want to redirect the Smart Gateway’s log to standard out for Docker to capture (using "LogDir": "-"), or to a file placed on a bind-mounted volume.

Using the Dockerfile above, make sure to place this configuration under /var/lib/gateway/etc/gateway.conf (replace YOUR_SIGNALFX_REALM by the name of the SignalFx realm your organization is hosted in and YOUR_SIGNALFX_API_TOKEN with your organization token):

{
  "ServerName": "smart-gateway-1",
  "ClusterName": "prod",
  "StatsDelay": "10s",
  "LogDir": "/var/lib/gateway/logs",
  "ListenFrom": [
    {
      "Type": "signalfx",
      "ListenAddr": "0.0.0.0:8080"
    }
  ],
  "ForwardTo": [
    {
      "Type": "signalfx",
      "URL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/datapoint",
      "EventURL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/event",
      "TraceURL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v1/trace",
      "DefaultAuthToken": "YOUR_SIGNALFX_API_TOKEN",
      "Name": "smart-gateway-forwarder",
      "TraceSample": {
        "BackupLocation": "/var/lib/gateway/data"
      }
    }
  ]
}

Running the Smart Gateway

After making sure that all the directory locations the Smart Gateway uses have the appropriate permissions set, start your Smart Gateway by running its binary, specifying the appropriate configuration file location:

$ ./smart-gateway --configfile /wherever/is/gateway.conf

Or, if you’re using Docker:

$ docker run -d --name smart-gateway -v /var/lib/gateway:/var/lib/gateway -p 8080 signalfx-smart-gateway

You can verify that your Smart Gateway is running and accepting traces by sending an empty payload to its /v1/trace endpoint and expecting a return value of "OK":

$ curl -d'[]' -H'Content-Type:application/json' http://<your-gateway>:8080/v1/trace
"OK"

Finally, make sure your deployed SignalFx Smart Agents are configured to send data through your Smart Gateway by configuring their ingestUrl to http://<your-gateway>:8080/ (as per the Smart Agent Configuration documentation).