Docs » µAPM Deployment Guide » Deploying the SignalFx Smart Gateway

Deploying the SignalFx Smart Gateway 🔗

The Smart Gateway is a key component in your deployment of SignalFx Microservices APM: it receives all the distributed traces from your instrumented applications, generates metrics for each unique span and trace path, and selects the interesting, erroneous or outlier traces to forward to SignalFx. It is designed to run within your environment, close to your application, and operate reliably and at scale.

The Smart Gateway is an extension of SignalFx’s Gateway (formely known as the Metric Proxy) with support for Microservices APM’s NoSample™ Tail-Based Distributed Tracing features. It is available for download as a statically-linked Linux x86_64 binary that you can deploy and run in your environment. While most applications only require a single Smart Gateway instance, high-availability and scale can be achieved by running multiple Smart Gateway instances as a cluster.

Note

If you deploy a Smart Gateway (single instance, or cluster) for µAPM, it can play the role of any existing (regular) Gateway that you may have deployed, allowing you to configure your applications and Smart Agents to report metrics through that Smart Gateway deployment; see the information on forwarding metrics in the SignalFx Gateway documentation for more details.

Sizing recommendations 🔗

Many factors are involved in the resource utilization of the Smart Gateway, ranging from how detailed your tracing instrumentation is to, of course, the volume of transactions being captured and analyzed by the Smart Gateway. Generally, we recommend current generation instances like the c5d.9xlarge AWS instances, offering high speed networking, latest generation CPU cores, plenty of system memory and fast local storage.

Depending on your needs, the number of Smart Gateway instances that you need to run and the class of hardware supporting them will vary:

  • For a trial, for testing, low-volume applications or when high-availability is not a strong requirement, deploying a single instance is sufficient and can be scaled vertically if necessary.
  • For volume applications, higher scale or high-availability requirements, up to 10 million traces/minute, we recommend running multiple Smart Gateway instances configured to operate as a coordinated cluster behind a load balancer.
  • For very large scale applications generating large volumes of tracing data above 10 million traces/minute, we recommend running a larger, high-scale Smart Gateway cluster with a separate Etcd cluster.

Recommended deployment and sizing to run the Smart Gateway are as follows, based on your expected volume of traces analyzed per minute (TAPM) and number of APM identities. If you’re unsure which one to use, err on the side of larger instances and monitor your Smart Gateway instances to evaluate your actual resource utilization.

TAPM SPM APM Identities Recommended deployment
Up to 1M ~10M ~1,000 1x c5d.9xlarge, or 3x c5d.4xlarge
Up to 5M ~50M ~5,000 3x c5d.9xlarge
Up to 10M ~100M ~5,000 6x c5d.9xlarge
Up to 50M ~750M ~5,000 36x c5d.9xlarge

Note

Those sizing recommendations are based on the performance and throughput characteristics of the SignalFx Smart Gateway version 2.0 and above, and expressed using AWS EC2 instance types. If you operate on other cloud providers or on your own infrastructure, simply match the CPU, memory and disk capacity of those instance types.

Versioning 🔗

The Smart Gateway uses version numbers based on Semantic Versioning. When deploying a Smart Gateway cluster, instances with different major versions cannot be running in the same cluster, so it’s not possible to do a rolling upgrade to a new major version. It is possible to do rolling upgrades within the same major version.

Checking the current version of Smart Gateway 🔗

To check the current version, you can check the built-in Cluster(s) dashboard for the Smart Gateway.

../../_images/clusters-dashboard.png

The chart shown below will display the version of the Gateway and NoSample module.

../../_images/version-chart.png

Downloading a specific version of the Smart Gateway 🔗

Note about realms

A realm is a self-contained deployment of SignalFx in which your organization is hosted. Different realms have different API endpoints (e.g. the endpoint for sending data is ingest.us1.signalfx.com for the us1 realm, and ingest.eu0.signalfx.com for the eu0 realm).

Various statements in the instructions below include a YOUR_SIGNALFX_REALM placeholder that you should replace with the actual name of your realm. This realm name is shown on your profile page in SignalFx. If you do not include the realm name when specifying an endpoint, SignalFx will interpret it as pointing to the us0 realm.

To download a specific version of the Smart Gateway, visit the Smart Gateway tile in your Integrations page. There you will find a changelog along with download links for all previous versions of the Smart Gateway. Or, if you know which version of the Smart Gateway you want to download, you can use the following curl command pointed to the proper version:

$ curl -qs -H"X-SF-Token:YOUR_ACCESS_TOKEN" https://app.YOUR_SIGNALFX_REALM.signalfx.com/v2/smart-gateway/download/<version> | gunzip > smart-gateway
$ chmod +x smart-gateway

Deploying a Smart Gateway instance 🔗

The Smart Gateway is available as a statically-linked Linux x86_64 binary. To download the latest version of the Smart Gateway, open the Integrations page, click on the SignalFx Smart Gateway tile, and follow the instructions in the Setup tab. Alternatively, if you have an API token and know which version of the Smart Gateway you need, use the curl command above to download it.

Smart Gateway configuration 🔗

The Smart Gateway, like the SignalFx Gateway it extends, reads its configuration from a single JSON file. By default, the Smart Gateway will look for this configuration file at /etc/gateway.conf; to specify a different location, use the command line flag --configfile. The configuration is composed of three main elements: top-level configuration of the Smart Gateway, listeners (ListenFrom) and forwarders (ForwardTo); for a complete reference of those sections, refer to this section in the SignalFx Gateway’s configuration reference.

The following instructions enable our NoSample™ Tail-Based Distributed Tracing feature, and “transform” the SignalFx Gateway into the SignalFx Smart Gateway. To this effect, there are four key configuration elements to pay attention to:

  • the ServerName of this Smart Gateway instance. This should match the hostname of the underlying instance as reported by the SignalFx Smart Agent to enable monitoring of the Smart Gateway;
  • the ClusterName of the cluster this Smart Gateway belongs to. This is typically an environment name, such as qa or prod. It is required even for single-instance deployments. For more information on the meaning of cluster names, see Managing multiple application environments with SignalFx µAPM;
  • a signalfx listener must be configured for the Smart Gateway to receive traces;
  • a signalfx forwarder with a TraceSample section and persistent BackupLocation must be configured to enable smart trace sampling and send the selected traces to SignalFx.

When put together, your Smart Gateway configuration should look as follows (replace YOUR_SIGNALFX_REALM by the name of the SignalFx realm your organization is hosted in and YOUR_SIGNALFX_API_TOKEN with your organization token):

{
  "ServerName": "smart-gateway-1",
  "ClusterName": "prod",
  "LogDir": "/var/log/gateway",
  "ListenFrom": [
    {
      "Type": "signalfx",
      "ListenAddr": "0.0.0.0:8080"
    }
  ],
  "ForwardTo": [
    {
      "Type": "signalfx",
      "URL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/datapoint",
      "EventURL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/event",
      "TraceURL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v1/trace",
      "DefaultAuthToken": "YOUR_SIGNALFX_API_TOKEN",
      "Name": "smart-gateway-forwarder",
      "TraceSample": {
        "BackupLocation": "/var/lib/gateway/data"
      }
    }
  ]
}

Note that the ServerName of your SignalFx Smart Gateway must be unique within a given Smart Gateway cluster.

Running the Smart Gateway 🔗

After making sure that all the directory locations the Smart Gateway uses have the appropriate permissions set, start your Smart Gateway by running its binary, specifying the appropriate configuration file location:

$ ./smart-gateway --configfile /wherever/is/gateway.conf

You can verify that your Smart Gateway is running and accepting traces by sending an empty payload to its /v1/trace endpoint and expecting a return value of "OK":

$ curl -d'[]' -H'Content-Type:application/json' http://<your-gateway>:8080/v1/trace
"OK"

Finally, make sure your deployed SignalFx Smart Agents are configured to send data through your Smart Gateway by configuring their ingestUrl to http://<your-gateway>:8080/ (as per the Smart Agent Configuration documentation).

Running the Smart Gateway as a Docker container 🔗

Building a Docker image 🔗

If you intend to deploy your Smart Gateway as a Docker image for easier orchestration, you can use the following Dockerfile to build a functioning Docker image. This step is optional; you can distribute and orchestrate the Smart Gateway whichever way best fits your current infrastructure and environment.

FROM scratch
ADD smart-gateway /
ADD https://raw.githubusercontent.com/signalfx/gateway/master/ca-bundle.crt /etc/pki/tls/certs/ca-bundle.crt
VOLUME /var/lib/gateway
CMD ["/smart-gateway", "--configfile", "/var/lib/gateway/etc/gateway.conf"]

Assuming both the smart-gateway binary and this Dockerfile are in your current working directory, you can build the corresponding Docker image with:

$ docker build -t signalfx-smart-gateway .

Dockerized Smart Gateway configuration 🔗

When running the Smart Gateway in a Docker container, you must ensure that the configured BackupLocation persists across container restarts. This can be achieved by placing your BackupLocation on a bind-mounted volume like /var/lib/gateway in the example above. Similarly, you might want to redirect the Smart Gateway’s log to standard out for Docker to capture (using "LogDir": "-"), or to a file placed on a bind-mounted volume.

Using the Dockerfile above, make sure to place this configuration under /var/lib/gateway/etc/gateway.conf (replace YOUR_SIGNALFX_REALM by the name of the SignalFx realm your organization is hosted in and YOUR_SIGNALFX_API_TOKEN with your organization token):

{
  "ServerName": "smart-gateway-1",
  "ClusterName": "prod",
  "LogDir": "/var/lib/gateway/logs",
  "ListenFrom": [
    {
      "Type": "signalfx",
      "ListenAddr": "0.0.0.0:8080"
    }
  ],
  "ForwardTo": [
    {
      "Type": "signalfx",
      "URL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/datapoint",
      "EventURL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/event",
      "TraceURL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v1/trace",
      "DefaultAuthToken": "YOUR_SIGNALFX_API_TOKEN",
      "Name": "smart-gateway-forwarder",
      "TraceSample": {
        "BackupLocation": "/var/lib/gateway/data"
      }
    }
  ]
}

Running the Smart Gateway container 🔗

$ docker run -d --name smart-gateway -v /var/lib/gateway:/var/lib/gateway -p 8080:8080 signalfx-smart-gateway

You can verify that your Smart Gateway is running and accepting traces by sending an empty payload to its /v1/trace endpoint and expecting a return value of "OK":

$ curl -d'[]' -H'Content-Type:application/json' http://<your-gateway>:8080/v1/trace
"OK"