Docs » µAPM Deployment Guide » Advanced Smart Gateway Features

Advanced Smart Gateway Features 🔗

Note

Most of the features highlighted below require additional Smart Gateway configuration. When running a cluster of Smart Gateway instances, you must ensure that the configuration of those features is the same on all the instances of the cluster. Generally speaking, except for IP addresses, the configuration of all Smart Gateway instances in a cluster should be exactly the same.

High cardinality µAPM identities generated by variables in span names 🔗

Many applications are instrumented with variable span names. This is an anti-pattern and will lead to poor performance and sampling accuracy of the Smart Gateway; it creates a very large number of span and trace identities, which results in the Smart Gateway’s inability to construct consistent baselines for those spans and traces while also consuming more memory resources. This pattern will also impact the performance and user experience of the SignalFx APM UI.

Instead of using variable span names, we recommend using span tags. However, if you are unable to modify your application to not emit variable span names and leverage tags instead for those variable elements, the Smart Gateway can turn these high cardinality names into tags using a configurable set of replacement rules. For example, the following rule will extract the variable document ID from the span name and carry this variable value in a span tag instead:

{
  ...
  "ListenFrom": [
    {
      "Type": "signalfx",
      "ListenAddr": "0.0.0.0:8080",
      "SpanNameReplacementRules": ["^\/api\/v1\/document\/(?P<documentId>.*)\/update$"],
    }
  ],
  ...
}

Doing this will make the span identity space much smaller and will allow the Smart Gateway to establish span-level and trace-level baselines accurately, restoring the quality and accuracy of the trace selection algorithm while retaining all the required information on the spans, making them available for analysis by the Outlier Analyzer.

Span metadata scrubbing and obfuscation 🔗

If your trace spans contain metadata that you don’t want to send to SignalFx (PII, sensitive information, etc), the Smart Gateway can be configured to remove or obfuscate span tags. Metadata obfuscation replaces the value of a matching span tag with the string <obfuscated>, while metadata removal completely removes the tag key/value pair from the span. Obfuscation and removal can be configured to match on specific spans by service name and operation name, with * wildcard matching. Note that obfuscation rules are evaluated before removal rules. For a full description of how those features are configured, read the Obfuscating Span Tag Metadata and Removing Span Tag Metadata sections of the Gateway reference documentation.

{
  ...
  "ListenFrom": [
    {
      "Type": "signalfx",
      "ListenAddr": "0.0.0.0:8080",
      "ObfuscateSpanTags": [
          {
              "Service": "auth*",
              "Operation": "login",
              "Tags": ["password"]
          },
          {
              "Operation": "*credit-card*",
              "Tags": ["zipcode", "number", "CVV"]
          }
      ],
      "RemoveSpanTags": [
          {
              "Service": "notifications",
              "Tags": ["phone_number"]
          }
      ]
    }
  ],
  ...
}

Force trace retention 🔗

In some cases, it can be useful to force the Smart Gateway to retain a specific trace, for debugging purposes. If you want to force a single trace to be selected for retention by the Smart Gateway, set the sampling.priority tag to 1 on a span in the trace. If any span in a trace has this tag set, the entire trace will be selected by the Smart Gateway to be forwarded to SignalFx.

import io.opentracing.Tags;
...
Tags.SAMPLING_PRIORITY.set(span, 1);

You can find your trace on the Traces page by searching for sampling.priority:1 in the Filter field.

Your SignalFx subscription defines the maximum number of traces that are retained per minute. During normal operation, the Smart Gateway will use this budget to select interesting traces to retain. Force-retained traces will take precedence over the Smart Gateway selection algorithm and will be counted against the retention budget.

As more traces are force-retained, fewer traces will be selected by the Smart Gateway. In the extreme case, only force-retained traces will be selected, up to the maximum budget of retained traces per minute. We recommend that force-retained traces are used sparingly, to minimize impact to the Smart Gateway selection algorithm.

Force initiating spans 🔗

The Smart Gateway expects an initiating span to be sent within a trace, but this does not always happen. Sometimes an initiating span is not sent to the Smart Gateway. When no initiating span is identified, the Smart Gateway will not select the trace unless it is explicitely asked to do so by configuration.

When enabled, this functionality instructs the Smart Gateway to choose the earliest of the parent-most spans received after a fixed 5 minutes delay as the initiating span in the trace. To enable this feature, edit the TraceSample section in the signalfx forwarder config and set ForceTraceInitiatingSpan to true, as shown below.

Note about realms

A realm is a self-contained deployment of SignalFx in which your organization is hosted. Different realms have different API endpoints (e.g. the endpoint for sending data is ingest.us1.signalfx.com for the us1 realm, and ingest.eu0.signalfx.com for the eu0 realm).

Various statements in the instructions below include a YOUR_SIGNALFX_REALM placeholder that you should replace with the actual name of your realm. This realm name is shown on your profile page in SignalFx. If you do not include the realm name when specifying an endpoint, SignalFx will interpret it as pointing to the us0 realm.

{
  "ServerName": "smart-gateway-1",
  "LogDir": "/var/log/gateway",
  "ListenFrom": [
    {
      "Type": "signalfx",
      "ListenAddr": "0.0.0.0:8080"
    }
  ],
  "ForwardTo": [
    {
      "Type": "signalfx",
      "URL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/datapoint",
      "EventURL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/event",
      "TraceURL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v1/trace",
      "DefaultAuthToken": "YOUR_SIGNALFX_API_TOKEN",
      "Name": "smart-gateway-forwarder",
      "TraceSample": {
        "BackupLocation": "/var/config/gateway/data",
        "ForceTraceInitiatingSpan": true
      }
    }
  ]
}