Docs » µAPM Deployment Guide » Advanced Smart Gateway Features

Advanced Smart Gateway Features


Most of the features highlighted below require additional Smart Gateway configuration. When running a cluster of Smart Gateway instances, you must ensure that the configuration of those features is the same on all the instances of the cluster. Generally speaking, except for IP addresses, the configuration of all Smart Gateway instances in a cluster should be exactly the same.

High cardinality µAPM identities generated by variables in span names

Many applications are instrumented with variable span names. This is an anti-pattern and will lead to poor performance and sampling accuracy of the Smart Gateway; it creates a very large number of span and trace identities, which results in the Smart Gateway’s inability to construct consistent baselines for those spans and traces while also consuming more memory resources. This pattern will also impact the performance and user experience of the SignalFx APM UI.

Instead of using variable span names, we recommend using span tags. However, if you are unable to modify your application to not emit variable span names and leverage tags instead for those variable elements, the Smart Gateway can turn these high cardinality names into tags using a configurable set of replacement rules. For example, the following rule will extract the variable document ID from the span name and carry this variable value in a span tag instead:

    "Type": "signalfx",
    "ListenAddr": "",
    "SpanNameReplacementRules": ["^\/api\/v1\/document\/(?P<documentId>.*)\/update$"],

Doing this will make the span identity space much smaller and will allow the Smart Gateway to establish span-level and trace-level baselines accurately, restoring the quality and accuracy of the trace selection algorithm while retaining all the required information on the spans, making them available for analysis by our Outlier Analyzer.

Span metadata scrubbing and obfuscation

If your trace spans contain metadata that you don’t want to send to SignalFx (PII, sensitive information, etc), the Smart Gateway can be configured to remove or obfuscate span tags. Metadata obfuscation replaces the value of a matching span tag with the string <obfuscated>, while metadata removal completely removes the tag key/value pair from the span. Obfuscation and removal can be configured to match on specific spans by service name and operation name, with * wildcard matching. Note that obfuscation rules are evaluated before removal rules. For a full description of how those features are configured, read the Obfuscating Span Tag Metadata and Removing Span Tag Metadata sections of the Gateway reference documentation.

    "Type": "signalfx",
    "ListenAddr": "",
    "ObfuscateSpanTags": [
            "Service": "auth*",
            "Operation": "login",
            "Tags": ["password"]
            "Operation": "*credit-card*",
            "Tags": ["zipcode", "number", "CVV"]
    "RemoveSpanTags": [
            "Service": "notifications",
            "Tags": ["phone_number"]

Trace debugging

In some cases, it can be useful to force the Smart Gateway to retain a specific trace, for debugging purposes. If you want to force a single trace to be selected for retention by the Smart Gateway, set the sampling.priority tag to 1 on a span in the trace. If any span in a trace has this tag set, the entire trace will be selected by the Smart Gateway to be forwarded to SignalFx.

import io.opentracing.Tags;
Tags.SAMPLING_PRIORITY.set(span, 1);

You can find your trace on the Traces page by searching for sampling.priority:1 in the Filter field.

Trace debugging should be used sparingly. When a debug trace is retained, the Smart Gateway still counts the trace against the overall trace retention budget. If a high number of traces is retained using this tag, the trace selection algorithm will not be able to select as many traces for retention.

The number of force retained traces is also limited by the number of traces retained per minute by the Smart Gateway. Your SignalFx subscription allows you to retain a certain number of traces per minute; any force retained traces above this number will not be forwarded to SignalFx.