Docs » µAPM Deployment Guide » Advanced Smart Gateway Features

Advanced Smart Gateway Features 🔗

Note

Most of the features highlighted below require additional Smart Gateway configuration. When running a cluster of Smart Gateway instances, you must ensure that the configuration of those features is the same on all the instances of the cluster. Generally speaking, except for IP addresses, the configuration of all Smart Gateway instances in a cluster should be exactly the same.

High cardinality µAPM identities generated by variables in span names 🔗

Many applications are instrumented with variable span names. This is an anti-pattern and will lead to poor performance and sampling accuracy of the Smart Gateway; it creates a very large number of span and trace identities, which results in the Smart Gateway’s inability to construct consistent baselines for those spans and traces while also consuming more memory resources. This pattern will also impact the performance and user experience of the SignalFx APM UI.

Instead of using variable span names, we recommend using span tags. However, if you are unable to modify your application to not emit variable span names and leverage tags instead for those variable elements, the Smart Gateway can turn these high cardinality names into tags using a configurable set of replacement rules. For example, the following rule will extract the variable document ID from the span name and carry this variable value in a span tag instead:

{
  ...
  "ListenFrom": [
    {
      "Type": "signalfx",
      "ListenAddr": "0.0.0.0:8080",
      "SpanNameReplacementRules": ["^\/api\/v1\/document\/(?P<documentId>.*)\/update$"],
    }
  ],
  ...
}

Doing this will make the span identity space much smaller and will allow the Smart Gateway to establish span-level and trace-level baselines accurately, restoring the quality and accuracy of the trace selection algorithm while retaining all the required information on the spans, making them available for analysis by the Outlier Analyzer.

Span metadata scrubbing and obfuscation 🔗

If your trace spans contain metadata that you don’t want to send to SignalFx (PII, sensitive information, etc), the Smart Gateway can be configured to remove or obfuscate span tags. Metadata obfuscation replaces the value of a matching span tag with the string <obfuscated>, while metadata removal completely removes the tag key/value pair from the span. Obfuscation and removal can be configured to match on specific spans by service name and operation name, with * wildcard matching. Note that obfuscation rules are evaluated before removal rules. For a full description of how those features are configured, see Obfuscating Span Tag Metadata and Removing Span Tag Metadata of the Gateway reference documentation.

{
  ...
  "ListenFrom": [
    {
      "Type": "signalfx",
      "ListenAddr": "0.0.0.0:8080",
      "ObfuscateSpanTags": [
          {
              "Service": "auth*",
              "Operation": "login",
              "Tags": ["password"]
          },
          {
              "Operation": "*credit-card*",
              "Tags": ["zipcode", "number", "CVV"]
          }
      ],
      "RemoveSpanTags": [
          {
              "Service": "notifications",
              "Tags": ["phone_number"]
          }
      ]
    }
  ],
  ...
}

Extended Span Identity Metrics With Custom Dimensionalization 🔗

A span identity includes its service name (service), operation name (operation), HTTP method (method), and kind (kind). In addition to this base identity and its corresponding tracing metrics, it may be useful to look at a metrics for a subset of spans in a given identity based on tags attached to a span. Custom dimensionalization allows you to define rules for syntehsizing span identities with tag values that will generate histogram level metrics with dimensions representing those tags. The extended span identities will not be used for sampling decisions.

To use custom dimensionalization, you must define a rule in the TraceSample section of your SignalFx Smart Gateway’s signalfx forwarder config. The rule must have glob patterns defined for Service and Operation and at least one tag name and value pattern under Tags. Tag keys are plain text strings and must directly match, but tag values are glob patterns. You may optionally specify glob patterns for HTTPMethod and Kind.

If HTTPMethod or Kind is not specified, then any span with values for HTTPMethod or Kind will not matched the rule. If you want to allow any value for HTTPMethod or Kind, then you must define a pattern for them ("HTTPMethod": "*").

Please note that histogram metrics reported for extended span identities may be sparse if matching spans are not received frequently.

Example Configuration:

{
  ...
  "ForwardTo": [
    {
      "type": "signalfx",
      "TraceSample": {
        "CustomDimensionalization": [
          {
            "Service": "*",
            "Operation": "*",
            "HTTPMethod": "*",
            "Kind": "*",
            "Tags": {
              "d16n": "val*"
            }
          }
        ]
      }
    }
  ]
  ...
}

This example matches all spans that have the tag d16n for each tag value with the prefix val.

Example: Identities for all canary deployments of a service named foo

...
"CustomDimensionalization": [
  {
    "Service": "foo",
    "Operation": "*",
    "HTTPMethod": "*",
    "Kind": "*",
    "Tags": {
      "release": "canary",
    }
  }
]
...

All spans for the service foo with the release tag set to canary will be treated as a separate extra identity. The Smart Gateway will report span histogram metrics with the added dimension release with the value canary.

Example: Identities for canaray deployments of a service named foo by version:

...
"CustomDimensionalization": [
  {
    "Service": "foo",
    "Operation": "*",
    "HTTPMethod": "*",
    "Kind": "*",
    "Tags": {
      "release": "canary",
      "version": "*"
    }
  }
]
...

All spans for the service foo with the release tag set to canary and any version tag will be treated as separate extra identities for each canary version. The Smart Gateway will report span histogram metrics with the added dimensions release and version.

Please note that you may define more than one rule in the CustomDimensionalization array. The rules are treated independently even if multiple rules effectively cover the same subset of spans.

Force trace retention 🔗

In some cases, it can be useful to force the Smart Gateway to retain a specific trace, for debugging purposes. If you want to force a single trace to be selected for retention by the Smart Gateway, set the sampling.priority tag to 1 on a span in the trace. If any span in a trace has this tag set, the entire trace will be selected by the Smart Gateway to be forwarded to SignalFx.

import io.opentracing.Tags;
...
Tags.SAMPLING_PRIORITY.set(span, 1);

You can find your trace on the Traces page by searching for sampling.priority:1 in the Filter field.

Your SignalFx subscription defines the maximum number of traces that are retained per minute. During normal operation, the Smart Gateway will use this budget to select interesting traces to retain. Force-retained traces will take precedence over the Smart Gateway selection algorithm and will be counted against the retention budget.

As more traces are force-retained, fewer traces will be selected by the Smart Gateway. In the extreme case, only force-retained traces will be selected, up to the maximum budget of retained traces per minute. We recommend that force-retained traces are used sparingly, to minimize impact to the Smart Gateway selection algorithm.

Force initiating spans 🔗

The Smart Gateway expects an initiating span to be sent within a trace, but this does not always happen. Sometimes an initiating span is not sent to the Smart Gateway. When no initiating span is identified, the Smart Gateway will not select the trace unless it is explicitely asked to do so by configuration.

When enabled, this functionality instructs the Smart Gateway to choose the earliest of the parent-most spans received after a fixed 5 minutes delay as the initiating span in the trace. To enable this feature, edit the TraceSample section in the signalfx forwarder config and set ForceTraceInitiatingSpan to true, as shown below.

Note about realms

A realm is a self-contained deployment of SignalFx in which your organization is hosted. Different realms have different API endpoints (e.g. the endpoint for sending data is ingest.us1.signalfx.com for the us1 realm, and ingest.eu0.signalfx.com for the eu0 realm).

Various statements in the instructions below include a YOUR_SIGNALFX_REALM placeholder that you should replace with the actual name of your realm. This realm name is shown on your profile page in SignalFx. If you do not include the realm name when specifying an endpoint, SignalFx will interpret it as pointing to the us0 realm.

{
  "ServerName": "smart-gateway-1",
  "LogDir": "/var/log/gateway",
  "ListenFrom": [
    {
      "Type": "signalfx",
      "ListenAddr": "0.0.0.0:8080"
    }
  ],
  "ForwardTo": [
    {
      "Type": "signalfx",
      "URL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/datapoint",
      "EventURL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v2/event",
      "TraceURL": "https://ingest.YOUR_SIGNALFX_REALM.signalfx.com/v1/trace",
      "DefaultAuthToken": "YOUR_SIGNALFX_API_TOKEN",
      "Name": "smart-gateway-forwarder",
      "TraceSample": {
        "BackupLocation": "/var/config/gateway/data",
        "ForceTraceInitiatingSpan": true
      }
    }
  ]
}