Integrating with Notification Services 🔗

Need some context? Detectors and Alerts

In addition to sending alert notifications to email addresses, you can configure SignalFx to send alert notifications to the services listed below.

About naming your integrations

The name you give an integration appears in the dropdown list when someone is adding recipients to a detector. If you plan to implement multiple integrations for a particular service, we suggest changing the name of each integration to something more meaningful than the default provided. Giving each integration a descriptive name ensures that alert notifications will be sent to the correct recipient.

Integrate with Amazon EventBridge 🔗

If you’re AWS user, SignalFx can automatically send alert notifications as partner events to your AWS account via Amazon EventBridge. You must be an administrator of your SignalFx account to connect SignalFx to Amazon EventBridge. Every time you connect SignalFx to Amazon EventBridge, you will:

Create Amazon EventBridge integration in SignalFx, using your AWS account Id
Accept SignalFx as an Event Source in your account

Please review this note about naming your integrations before you proceed.

Part 1 - Create Amazon EventBridge integration in SignalFx 🔗

Log in to SignalFx and click the Integrations tab to open the Integrations page. Look for the Amazon EventBridge tile. You can search for it by name, or find it in the Notification Services section.
Click the Amazon EventBridge tile, then click New Integration to display the configuration options.
Provide your AWS account id in the field AWS Account Id. This is the only account where you will be able to receive SignalFx events. If you need more than one, please create more integrations to support that.
By default, the name of the integration is Amazon EventBridge. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You can change the name to be more descriptive; doing so is highly recommended if you want to create more than one Amazon EventBridge integration.
Field Event Source incorporates the Partner Name and Event Source Id together in the form: Partner Name/Event Source Id. You will need it later when accepting the Event Source in your AWS account - copy it for future use.
From the AWS Region dropdown menu, choose the region where you want the Event Source created.
Click the Save and Enable button. You don’t need to perform any additional steps on this page. Continue to the next section to complete the SignalFx integration process in your AWS account.

Part 2 - Accepting SignalFx as an Event Source in Amazon EventBridge 🔗

Having copied the Event Source name, please refer to Amazon EventBridge documentation for steps to accept the source.

Add an Amazon EventBridge notification to a detector 🔗

Create, edit, or subscribe to a detector that you want to have sent events via Amazon EventBridge (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Click Add recipient, select Amazon EventBridge, then select the integration name that specifies where the notification should be sent.

SignalFx will now send alert notifications as events via Amazon EventBridge whenever the detector rule conditions are met. It will also send the notification when the value goes back to normal.

Integrate with BigPanda 🔗

SignalFx can send notifications to BigPanda when an alert is triggered by a detector and when the alert clears.

You must be an administrator in SignalFx and in BigPanda to create a BigPanda integration. Any SignalFx user can send notifications to a BigPanda integration once it’s been created.

The process of setting up an integration with BigPanda requires a BigPanda admin to create a new integration in BigPanda.

Please review this note about naming your integrations before you proceed.

Set up an integration in BigPanda 🔗

Log into BigPanda, display the Integrations page, and then click New Integration.
Hover over the REST API tile and click Integrate.
Type a name for the integration, then click Generate App Key.
The integration name and app key will be displayed. Copy the app key for later use.
In the Step 2 section, click the REST API tile if it is not selected. In the Step 3 section, copy the token following “Authorization: Bearer” for later use.

You don’t need to perform any additional steps on this page. Continue to the next section to complete the SignalFx integration process.

Set up a BigPanda integration in SignalFx 🔗

To install this integration, you must know the BigPanda app key and token to associate with SignalFx.

Open SignalFx and click Integrations to open the Integrations page. Look for the tile named BigPanda. You can search for it by name, or find it in the Notification Services section.
Click the BigPanda tile, then click New Integration to display the configuration options.
By default, the name of the integration is BigPanda. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You can change the name to be more descriptive; doing so is highly recommended if you want to create more than one BigPanda integration.
Enter the BigPanda app key and token that you copied earlier, then click Save. A message appears that says “Validated!”. If an error appears instead, double-check the app key and token that you pasted. Please contact support@signalfx.com for help resolving errors.

You are now ready to configure detectors to send notifications to BigPanda.

Configure a Detector to Report to BigPanda 🔗

Create, edit, or subscribe to a detector for which you want alert notifications to be sent to BigPanda (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Select the name of your BigPanda integration as a notfication recipient. SignalFx will now send a notification to BigPanda whenever the detector rule conditions are met and when the alert clears.

In addition to sending a subject, description and other information to BigPanda, the integration maps certain detector information in SignalFx to corresponding BigPanda properties as follows:

SignalFx information	BigPanda property	Property value
Alert severity is Critical	status	Critical
Alert severity is Major, Minor, Warning, or Informational	status	Warning
Alert is cleared or manually resolved, or detector is stopped	status	OK
Detector rule name	check	Detector rule name
Metric has a dimension named `cluster`	cluster	Value of the `cluster` dimension
Metric has a dimension named `host`	host	Value of the `host` dimension
Metric has any other dimension(s)	Custom properties, each named `sfx_<dimension-name>`	Value of the dimension

Note that if there are any name collisions between SignalFx dimensions and BigPanda status or check properties, SignalFx creates a new custom property in BigPanda. For example, if there is a dimension named status associated with the metric, SignalFx creates a custom property named sfx_status and stores the value of the status dimension there.

Integrate with Office 365 (Microsoft Teams) 🔗

Integrating SignalFx with Office 365 allows you to send alert notifications from detectors to a Microsoft Teams channel. To add an Office 365 integration, you must be an administrator of your SignalFx organization and must be a member of the Microsoft team you want to connect to.

Please review this note about naming your integrations before you proceed.

Open SignalFx and click Integrations to open the Integrations page. Look for the tile named Office 365. You can search for it by name, or find it in the Notification Services section.
Click the Office 365 tile, then click New Integration. If you are not logged in to Office 365, you will be prompted to log in. You will see the option to select a team to connect to.
From the Add to a team dropdown menu, choose the team you want to connect to. Review the SignalFx privacy policy and terms of use, then do one of the following:
- If the team contains only one channel, click Available.
- If the team contains multiple channels, click Install.
Select the channel you want to connect to, then click Set up.
Click Visit site to install. (Note that clicking Done does not create the connector. It simply closes the dialog box.)

You will return to the Office 365 integration screen in SignalFx.
By default, the name of the integration is Office 365. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You can change the name to be more descriptive (for example, the name of the team and/or channel); doing so is highly recommended if you want to create more than one Office 365 integration.

You are now ready to configure detectors to send notifications to a Microsoft Team.

Configure a Detector to Report to a Microsoft Team 🔗

Create, edit, or subscribe to a detector for which you want alert notifications to be sent to an Office 365 channel (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector).
Select Office 365 as the notification recipient.
If there are multiple Office 365 integrations, select the name of the desired integration.

SignalFx will now send a notification to the channel whenever the detector rule conditions are met. It will also send a notification when the alert clears.

Integrate with Opsgenie 🔗

SignalFx can send notifications to one ore more Opsgenie teams automatically when an alert is triggered by a detector. To do this, you first add an API integration in Opsgenie, then you add an associated Opsgenie integration in SignalFx.

You must be an administrator of your SignalFx organization to add or modify integrations in SignalFx.

On the Opsgenie side, there are two ways to add an integration for SignalFx.

Add a SignalFx integration to multiple Opsgenie teams
Add a SignalFx integration to a single Opsgenie team

Add a SignalFx integration to multiple Opsgenie teams 🔗

This type of integration (not available for all Opsgenie accounts) can send alert notifications to multiple teams. To get started, you need an Opsgenie API key.

Log into your Opsgenie account and navigate to the Integration list page.
Search for or navigate to the API icon. Hover over it and click Add.
Enter a name for the integration, then copy the API key displayed on the page; you will need to paste it into SignalFx. Do not make any other changes. Click Save Integration.
Skip to Add Opsgenie integration in SignalFx to complete the integration in SignalFx.

Add a SignalFx integration to a single Opsgenie team 🔗

This type of integration will send alert notifications to a single Opsgenie team. To get started, you need an Opsgenie API key.

Log into your Opsgenie account and navigate to the Team Dashboard.
Click the name of the Opsgenie team that should receive notifications from SignalFx.
Display the Integrations tab and then click Add Integration.
Search for or navigate to the API icon. Hover over it and click Add.
Enter a name for the integration, then copy the API key displayed on the page; you will need to paste it into SignalFx. Click Save Integration.
Continue with Add Opsgenie integration in SignalFx (below) to complete the integration in SignalFx.

Add Opsgenie integration in SignalFx 🔗

Open SignalFx and click Integrations to open the Integrations page. Look for the tile named Opsgenie. You can search for it by name, or find it in the Notification Services section.
Click the Opsgenie tile, then click New Integration to display the configuration options.
Specify the name that should appear when you’re configuring a detector rule to send an alert notification. You will probably want to use the Opsgenie team or integration name; see this note about naming your integrations.
Paste your Opsgenie API key into the API key field, select your service region from the dropdown menu, then click Save.

You are now ready to configure detectors to send notifications to Opsgenie.

Add an Opsgenie notification to a detector 🔗

Create, edit, or subscribe to a detector that you want to send notifications to Opsgenie (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Click Add recipient, select Opsgenie, then select the integration name that specifies where the notifications should be sent.

If you select an integration associated with a particular team (see Add a SignalFx integration to a single Opsgenie team), notifications will be sent to that team. If you select an integration that can send notifications to multiple teams (see Add a SignalFx integration to multiple Opsgenie teams), you have two options:

Select a team to send the notification to a particular team instead of having Opsgenie determine how to handle the notification.
Select “No team” to indicate that Opsgenie should handle the notification based on settings you specified for the integration associated with the API key.

SignalFx will now notify the specified Opsgenie team(s) whenever the detector rule conditions are met.

Integrate with PagerDuty 🔗

SignalFx can open PagerDuty incidents automatically when an alert is triggered by a detector. You must be an administrator of your SignalFx organization to add or modify integrations.

Please review this note about naming your integrations before you proceed.

Create a PagerDuty Key 🔗

To integrate with SignalFx, you need a PagerDuty Service API Key. To get a key, follow PagerDuty’s instructions on Creating Services and Adding Integrations. Under Integration Settings in the Integration Type field, open the Select a Tool pulldown, and then select SignalFx.

After you have added the service, you’ll find the Integration Key in the Integration Settings section of the service page in PagerDuty. Copy the Integration key; you will need to paste it into SignalFx.

Add PagerDuty integration 🔗

Open SignalFx and click the Integrations tab to open the Integrations page. Look for the PagerDuty tile. You can search for it by name, or find it in the Notification Services section.
Click the PagerDuty tile, then click New Integration to display the configuration options.
By default, the name of the integration is PagerDuty. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You can change the name to be more descriptive; doing so is highly recommended if you want to create more than one PagerDuty integration to map to different escalation policies within PagerDuty.
Enter your PagerDuty Integration Key into the Integration Key field, and then click Save.

You are now ready to configure detectors to send notifications to PagerDuty.

Add a PagerDuty notification to a detector 🔗

Create, edit, or subscribe to a detector that you want to have trigger PagerDuty incidents (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Click Add recipient, select PagerDuty, then select the integration name that specifies where the notification should be sent.

SignalFx will now trigger PagerDuty incidents whenever the detector rule conditions are met. It will also clear the incident when the value goes back to normal.

Integrate with ServiceNow 🔗

You must be an administrator in SignalFx to create a new ServiceNow integration. Any SignalFx user can send notifications to a ServiceNow integration once it’s been created.

Please review this note about naming your integrations before you proceed.

The process of setting up your first integration with ServiceNow requires a ServiceNow admin to create a user in ServiceNow. The instructions in Set up a ServiceNow user show how to do this. Once the ServiceNow user has been created, you can create multiple ServiceNow integrations for that user.

If you know the ServiceNow instance name, username and password for SignalFx in ServiceNow, you can skip to Set up ServiceNow integration in SignalFx.

Set up a ServiceNow user 🔗

Log into your ServiceNow instance, e.g. example.service-now.com. If necessary, display the Service Management dashboard.
In the navigation panel on the left, scroll to User Administration and then click on Users.
To create a new user, click on the New button at the top of the screen.
Enter a User ID, First name and Last name that reflect that this user is associated with Signalfx. (The entries below are simply suggestions.) Enter a password. Make sure the checkbox that says Active is checked. Then click Submit to create the user.

Note

Make a note of the User ID and password, as you will need them when you are adding the integration in SignalFx.
To find your new user, either search for the User ID or do a reverse chronological sort on the Created column. Click on the User ID to open the user information window, scroll down and display the Roles tab, then click Edit.
In the Collection search box, type “web_service_admin”. Select the “web_service_admin” role and then click on > to move it the Roles List panel.
Similarly, search for “itil”. Select the “itil” role and move it to the Roles List panel. Then click Save. You should now see web_service_admin and itil under the Roles tab for this user (and possibly additional inherited roles).

Continue to the next section to complete the SignalFx integration process.

Set up ServiceNow integration in SignalFx 🔗

To install this integration, you must know the ServiceNow instance name, username and password associated with SignalFx.

Open SignalFx and click Integrations to open the Integrations page. Look for the tile named ServiceNow. You can search for it by name, or find it in the Notification Services section.
Click the ServiceNow tile, then click New Integration to display the configuration options.
By default, the name of the integration is ServiceNow. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You can change the name to be more descriptive; doing so is highly recommended if you want to create more than one ServiceNow integration.
Enter the ServiceNow username, password, and instance name. Note that the instance name must be in the format “example.service-now.com”. Do not include a leading http:// or a trailing /.
Choose whether you want SignalFx notifications to create a ServiceNow Incident (generally recommended) or Problem, then click Save.

You are now ready to configure detectors to send notifications to ServiceNow.

Tip

You may wish to create a second ServiceNow integration with a different issue type, so you can create either an Incident or a Problem depending on the detector rule that is sending the notification. Simply give each integration a different name. All the rest of the information remains the same.

Configure a Detector to Report to ServiceNow 🔗

Create, edit, or subscribe to a detector for which you want alert notifications to be sent to a ServiceNow issue (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Select the name of your ServiceNow integration as a notfication recipient. SignalFx will now send a notification to ServiceNow whenever the detector rule conditions are met. When the alert clears, SignalFx will send a notification that sets the state as “Resolved” in ServiceNow .

The ServiceNow integration maps alert severity in SignalFx to the Impact and Urgency fields in the corresponding ServiceNow incident as follows:

SignalFx Severity	ServiceNow Impact and Urgency
Critical	1
Major or Minor	2
Warning or Info	3

Integrate with Slack 🔗

Integrating SignalFx with Slack allows you to send alert notifications from detectors to a Slack channel. To add a Slack integration, you must be an administrator of your SignalFx organization and must be authorized to add apps to Slack.

Please review this note about naming your integrations before you proceed.

Open SignalFx and click Integrations to open the Integrations page. Look for the tile named Slack. You can search for it by name, or find it in the Notification Services section.
Click the Slack tile, then click New Integration.

Note

If you see an error message after clicking New Integration, you aren’t authorized to add apps to Slack and will not be able to add this integration. Contact your Slack administrator for assistance.
Review the permissions required by Slack, then click Authorize.

You will return to the Slack integration screen in SignalFx.
By default, the name of the integration is Slack. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You can change the name to be more descriptive; doing so is highly recommended if you want to create more than one Slack integration.
Click Save.

You are now ready to configure detectors to send notifications to Slack.

Recommended

This method of integrating with Slack replaces a prior design. If you have prior Slack integrations installed, you will see them listed in the Configure tab with an Upgrade option that says Click here to upgrade this integration. While the prior implementations will continue to work, we recommend you upgrade them all to use the newer integration method. You must upgrade if you want image previews to appear when you paste a chart’s URL into Slack.

Configure a Detector to Report to Slack 🔗

Create, edit, or subscribe to a detector for which you want alert notifications to be sent to a Slack channel (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector).
Select Slack as the notification recipient. If there are multiple Slack integrations, select the name of the desired integration.
Enter or select the name of the Slack channel to which the notifications should be sent. (You can start typing to display a list of matching channel names to choose from.)

SignalFx will now send a notification to the channel whenever the detector rule conditions are met. It will also send a notification when the alert clears.

Integrate with VictorOps 🔗

Integrating with VictorOps allows you to send alert notifications from SignalFx detectors to your VictorOps timeline.

You must be an administrator in both SignalFx and VictorOps to create a new VictorOps integration. Any SignalFx user can send notifications to a VictorOps integration once it’s been created.

Please review this note about naming your integrations before you proceed.

First: Enable the REST Endpoint Integration in VictorOps 🔗

Open VictorOps, click Settings, then click Integrations.
From the list of Incoming Alerts integrations on the right side of the page, click REST Endpoint.
If the text box labelled Post URL says “Enable this integration to generate settings”, click Enable Integration to generate a URL.
Copy the URL in the Post URL box to your clipboard.

VictorOps is now ready to receive data through the REST Endpoint.

Next: Create a VictorOps integration in SignalFx 🔗

Open SignalFx and click Integrations to open the Integrations page. Look for the tile named VictorOps. You can search for it by name, or find it in the Notification Services section.
Click the VictorOps tile, then click New Integration.
By default, the name of the integration is VictorOps. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You can change the name to be more descriptive; doing so is highly recommended if you want to create more than one VictorOps integration.
In the Post URL field, paste in the Post URL that you copied to your clipboard above.
Click Save. A message appears that says “Validated!”. If an error appears instead, double-check the URL that you pasted. Please contact support@signalfx.com for help resolving errors.

You are now ready to configure detectors to send notifications to VictorOps.

Add a VictorOps notification to a detector 🔗

Create, edit, or subscribe to a detector for which you want alert notifications to be sent to VictorOps (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). SignalFx will now send a notification to VictorOps whenever the detector rule conditions are met and when the alert clears. When you configure a detector to send alerts to VictorOps, you can supply a routing key.

Integrate with xMatters 🔗

To set up xMatters as an alert notification recipient, follow the instructions on the xMatters website.

Send notifications via a webhook URL 🔗

You can specify a webhook URL to receive notifications when an alert is triggered or cleared. The request will be an application/JSON POST with the following key/values making up the JSON object in the body:

-  detector (string): Name of this detector.
-  detectorUrl (string): URL of the detector, which includes a parameter to select
   this specific incident.
-  detectorId (string): ID of the detector
-  description (string, optional): Description of the detector
-  imageUrl (string): URL of the alert preview image
-  incidentId: Unique identifier for this alert notification.
-  eventType (string): Uniquely identifies the version of the detector that sent the
   notification.
-  rule (string): Name of the detector rule that triggered the alert.
-  severity (string): Severity level of this rule.
-  runbookUrl (string, optional): Runbook URL specified in this rule.
-  tip (string, optional): Tip specified in this rule.
-  messageTitle (string): Notification title for this rule
-  messageBody (string): Notification message for this rule
-  detectOnCondition (string): The trigger metric data and detection criteria in
   this rule, in SignalFlow format.
-  detectOffCondition (string, optional): The clear metric data and detection
   criteria in this rule, in SignalFlow format.
-  status: (kept for backwards compatibility; use statusExtended to receive
   more information)
   (string): The state of this incident, with one of the following values:
      -  "anomalous" -- the alert is firing because the detect conditions are met
      -  "ok" -- the alert was cleared because the detect conditions were no longer
          met or the clear conditions (if any) were met
-  statusExtended: The state of this incident, with one of the following values:
      -  "anomalous" -- the alert is firing because the detect conditions are met
      -  "ok" -- the alert was cleared because the detect conditions were no longer
          met or the clear conditions (if any) were met
      -  "manually resolved" -- a user resolved the alert through the UI or the API
      -  "stopped" -- the detector that triggered the alert was edited or deleted
-  timestamp (string): The time the event occurred, in ISO 8601 format.
-  inputs: The map of the inputs involved in this rule (see "Webhook inputs" below)
-  sf_schema (integer): The schema version for this event (value always = 2)

Tip

Webhooks can be shared across multiple detectors. On the Integrations page, click the Webhooks icon in the Notification services section. Enter a webhook URL, name it, and add custom HTTP headers as well. These webhooks will be recipient targets when you add a recipient to a detector.

Webhook inputs 🔗

As noted above, “inputs” is one of the elements in the JSON object. Each input is named after the program variable it is bound to; if not bound (anonymous input), it will have a name in the form “_S0”, “_S1”, etc.

Each input is an object, containing:

a key

the key object is a map of the dimensions of the input signal

the key object may be empty if there are no dimensions (rare)

the key field may not be present if the input was a static value (static thresholds for example / comparisons against scalar values)

a value

the value of that input at the time of the event (when the alert fires, and when it clears)

a fragment

the fragment of the SignalFlow program that represents this input

may be useful to display in a notification to “explain” what that input is

may not be present for some detectors or for static, anonymous inputs

Note about realms

A realm is a self-contained deployment of SignalFx in which your organization is hosted. Different realms have different API endpoints (e.g. the endpoint for sending data is ingest.us1.signalfx.com for the us1 realm, and ingest.eu0.signalfx.com for the eu0 realm).

Various statements in the instructions below include a YOUR_SIGNALFX_REALM placeholder that you should replace with the actual name of your realm. This realm name is shown on your profile page in SignalFx. If you do not include the realm name when specifying an endpoint, SignalFx will interpret it as pointing to the us0 realm.

Webhook example 1 🔗

The following example illustrates the parameters for a webhook, with sample values.

{
  "detector": "Memory usage detector",
  "detectorUrl": "https://app.YOUR_SIGNALFX_REALM.signalfx.com/#/detector/ABCDEFGHIJK/edit",
  "description": "A detector which alerts when memory usage exceeeds 90% for 10 minutes",
  "incidentId": "BCDEFGHIJKL",
  "eventType": "foo",
  "rule": "Running out of memory",
  "severity": "Minor",
  "description": "Memory has reached 90% of maximum for 10 minutes",
  "detectOnCondition": "when(A > 90, '10m')",
  "detectOffCondition": "when(A < 90, '15m')",
  "status": "ok",
  "statusExtended": "ok",
  "imageUrl": "https://org.YOUR_SIGNALFX_REALM.signalfx.com/#/chart/abCDefGHij",
  "timestamp": "2016-11-08T19:43:30Z",
  "inputs": {
    "_S1": {
      "dimensions": {
        "host": "i-346235qa",
        "plugin": "signalfx-metadata"
      },
      "value": 96.235234634345,
      "fragment": "data('memory.utilization')"
    }
  },
  "sf_schema": 2
}

Webhook example 2 🔗

Here’s another, more complex example of an alert emitted by an anomaly detector created using the SignalFx API. In this example, each host is emitting a metric called latency per API endpoint, so each datapoint will have 3 dimensions: the endpoint, the host, and the data center. The detector is comparing the 99th percentile of the latency of all the APIs of a particular host against the 99th percentile of the latencies of all the hosts in its data center. It alerts if the host latency is greater than the data center latency AND the latter is greater than 40 ms.

{
   "sf_schema": 2,
   "detector": "My detector",
   "detectorUrl": "https://app.YOUR_SIGNALFX_REALM.signalfx.com/#/detector/<id>/edit",
   "incidentId": "<id>",
   "eventType": "<event-type>",
   "rule": "My detector rule",
   "severity": "Critical",
   "description": "Latency of host myserver is 43.4, over datacenter-wide latency of 42.9 !",
   "status": "anomalous",
   "statusExtended": "anomalous",
   "imageUrl": "https://org.YOUR_SIGNALFX_REALM.signalfx.com/#/chart/abCDefGHij",
   "timestamp": "2016-10-25T21:19:38Z",
   "detectOnCondition": "when(a > b and b > 40)",
   "inputs": {
      "a": {
         "key": {
            "host": "myserver",
            "dc": "us-west-1"
         },
         "value": 43.4,
      "fragment": "data('latency').p99(by=['host', 'dc'])"
      },
      "b": {
         "key": {
            "dc": "us-west-1"
         },
         "value": 42.9,
         "fragment": "data('latency').p99(by='dc')"
      },
      "_S2": {
      "value": 40,
      "fragment": "40"
      }
   }
}