Integrating with Notification Services 🔗

Need some context? Detectors and Alerts

In addition to sending alert notifications via email, you can configure SignalFx to send alert notifications to the services listed below.

About naming your integrations

The name you give an integration appears in the dropdown list when someone is adding recipients to a detector. If you plan to implement multiple integrations for a particular service, we suggest changing the name of each integration to something more meaningful than the default provided. Giving each integration a descriptive name ensures that alert notifications will be sent to the correct recipient.

Integrate with Amazon EventBridge 🔗

If you’re an AWS user, SignalFx can automatically send alert notifications as partner events to your AWS account via Amazon EventBridge.

You must be an administrator of your SignalFx account to connect SignalFx to Amazon EventBridge. Every time you connect SignalFx to Amazon EventBridge, you will:

Create Amazon EventBridge integration in SignalFx, using your AWS account Id
Accept SignalFx as an Event Source in your account

Please review this note about naming your integrations before you proceed.

Part 1 - Create Amazon EventBridge integration in SignalFx 🔗

Log in to SignalFx and click the Integrations tab to open the Integrations page. Look for the Amazon EventBridge tile. You can search for it by name, or find it in the Notification Services section.
Click the Amazon EventBridge tile, then click Create New Integration or New Integration to display the configuration options.
In the AWS Account Id field, enter the ID for the AWS account that will receive SignalFx events. If you want to send events to additional AWS accounts, then you must create additional integrations for each AWS account.
By default, the name of the integration is Amazon EventBridge. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You can change the name to be more descriptive; doing so is highly recommended if you want to create more than one Amazon EventBridge integration.
Field Event Source incorporates the Partner Name and Event Source Id together in the form: Partner Name/Event Source Id. You will need it later when accepting the Event Source in your AWS account - copy it for future use.
From the AWS Region dropdown menu, choose the region where you want the Event Source created.
Click the Save and Enable button. You don’t need to perform any additional steps on this page. Continue to the next section to complete the SignalFx integration process in your AWS account.

Part 2 - Accepting SignalFx as an Event Source in Amazon EventBridge 🔗

Having copied the Event Source name, please refer to Amazon EventBridge documentation for steps to accept the source.

Add an Amazon EventBridge notification to a detector 🔗

Create, edit, or subscribe to a detector that you want to have sent events via Amazon EventBridge (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Click Add recipient, select Amazon EventBridge, then select the integration name that specifies where the notification should be sent.

SignalFx will now send alert notifications as events via Amazon EventBridge whenever the detector rule conditions are met. It will also send the notification when the value goes back to normal.

Integrate with BigPanda 🔗

SignalFx can send notifications to BigPanda when an alert is triggered by a detector and when the alert clears.

You must be an administrator in SignalFx and in BigPanda to create a BigPanda integration. Any SignalFx user can send notifications to a BigPanda integration once it’s been created.

The process of setting up an integration with BigPanda requires a BigPanda admin to create a new integration in BigPanda.

Please review this note about naming your integrations before you proceed.

Set up an integration in BigPanda 🔗

Log into BigPanda, display the Integrations page, and then click New Integration.
Hover over the REST API tile and click Integrate.
Type a name for the integration, then click Generate App Key.
The integration name and app key will be displayed. Copy the app key for later use.
In the Step 2 section, click the REST API tile if it is not selected. In the Step 3 section, copy the token following “Authorization: Bearer” for later use.

You don’t need to perform any additional steps on this page. Continue to the next section to complete the SignalFx integration process.

Set up a BigPanda integration in SignalFx 🔗

To install this integration, you must know the BigPanda app key and token to associate with SignalFx.

Open SignalFx and click Integrations to open the Integrations page. Look for the tile named BigPanda. You can search for it by name, or find it in the Notification Services section.
Click the BigPanda tile, then click Create New Integration or New Integration to display the configuration options.
By default, the name of the integration is BigPanda. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You can change the name to be more descriptive; doing so is highly recommended if you want to create more than one BigPanda integration.
Enter the BigPanda app key and token that you copied earlier, then click Save. A message appears that says “Validated!”. If an error appears instead, double-check the app key and token that you pasted. Please contact signalfx-support@splunk.com for help resolving errors.

You are now ready to configure detectors to send notifications to BigPanda.

Configure a Detector to Report to BigPanda 🔗

Create, edit, or subscribe to a detector for which you want alert notifications to be sent to BigPanda (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Select the name of your BigPanda integration as a notfication recipient. SignalFx will now send a notification to BigPanda whenever the detector rule conditions are met and when the alert clears.

In addition to sending a subject, description and other information to BigPanda, the integration maps certain detector information in SignalFx to corresponding BigPanda properties as follows:

SignalFx information	BigPanda property	Property value
Alert severity is Critical	status	Critical
Alert severity is Major, Minor, Warning, or Informational	status	Warning
Alert is cleared or manually resolved, or detector is stopped	status	OK
Detector rule name	check	Detector rule name
Metric has a dimension named `cluster`	cluster	Value of the `cluster` dimension
Metric has a dimension named `host`	host	Value of the `host` dimension
Metric has any other dimension(s)	Custom properties, each named `sfx_<dimension-name>`	Value of the dimension

Note that if there are any name collisions between SignalFx dimensions and BigPanda status or check properties, SignalFx creates a new custom property in BigPanda. For example, if there is a dimension named status associated with the metric, SignalFx creates a custom property named sfx_status and stores the value of the status dimension there.

Integrate with Jira 🔗

This integration supports both Jira Cloud and Jira Server. You must be an administrator in SignalFx to create the integration. In Jira the user account to create the integration must have permissions to browse projects, create issues, and add comments.

After integration, when an alert is triggered by a detector, SignalFx notifies Jira to create a new issue. When the alert condition clears, a comment is added to the issue.

Please review this note about naming your integrations before you proceed.

For Jira Cloud integrations, see the Jira documentation for making an API Token. You will use this API Token to continue with the integration steps below.
For Jira Server integrations, you will use a dedicated username and password for this integration.

Note: Currently, this integration supports (meaning that the fields can be automatically set when posting an issue) the following fields: project, issue type, summary, reporter, description, and assignee. If a Jira project is configured to require any additional fields when creating an issue, the integration setup will give you a validation error, and the integration cannot be made.

To configure your new integration, complete the steps below.

Open SignalFx and click Integrations to open the Integrations page. Look for the tile named Jira. You can search for it by name, or find it in the Notification Services section.
Click the Jira tile, then click Create New Integration or New Integration to display the configuration options page. See the screen shot below.

For the JIRA Base URL, use https://YOUR-DOMAIN.atlassian.net or http://YOUR-HOSTNAME:PORT

For Jira Cloud, provide the email and API token.

For Jira Server, provide the username and password.

For Project, select a project from the list of all the projects that you have access to in Jira, and then click Apply.

For Issue Type, select an issue type and then click Apply.

To set the Assignee for the tickets created by SignalFx Notification Service, you can name the Assignee in this integration or in the detector. The detector setting takes precedence over the integration setting, so in case you set two different Assignees, the one from the detector will be used. In that way, you can use a default Assignee (blank) for the integration, and selectively change it for some detectors. If your JIRA instance doesn’t require it, you can leave the Assignee setting blank. To learn more about setting the default Assignee in Jira, see this link or equivalent for your version of Jira product https://confluence.atlassian.com/jiracorecloud/editing-a-project-s-details-938021482.html.

When you are finished filling in the fields, click Save. The name of your new integration displays at the top of the list.

Test your integration (optional) 🔗

Return to the Integrations tile, select the integration you just made, and then click Create Test Issue.

A test Jira ticket is created for the named Assignee. After a short delay, a comment is made on the same ticket that states the alert has cleared. After both of these events have occurred, the status is indicated on the popup that is shown below. Click Close to dismiss the popup.

Your Jira integration is now complete and validated.

Add a Jira notification to a detector 🔗

Create, edit, or subscribe to a detector for which you want alert notifications to be sent via Jira (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector).
Click Add recipient, and select Jira.
From the dropdown list of Jira integrations, select the integration name that may include the assignee where the notification will be sent. You can overwrite the named integration assignee (or a blank if none was assigned) by typing in a new assignee. Currently, only a single Jira recipient per detector is supported.
When you are finished, click Done. You can then activate the detector you have configured.

SignalFx will now create a Jira issue based on an alert notification whenever the detector rule condition is met. It will also add a comment to that issue when the alert condition clears.

Integrate with Microsoft Teams 🔗

Overview 🔗

SignalFx can send notifications to a Microsoft Teams channel when an alert is triggered by a detector and when the alert clears.

At a high level, to integrate SignalFx with Microsoft Teams, you must:

Add the SignalFx Events connector to your channel in Microsoft Teams

Add your channel’s integration into SignalFx

Update an existing detector to send alerts

Note that existing Office 365 integrations will appear in SignalFx within the Microsoft Teams tile.

Prerequisites 🔗

To fully integrate SignalFx with Microsoft Teams, make sure that:

You are already a member of a Microsoft Teams channel; verify that the channel that will receive the triggered alerts from SignalFx has been created. (For your reference, this channel is also called the reporting channel.)

You must be an administrator of your SignalFx organization to add and modify integrations in SignalFx.

Step 1: Retrieve your channel’s webhook URL 🔗

Log into Microsoft Teams, and then navigate to the list of teams.
Select the desired team, and then expand the list of corresponding channels.
Locate and hover over the desired reporting channel, click the ellipses (…), and then select Connectors.

If you do not see Connectors, then you may not have permission to add a new connector. Contact the owner of the team to update your permissions.

Locate the incoming SignalFx Events connector, and then click Add or Configure.

If this connector has not been added to any channel in your team, then you will see Add.

If this connector has been added to a channel in your team, then you will see Configure.

(Optional) Update the default connection name. (After this step, you will not be able to change the connector’s name.)
Copy the webhook URL. You will need this information in a later step.
Click Save.

Step 2: Add your channel’s integration into SignalFx 🔗

Open SignalFx, and then click Integrations to open the Integrations page. Locate the Microsoft Teams tile.
Click the Microsoft Teams tile, and then click Create New Integration or New Integration to display the configuration options page.
By default, the name of the integration is Microsoft Teams. This name will appear when you update detector rules for alert notifications. SignalFx recommends that you change the default name of the integration to be more descriptive, especially if you plan to create more Microsoft Teams integrations. (To learn more about naming integrations, see About naming your integrations.)
In Webhook URL, paste the URL you had previously copied, and then click Save. After a moment, a Validated! message will appear. When you see this validation message, return to the reporting channel in Microsoft Teams to verify that another validation message was sent.

If you see an error message, verify that the webhook URL you entered is correct.

For additional troubleshooting information see Troubleshooting Microsoft Teams.

Step 3: Update an existing detector to send alerts 🔗

You can use these instructions to add your integration to an existing detector with rules and alerts.

(To learn more about detectors, including how to create a detector and add alerts, see Set Up Detectors to Trigger Alerts.)

In SignalFx, click Alerts on the navigation bar, and then click Detectors.
In the table, locate the desired detector, and then click the corresponding ellipses (…).
Click Manage Subscriptions.
In the window that appears, click Add Recipient, and then select Microsoft Teams.
Select the name of the desired integration.
Now, when the conditions for the detector’s rules are met, SignalFx will send a notification to the channel. Similarly, when the alert clears, a notification will also be sent.

Troubleshooting Microsoft Teams 🔗

If you cannot complete the integration or connection process, consider that:

You may have not copied or pasted the webhook URL correctly.
- In the SignalFx UI, if you receive an error message regarding the validation of the webhook URL, then you may have copied or pasted the webhook URL incorrectly.
- To troubleshoot, return to the Microsoft Team’s configuration page for the SignalFx Events connector and use the clipboard button to copy the entire webhook URL. Then, return to the SignalFx UI and paste the URL.
- Review the following example of a complete webhook URL: ”https://outlook.office.com/webhook/5ab8c38e-8b49-47fe-8134-4ff61715a23b@e06c3f21-566e-4e85-ad74-619dea7eef88/32a5e6c4-c9dd-4c12-89a8-34d2c496312f/aazzaazzaazzaazzaazzaazzaazzaazz/5c0bf566-05bc-4755-9f10-e3dbbb19a2b8”
You may not have properly saved the SignalFx connection in Microsoft Teams.
- Before you can create an integration in the SignalFx UI, you must first properly create and save the connection in Microsoft Teams.
- To troubleshoot, return to the Microsoft Team’s configuration page for the SignalFx Events connector, click Save, and then return to the SignalFx UI to create the integration.

If the previously configured reporting channel is no longer receiving notifications:

Verify that the SignalFx connection in Microsoft Teams still exists.
- To troubleshoot, in the SignalFx UI, navigate to the Microsoft Teams tile, expand the desired integration, click the ellipses, and then select Validate.
- If you see an error message, specifically Connector configuration not found, then the SignalFx Events connector was removed from the Microsoft Teams channel and must be re-established. Follow the steps outlined in this document to add a new connection and configure a new integration.
Research any possible configuration changes, such as:
- A different reporting channel has been added to the integration.
- The integration is no longer associated with the detector.
- The detector’s alert rules have been updated, which would cause notifications to be sent for different reasons.

Integrate with Opsgenie 🔗

SignalFx can send notifications to one or more Opsgenie teams automatically when an alert is triggered by a detector. To do this, you first add an API integration in Opsgenie, then you add an associated Opsgenie integration in SignalFx.

You must be an administrator of your SignalFx organization to add or modify integrations in SignalFx.

On the Opsgenie side, there are two ways to add an integration for SignalFx.

Add a SignalFx integration to multiple Opsgenie teams
Add a SignalFx integration to a single Opsgenie team

Add a SignalFx integration to multiple Opsgenie teams 🔗

This type of integration (not available for all Opsgenie accounts) can send alert notifications to multiple teams. To get started, you need an Opsgenie API key.

Log into your Opsgenie account and navigate to the Integration list page.
Search for or navigate to the API icon. Hover over it and click Add.
Enter a name for the integration, then copy the API key displayed on the page; you will need to paste it into SignalFx. Do not make any other changes. Click Save Integration.
Skip to Add Opsgenie integration in SignalFx to complete the integration in SignalFx.

Add a SignalFx integration to a single Opsgenie team 🔗

This type of integration will send alert notifications to a single Opsgenie team. To get started, you need an Opsgenie API key.

Log into your Opsgenie account and navigate to the Team Dashboard.
Click the name of the Opsgenie team that should receive notifications from SignalFx.
Display the Integrations tab and then click Add Integration.
Search for or navigate to the API icon. Hover over it and click Add.
Enter a name for the integration, then copy the API key displayed on the page; you will need to paste it into SignalFx. Click Save Integration.
Continue with Add Opsgenie integration in SignalFx (below) to complete the integration in SignalFx.

Add Opsgenie integration in SignalFx 🔗

Open SignalFx and click Integrations to open the Integrations page. Look for the tile named Opsgenie. You can search for it by name, or find it in the Notification Services section.
Click the Opsgenie tile, then click Create New Integration or New Integration to display the configuration options.
Specify the name that should appear when you’re configuring a detector rule to send an alert notification. You will probably want to use the Opsgenie team or integration name; see this note about naming your integrations.
Paste your Opsgenie API key into the API key field, select your service region from the dropdown menu, then click Save.

You are now ready to configure detectors to send notifications to Opsgenie.

Add an Opsgenie notification to a detector 🔗

Create, edit, or subscribe to a detector that you want to send notifications to Opsgenie (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Click Add recipient, select Opsgenie, then select the integration name that specifies where the notifications should be sent.

If you select an integration associated with a particular team (see Add a SignalFx integration to a single Opsgenie team), notifications will be sent to that team. If you select an integration that can send notifications to multiple teams (see Add a SignalFx integration to multiple Opsgenie teams), you have two options:

Select a team to send the notification to a particular team instead of having Opsgenie determine how to handle the notification.
Select “No team” to indicate that Opsgenie should handle the notification based on settings you specified for the integration associated with the API key.

SignalFx will now notify the specified Opsgenie team(s) whenever the detector rule conditions are met.

Integrate with PagerDuty 🔗

SignalFx can open PagerDuty incidents automatically when an alert is triggered by a detector. You must be an administrator of your SignalFx organization to add or modify integrations.

Please review this note about naming your integrations before you proceed.

Create a PagerDuty Key 🔗

To integrate with SignalFx, you need a PagerDuty Service API Key. To get a key, follow PagerDuty’s instructions on Creating Services and Adding Integrations. Under Integration Settings in the Integration Type field, open the Select a Tool pulldown, and then select SignalFx.

After you have added the service, you’ll find the Integration Key in the Integration Settings section of the service page in PagerDuty. Copy the Integration key; you will need to paste it into SignalFx.

Add PagerDuty integration 🔗

Open SignalFx and click the Integrations tab to open the Integrations page. Look for the PagerDuty tile. You can search for it by name, or find it in the Notification Services section.
Click the PagerDuty tile, then click Create New Integration or New Integration to display the configuration options.
By default, the name of the integration is PagerDuty. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You can change the name to be more descriptive; doing so is highly recommended if you want to create more than one PagerDuty integration to map to different escalation policies within PagerDuty.
Enter your PagerDuty Integration Key into the Integration Key field, and then click Save.

You are now ready to configure detectors to send notifications to PagerDuty.

Add a PagerDuty notification to a detector 🔗

Create, edit, or subscribe to a detector that you want to have trigger PagerDuty incidents (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Click Add recipient, select PagerDuty, then select the integration name that specifies where the notification should be sent.

SignalFx will now trigger PagerDuty incidents whenever the detector rule conditions are met. It will also clear the incident when the value goes back to normal.

Integrate with ServiceNow 🔗

You must be an administrator in SignalFx to create a new ServiceNow integration. Any SignalFx user can send notifications to a ServiceNow integration once it’s been created.

Please review this note about naming your integrations before you proceed.

The process of setting up your first integration with ServiceNow requires a ServiceNow admin to create a user in ServiceNow. The instructions in Set up a ServiceNow user show how to do this. Once the ServiceNow user has been created, you can create multiple ServiceNow integrations for that user.

If you know the ServiceNow instance name, username and password for SignalFx in ServiceNow, you can skip to Set up ServiceNow integration in SignalFx.

Set up a ServiceNow user 🔗

Log into your ServiceNow instance, e.g. example.service-now.com. If necessary, display the Service Management dashboard.
In the navigation panel on the left, scroll to User Administration and then click on Users.
To create a new user, click on the New button at the top of the screen.
Enter a User ID, First name and Last name that reflect that this user is associated with Signalfx. (The entries below are simply suggestions.) Enter a password. Make sure the checkbox that says Active is checked. Then click Submit to create the user.

Note

Make a note of the User ID and password, as you will need them when you are adding the integration in SignalFx.
To find your new user, either search for the User ID or do a reverse chronological sort on the Created column. Click on the User ID to open the user information window, scroll down and display the Roles tab, then click Edit.
In the Collection search box, type “web_service_admin”. Select the “web_service_admin” role and then click on > to move it the Roles List panel.
Similarly, search for “itil”. Select the “itil” role and move it to the Roles List panel. Then click Save. You should now see web_service_admin and itil under the Roles tab for this user (and possibly additional inherited roles).

Continue to the next section to complete the SignalFx integration process.

Set up ServiceNow integration in SignalFx 🔗

To install this integration, you must know the ServiceNow instance name, username and password associated with SignalFx.

Open SignalFx and click Integrations to open the Integrations page. Look for the tile named ServiceNow. You can search for it by name, or find it in the Notification Services section.
Click the ServiceNow tile, then click Create New Integration or New Integration to display the configuration options.
By default, the name of the integration is ServiceNow. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You can change the name to be more descriptive; doing so is highly recommended if you want to create more than one ServiceNow integration.
Enter the ServiceNow username, password, and instance name. Note that the instance name must be in the format “example.service-now.com”. Do not include a leading http:// or a trailing /.
- To troubleshoot potential blind server-side request forgeries (SSRF), SignalFx has whitelisted *.service-now.com. As a result, if you enter a domain name that is rejected by SignalFx, you may need to contact Support to update the list of allowed domain names. Additionally, you cannot enter local ServiceNow instances.
Choose whether you want SignalFx notifications to create a ServiceNow Incident (generally recommended) or Problem, then click Save.

You are now ready to configure detectors to send notifications to ServiceNow.

Tip

You may wish to create a second ServiceNow integration with a different issue type, so you can create either an Incident or a Problem depending on the detector rule that is sending the notification. Simply give each integration a different name. All the rest of the information remains the same.

Configure a Detector to Report to ServiceNow 🔗

Create, edit, or subscribe to a detector for which you want alert notifications to be sent to a ServiceNow issue (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Select the name of your ServiceNow integration as a notfication recipient. SignalFx will now send a notification to ServiceNow whenever the detector rule conditions are met. When the alert clears, SignalFx will send a notification that sets the state as “Resolved” in ServiceNow .

The ServiceNow integration maps alert severity in SignalFx to the Impact and Urgency fields in the corresponding ServiceNow incident as follows:

SignalFx Severity	ServiceNow Impact and Urgency
Critical	1
Major or Minor	2
Warning or Info	3

Integrate with Slack 🔗

Integrating SignalFx with Slack allows you to send alert notifications from detectors to a Slack channel. To add a Slack integration, you must be an administrator of your SignalFx organization and must be authorized to add apps to Slack.

Please review this note about naming your integrations before you proceed.

Open SignalFx and click Integrations to open the Integrations page. Look for the tile named Slack. You can search for it by name, or find it in the Notification Services section.
Click the Slack tile, then click Create New Integration or New Integration.

Note

If you see an error message after clicking New Integration, you aren’t authorized to add apps to Slack and will not be able to add this integration. Contact your Slack administrator for assistance.
Review the permissions required by Slack, then click Authorize.

You will return to the Slack integration screen in SignalFx.
By default, the name of the integration is Slack. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You can change the name to be more descriptive; doing so is highly recommended if you want to create more than one Slack integration.
Click Save.

You are now ready to configure detectors to send notifications to Slack.

Recommended

This method of integrating with Slack replaces a prior design. If you have prior Slack integrations installed, you will see them listed in the Configure tab with an Upgrade option that says Click here to upgrade this integration. While the prior implementations will continue to work, we recommend you upgrade them all to use the newer integration method. You must upgrade if you want image previews to appear when you paste a chart’s URL into Slack.

Configure a Detector to Report to Slack 🔗

Create, edit, or subscribe to a detector for which you want alert notifications to be sent to a Slack channel (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector).
Select Slack as the notification recipient. If there are multiple Slack integrations, select the name of the desired integration.
Enter or select the name of the Slack channel to which the notifications should be sent. (You can start typing to display a list of matching channel names to choose from.)

SignalFx will now send a notification to the channel whenever the detector rule conditions are met. It will also send a notification when the alert clears.

Integrate with VictorOps 🔗

Integrating with VictorOps allows you to send alert notifications from SignalFx detectors to your VictorOps timeline.

Prerequisites 🔗

You must be an administrator in both SignalFx and VictorOps to create a new VictorOps integration. After a VictorOps integration has been created, any SignalFx user can send notifications.

Please review this note about naming your integrations before you proceed.

Step 1: Locate and copy the endpoint (URL) in VictorOps 🔗

Open VictorOps, click Integrations.
In 3rd Party Integrations, locate and select SignalFx.
If you you do not see a URL, then click Enable Integration to generate a URL.
Copy the entire URL, including $routing_key.

Step 2: Create a VictorOps integration in SignalFx 🔗

Open SignalFx, and click Integrations to open the Integrations page. Look for the tile named VictorOps. You can search for it by name, or find it in the Notification Services section.
Click the VictorOps tile, then click New Integration or Create New Integration.
By default, the name of the integration is VictorOps. This name will appear when you configure a detector rule to send an alert notification. SignalFx recommends that you change the name to be more descriptive, especially if you plan to create multiple VictorOps integrations.
In Post URL, paste the endpoint (URL) you copied from VictorOps.
Click Save, and then the “Validated!” message will appear. (If you see an error message, verify the endpoint [URL] again. If you still receive an error message, then contact signalfx-support@splunk.com.)

You are now ready to configure detectors to send notifications to VictorOps.

Step 3: Add a VictorOps notification to a detector 🔗

Create, edit, or subscribe to a detector for which you want alert notifications to be sent to VictorOps (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector).
Select VictorOps as a notification recipient and specify a routing key. If there are multiple VictorOps integrations, select the name of the desired integration.

SignalFx will now send a notification to VictorOps whenever the detector rule conditions are met and when the alert clears.

(Optional) Review available fields 🔗

When you receive a notification in your VictorOps timeline, note that the notification contains information provided by both SignalFx and VictorOps. Based on your alert type and detection settings, review the following information provided by SignalFx:

Field	Description
Detector Definition	This field displays a link that you can click to access the detector in your SignalFx account to view the corresponding alert rules.
Graph	This field displays a snapshot view of the signal that triggered the alert.
detector	This field displays the name of the detector in your SignalFx account.
inputs	This field provides detailed information about the alert, including the rule and detector name, alert triggering conditions, and signal details.
rule	This field displays the name of the alert rule in SignalFx where the conditions to trigger alert events and clear events were defined.
entity_display_name	This field displays the SignalFx rule and detector name. (This information also appears in the rule and detector fields within the notification.)
state_message	When the alert triggers, this field displays the alert’s severity, which includes critical, major, minor, warning, or info. When the alert is resolved, this field displays back to normal, stopped, or manually resolved.
entity_id	This field displays the incident’s ID in a string.
monitoring_tool	By default, this field displays signalfx.
message_type	This field displays the severity of the alert, which includes critical, warning, acknowledgement, info, and recovery.

Integrate with xMatters 🔗

To set up xMatters as an alert notification recipient, follow the instructions on the xMatters website.

Send notifications via a webhook URL 🔗

You can specify a webhook URL to receive notifications when an alert is triggered or cleared. The request will be an application/JSON POST with the following key/values making up the JSON object in the body:

-  detector (string): Name of this detector.
-  detectorUrl (string): URL of the detector, which includes a parameter to select
   this specific incident.
-  detectorId (string): ID of the detector
-  description (string, optional): Description of the detector
-  imageUrl (string): URL of the alert preview image
-  incidentId: Unique identifier for this alert notification.
-  eventType (string): Uniquely identifies the version of the detector that sent the
   notification.
-  rule (string): Name of the detector rule that triggered the alert.
-  severity (string): Severity level of this rule.
-  runbookUrl (string, optional): Runbook URL specified in this rule.
-  tip (string, optional): Tip specified in this rule.
-  messageTitle (string): Notification title for this rule
-  messageBody (string): Notification message for this rule
-  detectOnCondition (string): The trigger metric data and detection criteria in
   this rule, in SignalFlow format.
-  detectOffCondition (string, optional): The clear metric data and detection
   criteria in this rule, in SignalFlow format.
-  status: (kept for backwards compatibility; use statusExtended to receive
   more information)
   (string): The state of this incident, with one of the following values:
      -  "anomalous" -- the alert is firing because the detect conditions are met
      -  "ok" -- the alert was cleared because the detect conditions were no longer
          met or the clear conditions (if any) were met
-  statusExtended: The state of this incident, with one of the following values:
      -  "anomalous" -- the alert is firing because the detect conditions are met
      -  "ok" -- the alert was cleared because the detect conditions were no longer
          met or the clear conditions (if any) were met
      -  "manually resolved" -- a user resolved the alert through the UI or the API
      -  "stopped" -- the detector that triggered the alert was edited or deleted
-  timestamp (string): The time the event occurred, in ISO 8601 format.
-  inputs: The map of the inputs involved in this rule (see "Webhook inputs" below)
-  sf_schema (integer): The schema version for this event (value always = 2)

Tip

Webhooks can be shared across multiple detectors. On the Integrations page, click the Webhooks icon in the Notification services section. Enter a webhook URL, name it, and add custom HTTP headers as well. These webhooks will be recipient targets when you add a recipient to a detector.

Webhook inputs 🔗

As noted above, “inputs” is one of the elements in the JSON object. Each input is named after the program variable it is bound to; if not bound (anonymous input), it will have a name in the form “_S0”, “_S1”, etc.

Each input is an object, containing:

a key

the key object is a map of the dimensions of the input signal

the key object may be empty if there are no dimensions (rare)

the key field may not be present if the input was a static value (static thresholds for example / comparisons against scalar values)

a value

the value of that input at the time of the event (when the alert fires, and when it clears)

a fragment

the fragment of the SignalFlow program that represents this input

may be useful to display in a notification to “explain” what that input is

may not be present for some detectors or for static, anonymous inputs

Note about realms

A realm is a self-contained deployment of SignalFx in which your organization is hosted. Different realms have different API endpoints (e.g. the endpoint for sending data is ingest.us1.signalfx.com for the us1 realm, and ingest.eu0.signalfx.com for the eu0 realm).

Various statements in the instructions below include a YOUR_SIGNALFX_REALM placeholder that you should replace with the actual name of your realm. This realm name is shown on your profile page in SignalFx. If you do not include the realm name when specifying an endpoint, SignalFx will interpret it as pointing to the us0 realm.

Webhook example 1 🔗

The following example illustrates the parameters for a webhook, with sample values.

{
  "detector": "Memory usage detector",
  "detectorUrl": "https://app.YOUR_SIGNALFX_REALM.signalfx.com/#/detector/ABCDEFGHIJK/edit",
  "description": "A detector which alerts when memory usage exceeeds 90% for 10 minutes",
  "incidentId": "BCDEFGHIJKL",
  "eventType": "foo",
  "rule": "Running out of memory",
  "severity": "Minor",
  "description": "Memory has reached 90% of maximum for 10 minutes",
  "detectOnCondition": "when(A > 90, '10m')",
  "detectOffCondition": "when(A < 90, '15m')",
  "status": "ok",
  "statusExtended": "ok",
  "imageUrl": "https://org.YOUR_SIGNALFX_REALM.signalfx.com/#/chart/abCDefGHij",
  "timestamp": "2016-11-08T19:43:30Z",
  "inputs": {
    "_S1": {
      "dimensions": {
        "host": "i-346235qa",
        "plugin": "signalfx-metadata"
      },
      "value": 96.235234634345,
      "fragment": "data('memory.utilization')"
    }
  },
  "sf_schema": 2
}

Webhook example 2 🔗

Here’s another, more complex example of an alert emitted by an anomaly detector created using the SignalFx API. In this example, each host is emitting a metric called latency per API endpoint, so each datapoint will have 3 dimensions: the endpoint, the host, and the data center. The detector is comparing the 99th percentile of the latency of all the APIs of a particular host against the 99th percentile of the latencies of all the hosts in its data center. It alerts if the host latency is greater than the data center latency AND the latter is greater than 40 ms.

{
   "sf_schema": 2,
   "detector": "My detector",
   "detectorUrl": "https://app.YOUR_SIGNALFX_REALM.signalfx.com/#/detector/<id>/edit",
   "incidentId": "<id>",
   "eventType": "<event-type>",
   "rule": "My detector rule",
   "severity": "Critical",
   "description": "Latency of host myserver is 43.4, over datacenter-wide latency of 42.9 !",
   "status": "anomalous",
   "statusExtended": "anomalous",
   "imageUrl": "https://org.YOUR_SIGNALFX_REALM.signalfx.com/#/chart/abCDefGHij",
   "timestamp": "2016-10-25T21:19:38Z",
   "detectOnCondition": "when(a > b and b > 40)",
   "inputs": {
      "a": {
         "key": {
            "host": "myserver",
            "dc": "us-west-1"
         },
         "value": 43.4,
      "fragment": "data('latency').p99(by=['host', 'dc'])"
      },
      "b": {
         "key": {
            "dc": "us-west-1"
         },
         "value": 42.9,
         "fragment": "data('latency').p99(by='dc')"
      },
      "_S2": {
      "value": 40,
      "fragment": "40"
      }
   }
}