Integrate with notification services
Need some context? Detectors and Alerts
In addition to sending alert notifications via email, you can configure Splunk Infrastructure Monitoring to send alert notifications to the services listed below.
About naming your integrations
The name you give an integration appears in the drop-down list when someone is adding recipients to a detector. If you plan to implement multiple integrations for a particular service, you should change each integration’s name to something more meaningful than the default provided. Giving each integration a descriptive name ensures that the correct recipient receives an alert notification.
Integrate with Amazon EventBridge
If you’re an AWS user, Splunk Infrastructure Monitoring can automatically send alert notifications as partner events to your AWS account via Amazon EventBridge.
You must be an administrator of your account to connect the application to Amazon EventBridge. Every time you connect Infrastructure Monitoring to Amazon EventBridge, you will:
- Create Amazon EventBridge integration in Infrastructure Monitoring, using your AWS account Id
- Accept Infrastructure Monitoring as an Event Source in your account
See this note about naming your integrations before you proceed.
Part 1 - Create Amazon EventBridge integration in Splunk Infrastructure Monitoring
- Log in to Infrastructure Monitoring and click the Integrations tab to open the Integrations page. Look for the Amazon EventBridge tile. You can search for it by name, or find it in the Notification Services section.
- Click the Amazon EventBridge tile, then click Create New Integration or New Integration to display the configuration options.
- In the AWS Account Id field, enter the ID for the AWS account that will receive Infrastructure Monitoring events. If you want to send events to additional AWS accounts, then you must create additional integrations for each AWS account.
- By default, the name of the integration is Amazon EventBridge. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You should change the name to be more descriptive; doing so is highly recommended if you want to create more than one Amazon EventBridge integration.
- Field Event Source incorporates the Partner Name and Event Source Id together in the form: Partner Name/Event Source Id. You will need it later when accepting the Event Source in your AWS account - copy it for future use.
- From the AWS Region drop-down list, choose the region where you want the Event Source created.
- Click the Save and Enable button. You don’t need to perform any additional steps on this page. Continue to the next section to complete the Infrastructure Monitoring integration process in your AWS account.
Part 2 - Accept Splunk Infrastructure Monitoring as an event source in Amazon EventBridge
Once you have copied the Event Source name, see Amazon EventBridge documentation for steps to accept the source.
Add an Amazon EventBridge notification to a detector
Create, edit, or subscribe to a detector that you want to have sent events via Amazon EventBridge (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Click Add recipient, select Amazon EventBridge, then select the integration name that specifies where the notification should be sent.
Infrastructure Monitoring will now send alert notifications as events via Amazon EventBridge whenever the detector rule conditions are met. It will also send the notification when the value goes back to normal.
Integrate with BigPanda
Splunk Infrastructure Monitoring can send notifications to BigPanda when an alert is triggered by a detector and when the alert clears.
You must be an administrator in the application and in BigPanda to create a BigPanda integration. Any Infrastructure Monitoring user can send notifications to a BigPanda integration once it’s been created.
The process of setting up an integration with BigPanda requires a BigPanda admin to create a new integration in BigPanda.
See this note about naming your integrations before you proceed.
Set up an integration in BigPanda
Log in to BigPanda, display the Integrations page, and then click New Integration.
Hover over the REST API tile and click Integrate.
Type a name for the integration, then click Generate App Key.
The integration name and app key will be displayed. Copy the app key for later use.
Click the REST API tile if it is not selected. Copy the token following “Authorization: Bearer” for later use.
You don’t need to perform any additional steps on this page. Continue to the next section to complete the Infrastructure Monitoring integration process.
Set up a BigPanda integration in Splunk Infrastructure Monitoring
To install this integration, you must know the BigPanda app key and token to associate with Infrastructure Monitoring.
Open the application and click Integrations to open the Integrations page. Look for the tile named BigPanda. You can search for it by name, or find it in the Notification Services section.
Click the BigPanda tile, then click Create New Integration or New Integration to display the configuration options.
By default, the name of the integration is BigPanda. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You should change the name to be more descriptive; doing so is highly recommended if you want to create more than one BigPanda integration.
Enter the BigPanda app key and token that you copied earlier, then click Save. A message appears that says “Validated!”. If an error appears instead, double-check the app key and token that you pasted. Contact signalfx-support@splunk.com for help resolving errors.
You are now ready to configure detectors to send notifications to BigPanda.
Integrate with Jira
This integration supports both Jira Cloud and Jira Server. You must be an administrator in Splunk Infrastructure Monitoring to create the integration. In Jira, the user account to create the integration must have permissions to browse projects, create issues, and add comments.
After integration, when a detector triggers an alert, Infrastructure Monitoring notifies Jira to create a new issue. When the alert condition clears, a comment is added to the issue.
See this note about naming your integrations before you proceed.
- For Jira Cloud integrations, see Jira documentation for making an API Token. You will use this API Token to continue with the integration steps below.
- For Jira Server integrations, you will use a dedicated username and password for this integration.
Note: This integration supports (meaning that the fields can be automatically set when posting an issue) the following fields: project, issue type, summary, reporter, description, and assignee. If a Jira project is configured to require any additional fields when creating an issue, the integration setup will give you a validation error, and the integration cannot be made.
To configure your new integration, complete the steps below.
Open Infrastructure Monitoring and click Integrations to open the Integrations page. Look for the tile named Jira. You can search for it by name, or find it in the Notification Services section.
Click the Jira tile, then click Create New Integration or New Integration to display the configuration options page. See the screen shot below.
- For the Jira Base URL, use
https://YOUR-DOMAIN.atlassian.net
or http://YOUR-HOSTNAME:PORT
- For Jira Cloud, provide the email and API token.
- For Jira Server, provide the username and password.
- For Project, select a project from the list of all the projects that you have access to in Jira, and then click Apply.
- For Issue Type, select an issue type and then click Apply.
- To set the Assignee for the tickets created by the Infrastructure Monitoring Notification Service, you can name the Assignee in this integration or in the detector. The detector setting takes precedence over the integration setting, so in case you set two different Assignees, the one from the detector will be used. In that way, you can use a default Assignee (blank) for the integration, and selectively change it for some detectors. If your Jira instance doesn’t require it, you can leave the Assignee setting blank. To learn more about setting the default Assignee in Jira, see Edit a project’s details or the equivalent for your version of the Jira product .
When you are finished filling in the fields, click Save. The name of your new integration displays at the top of the list.
Test your integration (optional)
- Return to the Integrations tile, select the integration you just made, and then click Create Test Issue.
A test Jira ticket is created for the named Assignee. After a short delay, a comment is made on the same ticket that states the alert has cleared. After both of these events have occurred, the status is indicated on the window, as shown below.
Add a Jira notification to a detector
- Create, edit, or subscribe to a detector for which you want alert notifications to be sent through Jira (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector).
- Click Add recipient, and select Jira.
- From the drop-down list of Jira integrations, select the integration name that may include the assignee where the notification will be sent. You can overwrite the named integration assignee (or a blank if none was assigned) by typing in a new assignee. Currently, only a single Jira recipient per detector is supported.
- Click Done. You can then activate the detector you have configured.
Infrastructure Monitoring will now create a Jira issue based on an alert notification whenever the detector rule condition is met. It will also add a comment to that issue when the alert condition clears.
Integrate with Microsoft Teams
Overview
Splunk Infrastructure Monitoring can send notifications to a Microsoft Teams channel when an alert is triggered by a detector and when the alert clears.
At a high level, to integrate Infrastructure Monitoring with Microsoft Teams, you must:
- Add the Infrastructure Monitoring Events connector to your channel in Microsoft Teams
- Add your channel’s integration into Infrastructure Monitoring
- Update an existing detector to send alerts
Note that existing Office 365 integrations will appear in Infrastructure Monitoring within the Microsoft Teams tile.
Prerequisites
To fully integrate Infrastructure Monitoring with Microsoft Teams, make sure that:
- You are already a member of a Microsoft Teams channel; verify that the channel that will receive the triggered alerts from Infrastructure Monitoring. For your reference, this channel is also called the reporting channel.
- You must be an administrator of your Infrastructure Monitoring organization to add and modify integrations in the application.
Step 1: Retrieve your channel’s webhook URL
- Log in to Microsoft Teams, and then navigate to the list of teams.
- Select the desired team, and then expand the list of corresponding channels.
- Locate and hover over the desired reporting channel, click the ellipses (…), and then select Connectors.
- If you do not see Connectors, then you may not have permission to add a new connector. Contact the owner of the team to update your permissions.
- Locate the incoming Events connector, and then click Add or Configure.
- If this connector has not been added to any channel in your team, then you will see Add.
- If this connector has been added to a channel in your team, then you will see Configure.
(Optional) Update the default connection name. After this step, you will not be able to change the connector’s name.
Copy the webhook URL. You will need this information in a later step.
Click Save.
Step 2: Add your channel’s integration into Infrastructure Monitoring
Open Infrastructure Monitoring, and then click Integrations to open the Integrations page. Locate the Microsoft Teams tile.
Click the Microsoft Teams tile, and then click Create New Integration or New Integration to display the configuration options page.
By default, the name of the integration is Microsoft Teams. This name will appear when you update detector rules for alert notifications. You should change the default name of the integration to be more descriptive, especially if you plan to create more Microsoft Teams integrations. To learn more about naming integrations, see About naming your integrations.
In Webhook URL, paste the URL you previously copied, and then click Save. After a moment, a Validated! message will appear. When you see this validation message, return to the reporting channel in Microsoft Teams to verify that another validation message was sent.
- If you see an error message, verify that the webhook URL you entered is correct.
- For additional troubleshooting information see Troubleshooting Microsoft Teams.
Step 3: Update an existing detector to send alerts
You can use these instructions to add your integration to an existing detector with rules and alerts. To learn more about detectors, including how to create a detector and add alerts, see Set Up Detectors to Trigger Alerts.
- In Infrastructure Monitoring, click Alerts > Detectors.
- In the table, locate the desired detector, and then click the corresponding ellipses (…).
- Click Manage Subscriptions.
- In the window that appears, click Add Recipient, and then select Microsoft Teams.
- Select the name of the desired integration.
Now, when the conditions for the detector’s rules are met, Infrastructure Monitoring will send a notification to the channel. Similarly, when the alert clears, a notification will also be sent.
Troubleshoot Microsoft Teams
If you cannot complete the integration or connection process, consider that:
- You may have not copied or pasted the webhook URL correctly.
-
- You may not have properly saved the Infrastructure Monitoring connection in Microsoft Teams:
- Before you can create an integration in the Infrastructure Monitoring UI, you must first properly create and save the connection in Microsoft Teams.
- To troubleshoot, return to the Microsoft Team’s configuration page for the Events connector, click Save, and then return to the Infrastructure Monitoring UI to create the integration.
If the previously configured reporting channel is no longer receiving notifications:
- Verify that the Infrastructure Monitoring connection in Microsoft Teams still exists.
- To troubleshoot, in the Infrastructure Monitoring UI, navigate to the Microsoft Teams tile, expand the desired integration, click the ellipses, and then select Validate.
- If you see an error message, specifically Connector configuration not found, then the Events connector was removed from the Microsoft Teams channel and must be re-established. Follow the steps outlined in this document to add a new connection and configure a new integration.
- Research any possible configuration changes, such as:
- A different reporting channel has been added to the integration.
- The integration is no longer associated with the detector.
- The detector’s alert rules have been updated, which would cause notifications to be sent for different reasons.
Integrate with Opsgenie
Splunk Infrastructure Monitoring can automatically send notifications to one or more Opsgenie teams when a detector triggers an alert. To do this, you first add an API integration in Opsgenie, then you add an associated Opsgenie integration in Infrastructure Monitoring.
You must be an administrator of your Infrastructure Monitoring organization to add or modify integrations in the application.
On the Opsgenie side, there are two ways to add an integration for Infrastructure Monitoring:
Add an Infrastructure Monitoring integration to multiple Opsgenie teams
This type of integration (not available for all Opsgenie accounts) can send alert notifications to multiple teams. To get started, you need an Opsgenie API key.
Log into your Opsgenie account and navigate to the Integration list page.
Search for or navigate to the API icon. Hover over it and click Add.
Enter a name for the integration, then copy the API key displayed on the page; you will need to paste it into Infrastructure Monitoring. Do not make any other changes. Click Save Integration.
Skip to Add an Opsgenie integration in Infrastructure Monitoring to complete the integration.
Add an Infrastructure Monitoring integration to a single Opsgenie team
This type of integration will send alert notifications to a single Opsgenie team. To get started, you need an Opsgenie API key.
Log into your Opsgenie account and navigate to the Team Dashboard.
Click the name of the Opsgenie team that should receive notifications from SignalFx.
Display the Integrations tab and then click Add Integration.
Search for or navigate to the API icon. Hover over it and click Add.
Enter a name for the integration, then copy the API key displayed on the page; you will need to paste it into Infrastructure Monitoring. Click Save Integration.
Continue to Add an Opsgenie integration in Infrastructure Monitoring to complete the integration.
Add an Opsgenie integration in Infrastructure Monitoring
Open Infrastructure Monitoring and click Integrations to open the Integrations page. Look for the tile named Opsgenie. You can search for it by name, or find it in the Notification Services section.
Click the Opsgenie tile, then click Create New Integration or New Integration to display the configuration options.
Specify the name that should appear when you’re configuring a detector rule to send an alert notification. You should use the Opsgenie team or integration name; see this note about naming your integrations.
Paste your Opsgenie API key into the API key field, select your service region from the drop-down list, and click Save.
You are now ready to configure detectors to send notifications to Opsgenie.
Add an Opsgenie notification to a detector
Create, edit, or subscribe to a detector that you want to send notifications to Opsgenie (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Click Add recipient, select Opsgenie, then select the integration name that specifies where the notifications should be sent.
If you select an integration associated with a particular team (see Add an Infrastructure Monitoring integration to a single Opsgenie team), notifications will be sent to that team. If you select an integration that can send notifications to multiple teams (see Add an Infrastructure Monitoring integration to multiple Opsgenie teams), you have two options:
- Select a team to send the notification to a particular team instead of having Opsgenie determine how to handle the notification.
- Select “No team” to indicate that Opsgenie should handle the notification based on settings you specified for the integration associated with the API key.
Infrastructure Monitoring will now notify the specified Opsgenie team(s) whenever the detector rule conditions are met.
Integrate with ServiceNow
You must be an administrator in Splunk Infrastructure Monitoring to create a new ServiceNow integration. Once created, any user of the application can send notifications to a ServiceNow integration.
See this note about naming your integrations before you proceed.
The process of setting up your first integration with ServiceNow requires a ServiceNow admin to create a user in ServiceNow. The instructions in Set up a ServiceNow user show how to do this. Once the ServiceNow user has been created, you can create multiple ServiceNow integrations for that user.
If you know the ServiceNow instance name, username and password for Infrastructure Monitoring in ServiceNow, you can skip to Set up ServiceNow integration in Infrastructure Monitoring.
Set up a ServiceNow user
Log into your ServiceNow instance, for example, example.service-now.com. If necessary, display the Service Management dashboard.
In the navigation panel on the left, scroll to User Administration and then click on Users.
Click New to create a new user.
Enter a User ID, first name and last name, that reflect that this user is associated with Infrastructure Monitoring . The examples shown below are only suggestions.
Enter a password. Make sure the checkbox that says Active is checked.
Click Submit to create the user.
Note
Make a note of the user ID and password, as you will need them when you are adding the integration in Infrastructure Monitoring.
To find your new user, either search for the user ID or do a reverse chronological sort on the Created column. Click the User ID to open the user information window, scroll down and display the Roles tab, then click Edit.
In the Collection search box, type “web_service_admin”. Select the “web_service_admin” role and then click > to move it the Roles List panel.
Similarly, search for “itil”. Select the “itil” role and move it to the Roles List panel. Now, click Save. You should now see web_service_admin and itil under the Roles tab for this user (and possibly additional inherited roles).
Continue to the next section to complete the Infrastructure Monitoring integration process.
Set up ServiceNow integration in Infrastructure Monitoring
To install this integration, you must know the ServiceNow instance name, username, and password associated with Infrastructure Monitoring
Open Infrastructure Monitoring and click Integrations to open the Integrations page. Look for the tile named ServiceNow. You can search for it by name, or find it in the Notification Services section.
Click the ServiceNow tile, then click Create New Integration or New Integration to display the configuration options.
By default, the name of the integration is ServiceNow. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You should change the name to be more descriptive; doing so is highly recommended if you want to create more than one ServiceNow integration.
Enter the ServiceNow username, password, and instance name. Note that the instance name must be in the format of “example.service-now.com”. Do not include a leading http:// or a trailing /.
- To troubleshoot potential blind server-side request forgeries (SSRF), Infrastructure Monitoring has whitelisted *.service-now.com. As a result, if you enter a domain name that is rejected by Infrastructure Monitoring, you may need to contact Support to update the list of allowed domain names. Additionally, you cannot enter local ServiceNow instances.
Choose whether you want Infrastructure Monitoring notifications to create a ServiceNow Incident (generally recommended) or Problem, then click Save.
You are now ready to configure detectors to send notifications to ServiceNow.
Tip
You may wish to create a second ServiceNow integration with a different issue type, so you can create either an Incident or a Problem depending on the detector rule that is sending the notification. Give each integration a different name. All the rest of the information remains the same.
Integrate with Slack
Integrating Splunk Infrastructure Monitoring with Slack allows you to send alert notifications from detectors to a Slack channel. To add a Slack integration, you must be an administrator of your Infrastructure Monitoring organization and must be authorized to add apps to Slack.
See this note about naming your integrations before you proceed.
Open Infrastructure Monitoring and click Integrations to open the Integrations page. Look for the tile named Slack. You can search for it by name, or find it in the Notification Services section.
Click the Slack tile, and then click Create New Integration or New Integration.
Note
If you see an error message after clicking New Integration, you aren’t authorized to add apps to Slack and will not be able to add this integration. Contact your Slack administrator for assistance.
Review the permissions required by Slack, then click Authorize. You will return to the Slack integration screen in Infrastructure Monitoring.
By default, the name of the integration is Slack. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You should change the name to be more descriptive; doing so is highly recommended if you want to create more than one Slack integration.
Click Save.
You are now ready to configure detectors to send notifications to Slack.
Recommended
This method of integrating with Slack replaces a prior design. If you have prior Slack integrations installed, you will see them listed in the Configure tab with an Upgrade option that says Click here to upgrade this integration. While the prior implementations will continue to work, you should upgrade all implementations to use the newer integration method. You must upgrade if you want image previews to appear when you paste a chart’s URL into Slack.
Integrate with Splunk On-Call
Integrating with Splunk On-Call allows you to send alert notifications from Splunk Infrastructure Monitoring detectors to your Splunk On-Call timeline.
Prerequisites
You must be an administrator in both Infrastructure Monitoring and Splunk On-Call to create a new Splunk On-Call integration. After a Splunk On-Call integration has been created, any Infrastructure Monitoring user can send notifications.
See this note about naming your integrations before you proceed.
Step 1: Locate and copy the endpoint (URL) in Splunk On-Call
Open Splunk On-Call, and click Integrations.
In 3rd Party Integrations, locate and select Splunk On-Call.
If you you do not see a URL, click Enable Integration to generate a URL.
Copy the entire URL, including $routing_key.
Step 2: Create a Splunk On-Call integration in Infrastructure Monitoring
Open Infrastructure Monitoring, and click Integrations to open the Integrations page. Look for the tile named Splunk On-Call. You can search for it by name, or find it in the Notification Services section.
Click the Splunk On-Call tile, then click New Integration or Create New Integration.
By default, the name of the integration is Splunk On-Call. This name will appear when you configure a detector rule to send an alert notification. You should change the name to be more descriptive, especially if you plan to create multiple Splunk On-Call integrations.
In Post URL, paste the endpoint (URL) you copied from Splunk On-Call.
Click Save, and then the “Validated!” message will appear. If you see an error message, verify the endpoint (URL) again. If you still receive an error message, then contact signalfx-support@splunk.com.
You are now ready to configure detectors to send notifications to Splunk On-Call.
Step 3: Add a Splunk On-Call notification to a detector
- Create, edit, or subscribe to a detector for which you want alert notifications to be sent to Splunk On-Call (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector).
- Select Splunk On-Call as a notification recipient and specify a routing key. If there are multiple Splunk On-Call integrations, select the name of the desired integration.
Infrastructure Monitoring will now send a notification to Splunk On-Call whenever the detector rule conditions are met and when the alert clears.
(Optional) Review available fields
When you receive a notification in your Splunk On-Call timeline, note that the notification contains information provided by both Infrastructure Monitoring and Splunk On-Call. Based on your alert type and detection settings, review the following information provided by Infrastructure Monitoring:
Field |
Description |
Detector Definition |
This field displays a link that you can click to access the detector in your Infrastructure Monitoring account to view the corresponding alert rules. |
Graph |
This field displays a snapshot view of the signal that triggered the alert. |
detector |
This field displays the name of the detector in your Infrastructure Monitoring account. |
inputs |
This field provides detailed information about the alert, including the rule and detector name, alert triggering conditions, and signal details. |
rule |
This field displays the name of the alert rule in SignalFx where the conditions to trigger alert events and clear events were defined. |
entity_display_name |
This field displays the SignalFx rule and detector name. This information also appears in the rule and detector fields within the notification. |
state_message |
When the alert triggers, this field displays the alert’s severity, which includes critical, major, minor, warning, or info. When the alert is resolved, this field displays back to normal, stopped, or manually resolved. |
entity_id |
This field displays the incident’s ID in a string. |
monitoring_tool |
By default, this field displays signalfx. |
message_type |
This field displays the severity of the alert, which includes critical, warning, acknowledgement, info, and recovery. |
Send notifications via a webhook URL
You can specify a webhook URL to receive notifications when an alert is triggered or cleared. The request will be an application/JSON POST with the following key/values making up the JSON object in the body:
- detector (string): Name of this detector.
- detectorUrl (string): URL of the detector, which includes a parameter to select
this specific incident.
- detectorId (string): ID of the detector
- description (string, optional): Description of the detector
- imageUrl (string): URL of the alert preview image
- incidentId: Unique identifier for this alert notification.
- eventType (string): Uniquely identifies the version of the detector that sent the
notification.
- rule (string): Name of the detector rule that triggered the alert.
- severity (string): Severity level of this rule.
- runbookUrl (string, optional): Runbook URL specified in this rule.
- tip (string, optional): Tip specified in this rule.
- messageTitle (string): Notification title for this rule
- messageBody (string): Notification message for this rule
- detectOnCondition (string): The trigger metric data and detection criteria in
this rule, in SignalFlow format.
- detectOffCondition (string, optional): The clear metric data and detection
criteria in this rule, in SignalFlow format.
- status: (kept for backwards compatibility; use statusExtended to receive
more information)
(string): The state of this incident, with one of the following values:
- "anomalous" -- the alert is firing because the detect conditions are met
- "ok" -- the alert was cleared because the detect conditions were no longer
met or the clear conditions (if any) were met
- statusExtended: The state of this incident, with one of the following values:
- "anomalous" -- the alert is firing because the detect conditions are met
- "ok" -- the alert was cleared because the detect conditions were no longer
met or the clear conditions (if any) were met
- "manually resolved" -- a user resolved the alert through the UI or the API
- "stopped" -- the detector that triggered the alert was edited or deleted
- timestamp (string): The time the event occurred, in ISO 8601 format.
- inputs: The map of the inputs involved in this rule (see "Webhook inputs" below)
- sf_schema (integer): The schema version for this event (value always = 2)
Tip
Webhooks can be shared across multiple detectors. On the Integrations page, click the Webhooks icon in the Notification services section. Enter a webhook URL, name it, and add custom HTTP headers as well. These webhooks will be recipient targets when you add a recipient to a detector.
Webhook example 1
The following example illustrates the parameters for a webhook, with sample values.
{
"detector": "Memory usage detector",
"detectorUrl": "https://app.YOUR_SIGNALFX_REALM.signalfx.com/#/detector/ABCDEFGHIJK/edit",
"description": "A detector which alerts when memory usage exceeeds 90% for 10 minutes",
"incidentId": "BCDEFGHIJKL",
"eventType": "foo",
"rule": "Running out of memory",
"severity": "Minor",
"description": "Memory has reached 90% of maximum for 10 minutes",
"detectOnCondition": "when(A > 90, '10m')",
"detectOffCondition": "when(A < 90, '15m')",
"status": "ok",
"statusExtended": "ok",
"imageUrl": "https://org.YOUR_SIGNALFX_REALM.signalfx.com/#/chart/abCDefGHij",
"timestamp": "2016-11-08T19:43:30Z",
"inputs": {
"_S1": {
"dimensions": {
"host": "i-346235qa",
"plugin": "signalfx-metadata"
},
"value": 96.235234634345,
"fragment": "data('memory.utilization')"
}
},
"sf_schema": 2
}
Webhook example 2
Here’s another, more complex example of an alert emitted by an anomaly detector created using the SignalFx API. In this example, each host is emitting a metric called latency
per API endpoint, so each data point will have 3 dimensions: the endpoint, the host, and the data center. The detector is comparing the 99th percentile of the latency of all the APIs of a particular host against the 99th percentile of the latencies of all the hosts in its data center. It alerts if the host latency is greater than the data center latency AND the latter is greater than 40 ms.
{
"sf_schema": 2,
"detector": "My detector",
"detectorUrl": "https://app.YOUR_SIGNALFX_REALM.signalfx.com/#/detector/<id>/edit",
"incidentId": "<id>",
"eventType": "<event-type>",
"rule": "My detector rule",
"severity": "Critical",
"description": "Latency of host myserver is 43.4, over datacenter-wide latency of 42.9 !",
"status": "anomalous",
"statusExtended": "anomalous",
"imageUrl": "https://org.YOUR_SIGNALFX_REALM.signalfx.com/#/chart/abCDefGHij",
"timestamp": "2016-10-25T21:19:38Z",
"detectOnCondition": "when(a > b and b > 40)",
"inputs": {
"a": {
"key": {
"host": "myserver",
"dc": "us-west-1"
},
"value": 43.4,
"fragment": "data('latency').p99(by=['host', 'dc'])"
},
"b": {
"key": {
"dc": "us-west-1"
},
"value": 42.9,
"fragment": "data('latency').p99(by='dc')"
},
"_S2": {
"value": 40,
"fragment": "40"
}
}
}