Docs » Administer Your Organization » Integrate with notification services

Integrate with notification services đź”—

Need some context?   Detectors and Alerts

In addition to sending alert notifications via email, you can configure Splunk Infrastructure Monitoring to send alert notifications to the services listed below.

About naming your integrations

The name you give an integration appears in the drop-down list when someone is adding recipients to a detector. If you plan to implement multiple integrations for a particular service, you should change each integration’s name to something more meaningful than the default provided. Giving each integration a descriptive name ensures that the correct recipient receives an alert notification.

Integrate with Amazon EventBridge đź”—

If you’re an AWS user, Splunk Infrastructure Monitoring can automatically send alert notifications as partner events to your AWS account via Amazon EventBridge.

You must be an administrator of your account to connect the application to Amazon EventBridge. Every time you connect Infrastructure Monitoring to Amazon EventBridge, you will:

  • Create Amazon EventBridge integration in Infrastructure Monitoring, using your AWS account Id
  • Accept Infrastructure Monitoring as an Event Source in your account

See this note about naming your integrations before you proceed.

Part 1 - Create Amazon EventBridge integration in Splunk Infrastructure Monitoring đź”—

  1. Log in to Infrastructure Monitoring and click the Integrations tab to open the Integrations page. Look for the Amazon EventBridge tile. You can search for it by name, or find it in the Notification Services section.
  2. Click the Amazon EventBridge tile, then click Create New Integration or New Integration to display the configuration options.
  3. In the AWS Account Id field, enter the ID for the AWS account that will receive Infrastructure Monitoring events. If you want to send events to additional AWS accounts, then you must create additional integrations for each AWS account.
  4. By default, the name of the integration is Amazon EventBridge. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You should change the name to be more descriptive; doing so is highly recommended if you want to create more than one Amazon EventBridge integration.
  5. Field Event Source incorporates the Partner Name and Event Source Id together in the form: Partner Name/Event Source Id. You will need it later when accepting the Event Source in your AWS account - copy it for future use.
  6. From the AWS Region drop-down list, choose the region where you want the Event Source created.
  7. Click the Save and Enable button. You don’t need to perform any additional steps on this page. Continue to the next section to complete the Infrastructure Monitoring integration process in your AWS account.

Part 2 - Accept Splunk Infrastructure Monitoring as an event source in Amazon EventBridge đź”—

Once you have copied the Event Source name, see Amazon EventBridge documentation for steps to accept the source.

Add an Amazon EventBridge notification to a detector đź”—

Create, edit, or subscribe to a detector that you want to have sent events via Amazon EventBridge (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Click Add recipient, select Amazon EventBridge, then select the integration name that specifies where the notification should be sent.

Infrastructure Monitoring will now send alert notifications as events via Amazon EventBridge whenever the detector rule conditions are met. It will also send the notification when the value goes back to normal.

Integrate with BigPanda đź”—

Splunk Infrastructure Monitoring can send notifications to BigPanda when an alert is triggered by a detector and when the alert clears.

You must be an administrator in the application and in BigPanda to create a BigPanda integration. Any Infrastructure Monitoring user can send notifications to a BigPanda integration once it’s been created.

The process of setting up an integration with BigPanda requires a BigPanda admin to create a new integration in BigPanda.

See this note about naming your integrations before you proceed.

Set up an integration in BigPanda đź”—

  1. Log in to BigPanda, display the Integrations page, and then click New Integration.

    ../_images/bp-new-integration.png


  2. Hover over the REST API tile and click Integrate.

    ../_images/bp-integrate.png


  3. Type a name for the integration, then click Generate App Key.

    ../_images/bp-app-key.png


  4. The integration name and app key will be displayed. Copy the app key for later use.

    ../_images/bp-copy-app-key.png


  5. Click the REST API tile if it is not selected. Copy the token following “Authorization: Bearer” for later use.

    ../_images/bp-copy-token.png

You don’t need to perform any additional steps on this page. Continue to the next section to complete the Infrastructure Monitoring integration process.

Set up a BigPanda integration in Splunk Infrastructure Monitoring đź”—

To install this integration, you must know the BigPanda app key and token to associate with Infrastructure Monitoring.

  1. Open the application and click Integrations to open the Integrations page. Look for the tile named BigPanda. You can search for it by name, or find it in the Notification Services section.

  2. Click the BigPanda tile, then click Create New Integration or New Integration to display the configuration options.

    ../_images/bp-in-signalfx.png


  3. By default, the name of the integration is BigPanda. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You should change the name to be more descriptive; doing so is highly recommended if you want to create more than one BigPanda integration.

  4. Enter the BigPanda app key and token that you copied earlier, then click Save. A message appears that says “Validated!”. If an error appears instead, double-check the app key and token that you pasted. Contact signalfx-support@splunk.com for help resolving errors.

You are now ready to configure detectors to send notifications to BigPanda.

Configure a detector to report to BigPanda đź”—

Create, edit, or subscribe to a detector for which you want alert notifications to be sent to BigPanda (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Select the name of your BigPanda integration as a notfication recipient. Infrastructure Monitoring will now send a notification to BigPanda whenever the detector rule conditions are met and when the alert clears.

In addition to sending a subject, description, and other information to BigPanda, the integration maps certain detector information in Infrastructure Monitoring to corresponding BigPanda properties as follows:

SignalFx information BigPanda property Property value
Alert severity is Critical status Critical
Alert severity is Major, Minor, Warning, or Informational status Warning
Alert is cleared or manually resolved, or detector is stopped status OK
Detector rule name check Detector rule name
Metric has a dimension named cluster cluster Value of the cluster dimension
Metric has a dimension named host host Value of the host dimension
Metric has any other dimension(s) Custom properties, each named sfx_<dimension-name> Value of the dimension

If there are any name collisions between Infrastructure Monitoring dimensions and BigPanda status or check properties, Infrastructure Monitoring creates a new custom property in BigPanda. For example, if there is a dimension named status associated with the metric, Infrastructure Monitoring creates a custom property named sfx_status and stores the value of the status dimension there.

Integrate with Jira đź”—

This integration supports both Jira Cloud and Jira Server. You must be an administrator in Splunk Infrastructure Monitoring to create the integration. In Jira, the user account to create the integration must have permissions to browse projects, create issues, and add comments.

After integration, when a detector triggers an alert, Infrastructure Monitoring notifies Jira to create a new issue. When the alert condition clears, a comment is added to the issue.

See this note about naming your integrations before you proceed.

  • For Jira Cloud integrations, see Jira documentation for making an API Token. You will use this API Token to continue with the integration steps below.
  • For Jira Server integrations, you will use a dedicated username and password for this integration.

Note: This integration supports (meaning that the fields can be automatically set when posting an issue) the following fields: project, issue type, summary, reporter, description, and assignee. If a Jira project is configured to require any additional fields when creating an issue, the integration setup will give you a validation error, and the integration cannot be made.

To configure your new integration, complete the steps below.

  1. Open Infrastructure Monitoring and click Integrations to open the Integrations page. Look for the tile named Jira. You can search for it by name, or find it in the Notification Services section.

  2. Click the Jira tile, then click Create New Integration or New Integration to display the configuration options page. See the screen shot below.

    • For the Jira Base URL, use https://YOUR-DOMAIN.atlassian.net or http://YOUR-HOSTNAME:PORT
    • For Jira Cloud, provide the email and API token.
    • For Jira Server, provide the username and password.
    • For Project, select a project from the list of all the projects that you have access to in Jira, and then click Apply.
    • For Issue Type, select an issue type and then click Apply.
    • To set the Assignee for the tickets created by the Infrastructure Monitoring Notification Service, you can name the Assignee in this integration or in the detector. The detector setting takes precedence over the integration setting, so in case you set two different Assignees, the one from the detector will be used. In that way, you can use a default Assignee (blank) for the integration, and selectively change it for some detectors. If your Jira instance doesn’t require it, you can leave the Assignee setting blank. To learn more about setting the default Assignee in Jira, see Edit a project’s details or the equivalent for your version of the Jira product .

  3. When you are finished filling in the fields, click Save. The name of your new integration displays at the top of the list.

    ../_images/jira-new-int-save.png

Test your integration (optional) đź”—

  1. Return to the Integrations tile, select the integration you just made, and then click Create Test Issue.
../_images/jira-new-int.png

  1. A test Jira ticket is created for the named Assignee. After a short delay, a comment is made on the same ticket that states the alert has cleared. After both of these events have occurred, the status is indicated on the window, as shown below.

    ../_images/jira-test-status.png

Add a Jira notification to a detector đź”—

  1. Create, edit, or subscribe to a detector for which you want alert notifications to be sent through Jira (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector).
  2. Click Add recipient, and select Jira.
  3. From the drop-down list of Jira integrations, select the integration name that may include the assignee where the notification will be sent. You can overwrite the named integration assignee (or a blank if none was assigned) by typing in a new assignee. Currently, only a single Jira recipient per detector is supported.
  4. Click Done. You can then activate the detector you have configured.

Infrastructure Monitoring will now create a Jira issue based on an alert notification whenever the detector rule condition is met. It will also add a comment to that issue when the alert condition clears.

Integrate with Microsoft Teams đź”—

Overview đź”—

Splunk Infrastructure Monitoring can send notifications to a Microsoft Teams channel when an alert is triggered by a detector and when the alert clears.

At a high level, to integrate Infrastructure Monitoring with Microsoft Teams, you must:

  • Add the Infrastructure Monitoring Events connector to your channel in Microsoft Teams
  • Add your channel’s integration into Infrastructure Monitoring
  • Update an existing detector to send alerts

Note that existing Office 365 integrations will appear in Infrastructure Monitoring within the Microsoft Teams tile.

Prerequisites đź”—

To fully integrate Infrastructure Monitoring with Microsoft Teams, make sure that:

  • You are already a member of a Microsoft Teams channel; verify that the channel that will receive the triggered alerts from Infrastructure Monitoring. For your reference, this channel is also called the reporting channel.
  • You must be an administrator of your Infrastructure Monitoring organization to add and modify integrations in the application.

Step 1: Retrieve your channel’s webhook URL 🔗

  1. Log in to Microsoft Teams, and then navigate to the list of teams.
  2. Select the desired team, and then expand the list of corresponding channels.
  3. Locate and hover over the desired reporting channel, click the ellipses (…), and then select Connectors.
  • If you do not see Connectors, then you may not have permission to add a new connector. Contact the owner of the team to update your permissions.
  1. Locate the incoming Events connector, and then click Add or Configure.
  • If this connector has not been added to any channel in your team, then you will see Add.
  • If this connector has been added to a channel in your team, then you will see Configure.

  1. (Optional) Update the default connection name. After this step, you will not be able to change the connector’s name.

    ../_images/ms-name-connection.png

  2. Copy the webhook URL. You will need this information in a later step.

  3. Click Save.

Step 2: Add your channel’s integration into Infrastructure Monitoring 🔗

  1. Open Infrastructure Monitoring, and then click Integrations to open the Integrations page. Locate the Microsoft Teams tile.

  2. Click the Microsoft Teams tile, and then click Create New Integration or New Integration to display the configuration options page.

    ../_images/create-new-ms-integration.png

  3. By default, the name of the integration is Microsoft Teams. This name will appear when you update detector rules for alert notifications. You should change the default name of the integration to be more descriptive, especially if you plan to create more Microsoft Teams integrations. To learn more about naming integrations, see About naming your integrations.

    ../_images/ms-name-signalfx.png

  4. In Webhook URL, paste the URL you previously copied, and then click Save. After a moment, a Validated! message will appear. When you see this validation message, return to the reporting channel in Microsoft Teams to verify that another validation message was sent.

  • If you see an error message, verify that the webhook URL you entered is correct.
  • For additional troubleshooting information see Troubleshooting Microsoft Teams.

Step 3: Update an existing detector to send alerts đź”—

You can use these instructions to add your integration to an existing detector with rules and alerts. To learn more about detectors, including how to create a detector and add alerts, see Set Up Detectors to Trigger Alerts.

  1. In Infrastructure Monitoring, click Alerts > Detectors.
  2. In the table, locate the desired detector, and then click the corresponding ellipses (…).
  3. Click Manage Subscriptions.
  4. In the window that appears, click Add Recipient, and then select Microsoft Teams.
  5. Select the name of the desired integration.

Now, when the conditions for the detector’s rules are met, Infrastructure Monitoring will send a notification to the channel. Similarly, when the alert clears, a notification will also be sent.

Troubleshoot Microsoft Teams đź”—

If you cannot complete the integration or connection process, consider that:

  • You may have not copied or pasted the webhook URL correctly.
  • You may not have properly saved the Infrastructure Monitoring connection in Microsoft Teams:
    • Before you can create an integration in the Infrastructure Monitoring UI, you must first properly create and save the connection in Microsoft Teams.
    • To troubleshoot, return to the Microsoft Team’s configuration page for the Events connector, click Save, and then return to the Infrastructure Monitoring UI to create the integration.

If the previously configured reporting channel is no longer receiving notifications:

  • Verify that the Infrastructure Monitoring connection in Microsoft Teams still exists.
    • To troubleshoot, in the Infrastructure Monitoring UI, navigate to the Microsoft Teams tile, expand the desired integration, click the ellipses, and then select Validate.
    • If you see an error message, specifically Connector configuration not found, then the Events connector was removed from the Microsoft Teams channel and must be re-established. Follow the steps outlined in this document to add a new connection and configure a new integration.
  • Research any possible configuration changes, such as:
    • A different reporting channel has been added to the integration.
    • The integration is no longer associated with the detector.
    • The detector’s alert rules have been updated, which would cause notifications to be sent for different reasons.

Integrate with Opsgenie đź”—

Splunk Infrastructure Monitoring can automatically send notifications to one or more Opsgenie teams when a detector triggers an alert. To do this, you first add an API integration in Opsgenie, then you add an associated Opsgenie integration in Infrastructure Monitoring.

You must be an administrator of your Infrastructure Monitoring organization to add or modify integrations in the application.

On the Opsgenie side, there are two ways to add an integration for Infrastructure Monitoring:

Add an Infrastructure Monitoring integration to multiple Opsgenie teams đź”—

This type of integration (not available for all Opsgenie accounts) can send alert notifications to multiple teams. To get started, you need an Opsgenie API key.

  1. Log into your Opsgenie account and navigate to the Integration list page.

  2. Search for or navigate to the API icon. Hover over it and click Add.

    ../_images/opsgenie-add.png


  3. Enter a name for the integration, then copy the API key displayed on the page; you will need to paste it into Infrastructure Monitoring. Do not make any other changes. Click Save Integration.

  4. Skip to Add an Opsgenie integration in Infrastructure Monitoring to complete the integration.

Add an Infrastructure Monitoring integration to a single Opsgenie team đź”—

This type of integration will send alert notifications to a single Opsgenie team. To get started, you need an Opsgenie API key.

  1. Log into your Opsgenie account and navigate to the Team Dashboard.

  2. Click the name of the Opsgenie team that should receive notifications from SignalFx.

  3. Display the Integrations tab and then click Add Integration.

    ../_images/opsgenie-tab.png


  4. Search for or navigate to the API icon. Hover over it and click Add.

    ../_images/opsgenie-add.png


  5. Enter a name for the integration, then copy the API key displayed on the page; you will need to paste it into Infrastructure Monitoring. Click Save Integration.

  6. Continue to Add an Opsgenie integration in Infrastructure Monitoring to complete the integration.

Add an Opsgenie integration in Infrastructure Monitoring đź”—

  1. Open Infrastructure Monitoring and click Integrations to open the Integrations page. Look for the tile named Opsgenie. You can search for it by name, or find it in the Notification Services section.

  2. Click the Opsgenie tile, then click Create New Integration or New Integration to display the configuration options.

    ../_images/opsgenie-new.png

  3. Specify the name that should appear when you’re configuring a detector rule to send an alert notification. You should use the Opsgenie team or integration name; see this note about naming your integrations.

  4. Paste your Opsgenie API key into the API key field, select your service region from the drop-down list, and click Save.

    ../_images/opsgenie-save.png

You are now ready to configure detectors to send notifications to Opsgenie.

Add an Opsgenie notification to a detector đź”—

Create, edit, or subscribe to a detector that you want to send notifications to Opsgenie (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Click Add recipient, select Opsgenie, then select the integration name that specifies where the notifications should be sent.

If you select an integration associated with a particular team (see Add an Infrastructure Monitoring integration to a single Opsgenie team), notifications will be sent to that team. If you select an integration that can send notifications to multiple teams (see Add an Infrastructure Monitoring integration to multiple Opsgenie teams), you have two options:

  • Select a team to send the notification to a particular team instead of having Opsgenie determine how to handle the notification.
  • Select “No team” to indicate that Opsgenie should handle the notification based on settings you specified for the integration associated with the API key.

Infrastructure Monitoring will now notify the specified Opsgenie team(s) whenever the detector rule conditions are met.

Integrate with PagerDuty đź”—

You can configure Splunk Infrastructure Monitoring to open PagerDuty incidents when a detector triggers an alert automatically. You must be an administrator of your Infrastructure Monitoring organization to add or modify integrations.

See this note about naming your integrations before you proceed.

Create a PagerDutykey đź”—

To integrate with Infrastructure Monitoring, you need a PagerDuty Service API Key. To get a key, follow PagerDuty’s instructions on Creating Services and Adding Integrations. Under Integration Settings in the Integration Type field, open the Select a Tool drop-down list, and select Infrastructure Monitoring.

After you have added the service, you’ll find the Integration Key in the Integration Settings section of the service page in PagerDuty. Copy the Integration key; you will need to paste it into Infrastructure Monitoring.

Add PagerDuty integration đź”—

  1. Open Infrastructure Monitoring and click the Integrations tab to open the Integrations page. Look for the PagerDuty tile. You can search for it by name, or find it in the Notification Services section.
  2. Click the PagerDuty tile, then click Create New Integration or New Integration to display the configuration options.
  3. By default, the name of the integration is PagerDuty. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You should change the name to be more descriptive; doing so is highly recommended if you want to create more than one PagerDuty integration to map to different escalation policies within PagerDuty.
  4. Enter your PagerDuty Integration Key into the Integration Key field, and then click Save.

You are now ready to configure detectors to send notifications to PagerDuty.

Add a PagerDuty notification to a detector đź”—

Create, edit, or subscribe to a detector that you want to have trigger PagerDuty incidents (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Click Add recipient, select PagerDuty, then select the integration name that specifies where to send the notification.

Infrastructure Monitoring will now trigger PagerDuty incidents whenever the detector rule conditions are met. It will also clear the incident when the value goes back to normal.

Integrate with ServiceNow đź”—

You must be an administrator in Splunk Infrastructure Monitoring to create a new ServiceNow integration. Once created, any user of the application can send notifications to a ServiceNow integration.

See this note about naming your integrations before you proceed.

The process of setting up your first integration with ServiceNow requires a ServiceNow admin to create a user in ServiceNow. The instructions in Set up a ServiceNow user show how to do this. Once the ServiceNow user has been created, you can create multiple ServiceNow integrations for that user.

If you know the ServiceNow instance name, username and password for Infrastructure Monitoring in ServiceNow, you can skip to Set up ServiceNow integration in Infrastructure Monitoring.

Set up a ServiceNow user đź”—

  1. Log into your ServiceNow instance, for example, example.service-now.com. If necessary, display the Service Management dashboard.

  2. In the navigation panel on the left, scroll to User Administration and then click on Users.

    ../_images/sn-click-users.png


  3. Click New to create a new user.

    ../_images/sn-click-new.png


  4. Enter a User ID, first name and last name, that reflect that this user is associated with Infrastructure Monitoring . The examples shown below are only suggestions.

  5. Enter a password. Make sure the checkbox that says Active is checked.

  6. Click Submit to create the user.

    Note

    Make a note of the user ID and password, as you will need them when you are adding the integration in Infrastructure Monitoring.

    ../_images/sn-new-user.png


  7. To find your new user, either search for the user ID or do a reverse chronological sort on the Created column. Click the User ID to open the user information window, scroll down and display the Roles tab, then click Edit.

    ../_images/sn-click-edit.png


  8. In the Collection search box, type “web_service_admin”. Select the “web_service_admin” role and then click > to move it the Roles List panel.

    ../_images/sn-collection-01.png


  9. Similarly, search for “itil”. Select the “itil” role and move it to the Roles List panel. Now, click Save. You should now see web_service_admin and itil under the Roles tab for this user (and possibly additional inherited roles).

    ../_images/sn-roles.png


Continue to the next section to complete the Infrastructure Monitoring integration process.

Set up ServiceNow integration in Infrastructure Monitoring đź”—

To install this integration, you must know the ServiceNow instance name, username, and password associated with Infrastructure Monitoring

  1. Open Infrastructure Monitoring and click Integrations to open the Integrations page. Look for the tile named ServiceNow. You can search for it by name, or find it in the Notification Services section.

  2. Click the ServiceNow tile, then click Create New Integration or New Integration to display the configuration options.

    ../_images/sn-sfx01.png


  3. By default, the name of the integration is ServiceNow. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You should change the name to be more descriptive; doing so is highly recommended if you want to create more than one ServiceNow integration.

  4. Enter the ServiceNow username, password, and instance name. Note that the instance name must be in the format of “example.service-now.com”. Do not include a leading http:// or a trailing /.

    • To troubleshoot potential blind server-side request forgeries (SSRF), Infrastructure Monitoring has whitelisted *.service-now.com. As a result, if you enter a domain name that is rejected by Infrastructure Monitoring, you may need to contact Support to update the list of allowed domain names. Additionally, you cannot enter local ServiceNow instances.
    ../_images/sn-save-int.png


  5. Choose whether you want Infrastructure Monitoring notifications to create a ServiceNow Incident (generally recommended) or Problem, then click Save.

You are now ready to configure detectors to send notifications to ServiceNow.

Tip

You may wish to create a second ServiceNow integration with a different issue type, so you can create either an Incident or a Problem depending on the detector rule that is sending the notification. Give each integration a different name. All the rest of the information remains the same.

Configure a detector to report to ServiceNow đź”—

Create, edit, or subscribe to a detector for which you want alert notifications to be sent to a ServiceNow issue (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector). Select the name of your ServiceNow integration as a notfication recipient. Infrastructure Monitoring will now send a notification to ServiceNow whenever the detector rule conditions are met. When the alert clears, Infrastructure Monitoring will send a notification that sets the state as “Resolved” in ServiceNow.

The ServiceNow integration maps alert severity in Infrastructure Monitoring to the Impact and Urgency fields in the corresponding ServiceNow incident as follows:

Infrastructure Monitoring Severity ServiceNow Impact and Urgency
Critical 1
Major or Minor 2
Warning or Info 3

Integrate with Slack đź”—

Integrating Splunk Infrastructure Monitoring with Slack allows you to send alert notifications from detectors to a Slack channel. To add a Slack integration, you must be an administrator of your Infrastructure Monitoring organization and must be authorized to add apps to Slack.

See this note about naming your integrations before you proceed.

  1. Open Infrastructure Monitoring and click Integrations to open the Integrations page. Look for the tile named Slack. You can search for it by name, or find it in the Notification Services section.

  2. Click the Slack tile, and then click Create New Integration or New Integration.

    Note

    If you see an error message after clicking New Integration, you aren’t authorized to add apps to Slack and will not be able to add this integration. Contact your Slack administrator for assistance.

  3. Review the permissions required by Slack, then click Authorize. You will return to the Slack integration screen in Infrastructure Monitoring.

  4. By default, the name of the integration is Slack. This is the name that will appear when you’re configuring a detector rule to send an alert notification. You should change the name to be more descriptive; doing so is highly recommended if you want to create more than one Slack integration.

  5. Click Save.

    ../_images/slack-save.png

You are now ready to configure detectors to send notifications to Slack.

Configure a detector to report to Slack đź”—

  1. Create, edit, or subscribe to a detector for which you want alert notifications to be sent to a Slack channel (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector).

  2. Select Slack as the notification recipient. If there are multiple Slack integrations, select the name of the desired integration.

    ../_images/slack-choose-integration.png

  3. Enter or select the name of the Slack channel to which the notifications should be sent. You can start typing to display a list of matching channel names to choose from.

../_images/slack-channel.png

Infrastructure Monitoring will now send a notification to the channel whenever the detector rule conditions are met. It will also send a notification when the alert clears.

Integrate with Splunk On-Call đź”—

Integrating with Splunk On-Call allows you to send alert notifications from Splunk Infrastructure Monitoring detectors to your Splunk On-Call timeline.

Prerequisites đź”—

You must be an administrator in both Infrastructure Monitoring and Splunk On-Call to create a new Splunk On-Call integration. After a Splunk On-Call integration has been created, any Infrastructure Monitoring user can send notifications.

See this note about naming your integrations before you proceed.

Step 1: Locate and copy the endpoint (URL) in Splunk On-Call đź”—

  1. Open Splunk On-Call, and click Integrations.

  2. In 3rd Party Integrations, locate and select Splunk On-Call.

    ../_images/image-select-signalfx-app.png

  3. If you you do not see a URL, click Enable Integration to generate a URL.

  4. Copy the entire URL, including $routing_key.

    ../_images/copy-victorops-url.png

Step 2: Create a Splunk On-Call integration in Infrastructure Monitoring đź”—

  1. Open Infrastructure Monitoring, and click Integrations to open the Integrations page. Look for the tile named Splunk On-Call. You can search for it by name, or find it in the Notification Services section.

  2. Click the Splunk On-Call tile, then click New Integration or Create New Integration.

  3. By default, the name of the integration is Splunk On-Call. This name will appear when you configure a detector rule to send an alert notification. You should change the name to be more descriptive, especially if you plan to create multiple Splunk On-Call integrations.

    ../_images/victorops-02a.png

  4. In Post URL, paste the endpoint (URL) you copied from Splunk On-Call.

  5. Click Save, and then the “Validated!” message will appear. If you see an error message, verify the endpoint (URL) again. If you still receive an error message, then contact signalfx-support@splunk.com.

You are now ready to configure detectors to send notifications to Splunk On-Call.

Step 3: Add a Splunk On-Call notification to a detector đź”—

  1. Create, edit, or subscribe to a detector for which you want alert notifications to be sent to Splunk On-Call (see Set Up Detectors to Trigger Alerts or Receiving alert notifications from a detector).
  2. Select Splunk On-Call as a notification recipient and specify a routing key. If there are multiple Splunk On-Call integrations, select the name of the desired integration.

Infrastructure Monitoring will now send a notification to Splunk On-Call whenever the detector rule conditions are met and when the alert clears.

(Optional) Review available fields đź”—

When you receive a notification in your Splunk On-Call timeline, note that the notification contains information provided by both Infrastructure Monitoring and Splunk On-Call. Based on your alert type and detection settings, review the following information provided by Infrastructure Monitoring:

Field Description
Detector Definition This field displays a link that you can click to access the detector in your Infrastructure Monitoring account to view the corresponding alert rules.
Graph This field displays a snapshot view of the signal that triggered the alert.
detector This field displays the name of the detector in your Infrastructure Monitoring account.
inputs This field provides detailed information about the alert, including the rule and detector name, alert triggering conditions, and signal details.
rule This field displays the name of the alert rule in SignalFx where the conditions to trigger alert events and clear events were defined.
entity_display_name This field displays the SignalFx rule and detector name. This information also appears in the rule and detector fields within the notification.
state_message When the alert triggers, this field displays the alert’s severity, which includes critical, major, minor, warning, or info. When the alert is resolved, this field displays back to normal, stopped, or manually resolved.
entity_id This field displays the incident’s ID in a string.
monitoring_tool By default, this field displays signalfx.
message_type This field displays the severity of the alert, which includes critical, warning, acknowledgement, info, and recovery.

Integrate with xMatters đź”—

To set up xMatters as an alert notification recipient, follow the instructions on the xMatters website.

Send notifications via a webhook URL đź”—

You can specify a webhook URL to receive notifications when an alert is triggered or cleared. The request will be an application/JSON POST with the following key/values making up the JSON object in the body:

-  detector (string): Name of this detector.
-  detectorUrl (string): URL of the detector, which includes a parameter to select
   this specific incident.
-  detectorId (string): ID of the detector
-  description (string, optional): Description of the detector
-  imageUrl (string): URL of the alert preview image
-  incidentId: Unique identifier for this alert notification.
-  eventType (string): Uniquely identifies the version of the detector that sent the
   notification.
-  rule (string): Name of the detector rule that triggered the alert.
-  severity (string): Severity level of this rule.
-  runbookUrl (string, optional): Runbook URL specified in this rule.
-  tip (string, optional): Tip specified in this rule.
-  messageTitle (string): Notification title for this rule
-  messageBody (string): Notification message for this rule
-  detectOnCondition (string): The trigger metric data and detection criteria in
   this rule, in SignalFlow format.
-  detectOffCondition (string, optional): The clear metric data and detection
   criteria in this rule, in SignalFlow format.
-  status: (kept for backwards compatibility; use statusExtended to receive
   more information)
   (string): The state of this incident, with one of the following values:
      -  "anomalous" -- the alert is firing because the detect conditions are met
      -  "ok" -- the alert was cleared because the detect conditions were no longer
          met or the clear conditions (if any) were met
-  statusExtended: The state of this incident, with one of the following values:
      -  "anomalous" -- the alert is firing because the detect conditions are met
      -  "ok" -- the alert was cleared because the detect conditions were no longer
          met or the clear conditions (if any) were met
      -  "manually resolved" -- a user resolved the alert through the UI or the API
      -  "stopped" -- the detector that triggered the alert was edited or deleted
-  timestamp (string): The time the event occurred, in ISO 8601 format.
-  inputs: The map of the inputs involved in this rule (see "Webhook inputs" below)
-  sf_schema (integer): The schema version for this event (value always = 2)

Tip

Webhooks can be shared across multiple detectors. On the Integrations page, click the Webhooks icon in the Notification services section. Enter a webhook URL, name it, and add custom HTTP headers as well. These webhooks will be recipient targets when you add a recipient to a detector.

Webhook inputs đź”—

As noted above, “inputs” is one of the elements in the JSON object. Each input is named after the program variable it is bound to; if not bound (anonymous input), it will have a name in the form “_S0”, “_S1”, etc.

Each input is an object, containing:

  • a key
    • the key object is a map of the dimensions of the input signal
    • the key object may be empty if there are no dimensions (rare)
    • the key field may not be present if the input was a static value (static thresholds for example / comparisons against scalar values)
  • a value
    • the value of that input at the time of the event (when the alert fires, and when it clears)
  • a fragment
    • the fragment of the SignalFlow program that represents this input
    • may be useful to display in a notification to “explain” what that input is
    • may not be present for some detectors or for static, anonymous inputs

Note about realms

A realm is a self-contained deployment of Splunk Infrastructure Monitoring or APM in which your organization is hosted. Different realms have different API endpoints. For example, the endpoint for sending data in the us1 realm is https://ingest.us1.signalfx.com, while the endpoint for sending data in the eu0 realm is https://ingest.eu0.signalfx.com.

Where you see a placeholder that refers to realms, such as <REALM> or <YOUR_REALM>, replace it with the actual name of your realm. You can find this name on your profile page in the user interface. If you don’t include the realm name when specifying an endpoint, Infrastructure Monitoring defaults to the us0 realm.

Webhook example 1 đź”—

The following example illustrates the parameters for a webhook, with sample values.

{
  "detector": "Memory usage detector",
  "detectorUrl": "https://app.YOUR_SIGNALFX_REALM.signalfx.com/#/detector/ABCDEFGHIJK/edit",
  "description": "A detector which alerts when memory usage exceeeds 90% for 10 minutes",
  "incidentId": "BCDEFGHIJKL",
  "eventType": "foo",
  "rule": "Running out of memory",
  "severity": "Minor",
  "description": "Memory has reached 90% of maximum for 10 minutes",
  "detectOnCondition": "when(A > 90, '10m')",
  "detectOffCondition": "when(A < 90, '15m')",
  "status": "ok",
  "statusExtended": "ok",
  "imageUrl": "https://org.YOUR_SIGNALFX_REALM.signalfx.com/#/chart/abCDefGHij",
  "timestamp": "2016-11-08T19:43:30Z",
  "inputs": {
    "_S1": {
      "dimensions": {
        "host": "i-346235qa",
        "plugin": "signalfx-metadata"
      },
      "value": 96.235234634345,
      "fragment": "data('memory.utilization')"
    }
  },
  "sf_schema": 2
}

Webhook example 2 đź”—

Here’s another, more complex example of an alert emitted by an anomaly detector created using the SignalFx API. In this example, each host is emitting a metric called latency per API endpoint, so each data point will have 3 dimensions: the endpoint, the host, and the data center. The detector is comparing the 99th percentile of the latency of all the APIs of a particular host against the 99th percentile of the latencies of all the hosts in its data center. It alerts if the host latency is greater than the data center latency AND the latter is greater than 40 ms.

{
   "sf_schema": 2,
   "detector": "My detector",
   "detectorUrl": "https://app.YOUR_SIGNALFX_REALM.signalfx.com/#/detector/<id>/edit",
   "incidentId": "<id>",
   "eventType": "<event-type>",
   "rule": "My detector rule",
   "severity": "Critical",
   "description": "Latency of host myserver is 43.4, over datacenter-wide latency of 42.9 !",
   "status": "anomalous",
   "statusExtended": "anomalous",
   "imageUrl": "https://org.YOUR_SIGNALFX_REALM.signalfx.com/#/chart/abCDefGHij",
   "timestamp": "2016-10-25T21:19:38Z",
   "detectOnCondition": "when(a > b and b > 40)",
   "inputs": {
      "a": {
         "key": {
            "host": "myserver",
            "dc": "us-west-1"
         },
         "value": 43.4,
      "fragment": "data('latency').p99(by=['host', 'dc'])"
      },
      "b": {
         "key": {
            "dc": "us-west-1"
         },
         "value": 42.9,
         "fragment": "data('latency').p99(by='dc')"
      },
      "_S2": {
      "value": 40,
      "fragment": "40"
      }
   }
}