Docs » Detectors and Alerts » Mute Notifications

Mute Notifications 🔗

You can stop sending, or mute, notifications based on specified properties. Notifications are muted based on properties specified in muting rules.

For example, you might want to mute certain notifications if you want to schedule downtime for maintenance on a server or set of servers, or if you are testing new code or settings, and want to test and tweak detector settings before pushing the changes (and associated notifications) live. You can mute notifications for a specified period of time or indefinitely.

If a notification is muted or scheduled to be muted, an indicator is displayed wherever the detector or alert is displayed, such as on the Active Alerts tab or an event feed (see Viewing muting information).

Important

It can take up to a minute before a new muting rule goes into effect.

A maximum of 9,500 muting rules are allowed.

What happens during a muting period 🔗

  • When a muting rule is active, alerts and events that are muted by the rule are still generated and appear in their respective locations in SignalFx.
  • If an alert was active before the muting period starts, and the alert clears during the muting period, a clear notification is sent immediately (during the muting period).
  • No other notifications are sent during the muting period. For example, if an alert is triggered and cleared during the muting period, no notifications are sent.

What happens when a muting period ends 🔗

If the muting rule specifies that notifications should be sent when the muting period has ended (see Configuring a muting rule’s conditions and schedule), notifications will be sent at the end of the muting period for alerts that meet the following conditions:

  • The alert was triggered within the last 90 days of the muting period, and
  • The alert was still active when the muting period ended.

No other notifications are sent for alerts that are still active when the muting period ends. For example, if an alert was active before the muting period started, or was triggered during the muting period but more than 90 days before the end of the muting period, no notifications will be sent when the muting period ends.

The following illustration provides a graphic representation of how muting rules affect notifications during and after a muting period. Solid envelopes represent alert notifications and outlined envelopes represent clear notifications. The illustration reflects the following assumptions:

  • The muting rule specifies that notifications should be sent at the end of the muting period, and
  • The alert indicated by the green envelope was triggered within the last 90 days of the muting period.
../_images/muting-period.png

Creating muting rules 🔗

You can create muting rules to mute a specific detector or alert, a property specified in a group-by option, or by manually specifying muting conditions.

Because it can take up to a minute for a muting rule to go into effect, recipients may briefly continue to receive notifications for muted alerts.

Create muting rules based on detectors or active alerts 🔗

  1. Open the Alerts page.

    • To mute all notifications from a detector, find the detector on the Detectors tab.
    • To mute all notifications from an alert, find the alert on the Active Alerts or Detectors tab.
  2. From the item’s Actions menu at far right, select Create muting rule.

    In the illustration below, all notifications from the “jvm.cpu.load high” detector will be muted.

    ../_images/create-rule.png

    Tip

    You can also mute an open detector; select Mute from the detector’s Actions menu.

  3. Skip to Configuring a muting rule’s conditions and schedule.

Create muting rules based on group-by properties 🔗

  1. Open the Alerts page, then display the Active Alerts or Detectors tab.

  2. Specify the group‑by dimensions; you can create muting rules based on a single dimension or by two dimensions. If you are creating a rule from a group‑by, notifications from multiple detectors might be muted.

  3. From an item’s Actions menu at far right, select Create muting rule.

    In the illustration below, all notifications from aws_availability_zone us-east-1a in aws_region us-east-1 will be muted. (If you wanted to mute all notifications from aws_region us-east-1, you would select Create muting rule from the us-east-1 Actions menu.)

    ../_images/mute-group-by-actions-menu.png


  4. Skip to Configuring a muting rule’s conditions and schedule.

Create muting rules from scratch 🔗

  1. To manually specify properties for which notifications should be muted, open the Alerts page, go to the Muting Rules tab, and click the New Muting Rule button. Notifications from multiple detectors might be muted.

    ../_images/muting-tab.png
  2. Add one or more properties for which you want to mute notifications. You can also type sf_tags to find a list of tags.

    ../_images/muting-add-property.png


  3. Continue to Configuring a muting rule’s conditions and schedule.

Configuring a muting rule’s conditions and schedule 🔗

After you have specified conditions for which notifications should be muted, finish setting options for the muting rule.

../_images/muting-set-options.png
  1. If you wish, you can edit or delete any of the pre-filled properties, as well as add new properties. (If the rule is based on a detector or alert, you cannot edit or remove the detector specification from the rule.)

  2. Specify the time period during which notifications should be muted. In this example, we are creating a muting rule from scratch, and we are going to mute notifications from an AWS availability zone for the next hour.

    ../_images/muting-choose-time.png

    Instead of choosing one of the schedule options on the dropdown menu, you could:

    • Select Custom to specify a start time up to 1 year in the future, such as a maintenance window scheduled for an upcoming weekend.
    • Click Mute Indefinitely to mute notifications until you manually cancel the muting rule to resume notifications.
  3. Explain why you are muting the notifications. The text you enter here is displayed when you hover over a rule in the Muting Rules tab, and can help others understand why alerts are being muted.

    ../_images/muting-describe.png


  4. Specify whether you want to clear any existing alerts that match the conditions you have set.

    If you are muting certain alerts to address a known problem, you may want to clear existing alerts so you are starting from a clean slate. Clearing these alerts also notifies downstream systems, such as VictorOps, OpsGenie, and PagerDuty.

    Alerts are cleared when you save the rule, not when the muting period starts. If you clear alerts for a muting rule that starts in the future, new alerts may come in after you save the rule. These alerts will not be cleared when the muting rule goes into effect.

  5. Specify whether you want to receive notifications for alerts that are still active when the muting period ends. Notifications will be sent only for alerts that meet the following conditions:

    • The alert was triggered within the last 90 days of the muting period, and
    • The alert is still active when the muting period ends.
  6. Click Next to view a summary of the muting conditions. Then click Back if you want to make any changes, or Save if you are ready to add the muting rule.

    It can take up to a minute before a new muting rule goes into effect.

Viewing muting information 🔗

Active and scheduled muting rules 🔗

If you open a detector to which active or scheduled muting rules apply, a View muting rules message is displayed. Click the message to view and edit associated muting rules.

More generally, you can view information about active and scheduled muting rules from different places on the Alerts page.

  • On the Muting Rules tab, you can view a list of all active and scheduled muting rules. In the illustration below, two muting rules have been specified. The first one is currently active and will be in effect indefinitely, as shown by start time in red and no end time. It is also associated with a specific detector; you can click the detector name to open the detector.

    The second rule is scheduled for a time in the future, indicated by its dates being black instead of red. This rule is based only on the specified property and is not associated with a specific detector. The text that was entered in the Reason field for the rule is displayed on hover.

../_images/view-muting-rules.png
  • On the Detectors tab, running or scheduled muting rules related to a group‑by condition or detector are indicated by icons next to the item. On the Active Alerts tab, running or scheduled muting rules related to a group‑by condition are indicated by the icon. You can click the icon to view, edit, delete, or cancel the muting rules.

    A red Muted icon indicates that notifications for that item are currently muted. Note that items displaying this icon might have scheduled muting rules in addition to currently active rules.

    ../_images/active-muted-detector.png

    A grey Muted icon indicates that notifications for that item are scheduled to be muted.

    ../_images/scheduled-muted-detector.png

Notifications that were muted 🔗

  • If a notification was muted, an indicator will be displayed wherever the event would have sent the notification, such as on the Active Alerts tab (shown below) or in an event feed.

    ../_images/muted-alert.png
  • To see events related to inactive (past) muting rules, you can use the Events sidebar or the Event overlay. Events are generated when the rule becomes active (notifications stop) and when the rule becomes inactive (notifications resume).

    • To find events related to muting in the Events sidebar, search for sf_eventType:alertMuting.

      ../_images/find-muting-rules.png
    • To overlay events related to muting on a dashboard, search for alertMuting in the Event Overlay search box.

      ../_images/view-muting-overlay.png

Cancelling active muting rules 🔗

You can cancel muting (resume notifications) for a rule that is currently active from different places on the Alerts page.

  • On the Muting Rules tab, select Resume notifications from the Actions menu of an active rule (start and end times are displayed in red).
../_images/resume-notifications.png
  • On the Active Alerts or Detectors tab, click the red Muted icon for the item for which notifications should resume, then select Resume notifications.
../_images/click-to-resume.png

In each case, you can confirm that you want to resume sending notifications.

Editing or deleting muting rules 🔗

You can edit or delete a scheduled muting rule from different places on the Alerts page. (You can’t edit or delete currently running rules, but you can cancel them to resume notifications.)

  • On the Muting Rules tab, select Edit or Delete from the Actions menu of a scheduled rule (start and end times are not displayed in red).
../_images/edit-delete-muting.png
  • On the Active Alerts or Detectors tab, click the grey Muted icon for the item whose schedule you want to edit or delete. If there are multiple rules, click the rule you want to edit or delete.
../_images/click-to-edit.png

Note

As discussed in Cancelling active muting rules, a red Muted icon indicates a currently running rule. However, an item displaying this icon might have scheduled rules as well. To be sure you are editing or deleting all the rules you want to, view all muting rules on the Muting Rules tab.

If you are deleting the rule, you’ll have a chance to confirm the deletion request.