How to mute the alerts for a particular time in Alert Manager?

Alert Manager is a powerful open-source tool for handling and managing alerts from different systems. You can organize and react to notifications fast and effectively with the aid of Alertmanager. It receives notifications via the channel of your choice, such as email or Slack, and manages silence, inhibition, grouping, and routing.

🔹 One of the key features of Alert Manager is the ability to mute or suppress alerts temporarily. This feature is useful for times when you need to perform maintenance or investigate an issue, and you do not want to receive any alerts for a specific time period.

For Example, I have a few servers that every day stop at night and start in the morning. What should I use if I want to mute the notifications in AlertManager for this?

For the above cases, Muting alerts during specific times can assist reduce alarm fatigue, eliminate unwanted alerts and notifications, and keep work from being disrupted by distracting messages and notifications.

In this blog, we’ll discuss how to mute alerts for a specific time period at the Alert Manager level.
We’ll cover two methods for muting alerts: using the mute_time_intervals and silence feature.

Using mute_time_intervals to Mute Alerts in Alert Manager

The mute_time_intervals feature in Alert Manager is available in the newer version of AlertManager > 0.21.

It allows you to mute alerts during a specific time range. Any alerts that match the defined labels during this time range will be suppressed, and they will not be sent to the configured receivers.

You must make changes to the Alert Manager configuration file in order to utilize mute_time_intervals to mute alerts.

The steps are as follows:

1. Open the Configuration file for AlertManager i.e. alertmanager.yml.
2. Add the mute_time_intervals in the route sections.

route:
  routes:
  - receiver: Email
    match_re:
      <LABEL_NAME>: "<LABEL_VALUE>"
    mute_time_intervals:
    - offhours
    continue: true
  - receiver: Slack
    match_re:
      <LABEL_NAME>: "<LABEL_VALUE>"
    mute_time_intervals:
    - offhours
  - receiver: Pager Duty
    match_re:
      <LABEL_NAME>: "<LABEL_VALUE>"
    mute_time_intervals:
    - offhours
  - receiver: 'your-receiver'
  - match:
      # Specify the labels that should be muted
      alertname: HighErrorRate
      severity: critical

mute_time_intervals:
- name: offhours
  time_intervals:
  - times:
    - start_time: "01:00"
      end_time: "03:00"

In this configurations, replace the <LABEL_NAME>: <LABEL_VALUE> with the values you want to match. We can adjust the time_intervals as per the requirements. In this example, the time range is set to mute alerts from 1 AM to 3 AM every day.

We can also use the location parameter to provide the specific time zone for the time interval.

If no timezone is provided, the default time interval is taken to be in UTC time.

Using the silence Feature to Mute Alerts

There is another straightforward way to mute the alerts in Alertmanager for a specific duration which is using Silences. A silence is configured based on matchers, just like the routing tree.

Please Note, Using this feature, the alert gets muted for specific time period and not for specific time range.

To snooze alerts using the Alert Manager silence feature,

Follow these steps:

1. Open the Alert Manager web interface.

2. Click the “New Silence” button on the top right corner.

3. In the matchers section, specify the labels that match with the alerts you want to silence. For example, you can use the custom-defined labels here to silence the alerts.
In the StartsAt and EndsAt sections, specify the time range during which you want to silence the alerts. You can specify a specific date and time.

4. Optionally, you can add a comment to describe the reason for the silence.

5. Click the “Create” button to create the silence.

All alerts that match the supplied labels will be silenced during the specified time period once you have created the silence. The “Silences” tab of the Alert Manager web interface allows you to view and manage your silences.

Conclusion

By using these features, you can reduce the noise of alerts and focus on the issues that are most critical. I hope this blog post has helped you understand how to mute alerts for a particular time in Alert Manager using the mute_time_intervals and silence feature.

This was my little attempt to share the process of muting alerts at the AlertManager level with you all.

👍 Please like this article if you found it helpful.

And please feel free to share your ideas for improvement with us in the Comment Section.

🤞 Stay tuned for future posts.

Feel free to contact us for any more conversations regarding Cloud Computing, DevOps, etc.

Our Recent Posts

• How to configure AlertManager? Improve observability of your system easily – Beginners guide
• What is Observability, Monitoring & Alerting? How to avoid downtime of the System?

Simran Gera

I’m a DevOps Engineer with 3 years of experience, passionate about building scalable and automated infrastructure. I write about Kubernetes, cloud automation, cost optimization, and DevOps tooling, aiming to simplify complex concepts with real-world insights. Outside of work, I enjoy exploring new DevOps tools, reading tech blogs, and play badminton.