How to suppress alerts in Prometheus for non-working hours
24.01.2024
When it comes to the topic how Prometheus and Alertmanager work together and how to restrict alert notification to fire only on working hours, typicall solution is to apply workaround to avaluated alert rule expession like this
and ON() (hour() < 19 and hour() > 8) and ON() (day_of_week() > 0 and day_of_week() < 6)
The full example of such rule is the following
- alert: DummyWorkhoursAlert
expr: 1 and ON() (hour() < 19 and hour() > 8) and ON() (day_of_week() > 0 and day_of_week() < 6)
for: 5m
annotations:
identifier: "{{ $labels.exporter }}/{{ $labels.queue }}"
summary: "Buyerportal: Dummy alert for working hours {{ $labels.queue }}"
description: "* don't do anything, dummy alert *"But the proper solution is to apply mute_time_intervals, as per example below
time_intervals:
- name: offhours
time_intervals:
# Monday to Friday 18:00-8:00
- weekdays: ['monday:friday']
times:
- start_time: '16:00'
end_time: '23:59'
- start_time: '00:00'
end_time: '08:00'
location: "Europe/Berlin"
- weekdays: ['saturday', 'sunday']
location: "Europe/Berlin"
route:
receiver: slack-monitoring
group_interval: 5m
repeat_interval: 24h
routes:
- matchers:
- team = buyers
- severity = 24x7
receiver: slack-team-buyers
- matchers:
- team = buyers
- severity = workhours
mute_time_intervals:
- offhours
receiver: slack-team-buyersAnd the rule starts having extra label "severity": workhours
- alert: DummyWorkhoursAlert
expr: 1
for: 5m
annotations:
identifier: "{{ $labels.exporter }}/{{ $labels.queue }}"
summary: "Buyerportal: Dummy alert for working hours {{ $labels.queue }}"
description: "* don't do anything, dummy alert *"
labels:
severity: workhours
team: buyers