Vacation Mode: Monitoring and Alerting with Reduced Staffing

Summer hits the IT department of a mid-sized company in the same pattern every year: by the end of July, four out of seven colleagues are on holiday, but manufacturing keeps running, invoicing keeps running and the backup window hasn’t moved. Anyone who wants to protect the remaining on-call rotation without risking critical outages will not get around a temporary re-evaluation of their monitoring and alerting rules. That is exactly the focus of this article.

In the following sections we describe a pragmatic approach that has proven itself in our customer projects: severity tuning, on-call rotation via Grafana OnCall or PagerDuty, automatic ticketing to a defined primary contact, escalation chains with clear timeouts and a deliberate prioritisation of what has to run during the vacation weeks and what can be deferred.

Severity tuning: suppressing alert noise for four weeks

The most important lever is the temporary re-evaluation of alert severities. A full log volume alert at 2 a.m. is a “warning” in normal operation — during vacation with only one person on call it must either be lowered to “info” or given a longer tolerance window so it does not make the phone ring.

We work with a dedicated Alertmanager routing branch that is activated via a label such as season: vacation. Example of a Prometheus alert that is grouped differently during vacation mode:

groups:
- name: storage.rules
  rules:
  - alert: TrueNASPoolCapacityWarning
    expr: zfs_pool_capacity_ratio > 0.85
    for: 30m
    labels:
      severity: warning
      season: vacation-defer
    annotations:
      summary: "Pool {{ $labels.pool }} above 85 percent"
      runbook: "https://wiki.example.local/runbooks/zfs-capacity"

  - alert: TrueNASPoolCapacityCritical
    expr: zfs_pool_capacity_ratio > 0.92
    for: 10m
    labels:
      severity: critical
      season: vacation-page
    annotations:
      summary: "Pool {{ $labels.pool }} above 92 percent -- action required"

The label season: vacation-defer lands in an Alertmanager receiver that only opens a ticket in the helpdesk and does not trigger a push notification. vacation-page, on the other hand, goes directly into the on-call rotation. It is important that this switch is documented and time-boxed — a simple calendar entry with the switch-back date prevents you from ending up with a numbed monitoring landscape in September.

On-call rotation: Grafana OnCall or PagerDuty in the mid-market

For the actual on-call duty, we increasingly use Grafana OnCall (stably integrated in Grafana 11 since version 1.9) or PagerDuty for small teams. Both support time-shifted rotations, handover times and escalation levels. For an SMB team of three to five people, a weekly rotation with a clear Monday-morning handover is usually enough.

A proven schedule for four vacation weeks:

Week	Primary On-Call	Secondary	Escalation after 15 min
WK 31	Employee A	External partner	Management
WK 32	Employee B	External partner	Management
WK 33	External partner	Employee A	Management
WK 34	Employee B	External partner	Management

The external partner — in many of our projects that is DATAZONE itself — either takes an entire week or covers the secondary role. Important: the handover must contain a technical handover (open tickets, ongoing changes, planned maintenance), not just the calendar switch. We document this in a one-page handover sheet that is pinned in the team channel every Monday at 9 a.m.

Automatic ticketing to the primary contact

Alerts of severity “warning” should not get anyone out of bed during vacation, but must not be lost either. We therefore route them via Alertmanager directly into the ticket system — in current projects mostly Zammad 6 or OTRS/Znuny — and assign them to a generic “vacation queue” whose only handler is the current on-call primary contact.

The effect is twofold: first, the on-call engineer sees in the morning exactly what happened during the night without every single incident having forced a manual ticket. Second, traceability is preserved — especially for capacity-critical systems such as TrueNAS pools or OPNsense firewalls, the temporal sequence of warnings before an actual outage is the decisive diagnostic foundation.

A simple Alertmanager routing for this:

route:
  receiver: default-oncall
  routes:
  - match:
      season: vacation-defer
    receiver: ticket-only
    group_wait: 2m
    group_interval: 15m
    repeat_interval: 12h

receivers:
- name: ticket-only
  webhook_configs:
  - url: "https://zammad.example.local/api/v1/alertmanager/webhook"
    send_resolved: true

Escalation chains: who gets woken up when

An escalation chain without timeouts is not a chain, it’s a wish list. In Grafana OnCall we typically configure three stages: primary on-call gets push and SMS, after 5 minutes without acknowledge a phone call follows, after another 10 minutes the secondary is alerted, after another 15 minutes management or the on-call manager at the external partner.

These numbers are not dogma. In production-critical environments (ERP, MES, warehouse management) we shorten the first stage to 2 minutes. In environments with high redundancy — such as clustered Proxmox hosts with working HA — we deliberately extend it because the cluster initially compensates for the outage itself and only the post-processing is important.

The regular drill is essential: we recommend at least one dry test of the complete chain per quarter, in which a synthetic test alert runs through all stages. Nothing is more frustrating than finding out during a real night shift that the SMS route has not worked since the last provider change.

What stays up during vacation — and what can wait

Beyond alerting, this is about deliberate prioritisation of ongoing work. We divide the IT tasks for vacation weeks into three categories:

• Must run: Backup jobs, replication tasks, antivirus definitions, certificate renewals with expiry in the next six weeks, security patches of the “critical” category according to vendor CVE score.
• Can wait: feature upgrades of operating systems, service migrations, larger configuration rebuilds, non-critical firmware updates.
• Actively deferred: large maintenance windows with potential reboot need for central services, storage expansions, network rebuilds. We deliberately schedule these topics for calendar weeks 36 or 37, when the team is complete again.

An additional tip from our projects: certificates expiring during the vacation weeks are renewed two weeks in advance, not “just before expiry”. Auto-renewal via ACME/Let’s Encrypt for internal services — via caddy, traefik or directly in OPNsense — significantly reduces this concern.

Conclusion

A cleanly prepared vacation operation is not an emergency programme, but good operational management. The three core elements — temporary severity tuning, clear on-call rotation with escalation chain and deliberate prioritisation of work packages — can be set up in a few days and bring the whole team noticeable calm. The rollback is essential: on the first working day after vacation, the Alertmanager labels are reset, the handover sheet is archived and the deferred maintenance is rescheduled.

DATAZONE supports you in setting up your monitoring and alerting landscape — from Prometheus/Grafana through Zammad integration to temporary take-over of on-call slots by our team. Talk to us if you want to get through summer 2026 in a plannable way: get in touch.