TrueNAS Alert System: Configure Notifications and Avoid Alert Fatigue

A storage system that stays silent during problems is just as dangerous as one that triggers alarms for every minor event. With the overhauled alert system in TrueNAS 26, iXsystems has fundamentally modernized the notification logic. New alert categories, machine-readable identifiers, and context-rich information make it significantly easier to notify the right people at the right time about the right problems.

Why the Old Alert System Falls Short

In earlier TrueNAS versions, the alert system was relatively simplistic: a flat list of warnings that differed in severity but offered little granular filtering. Administrators faced a dilemma — either enable all alerts and drown in a flood of notifications, or silence liberally and miss critical warnings.

TrueNAS 26 solves this problem through a structured alert architecture:

• Alert categories group warnings by subsystem (Storage, Network, Hardware, Services)
• Alert identifiers enable machine-readable mapping for automation
• Context information delivers details directly within the warning (affected pool, disk serial, temperature value)
• Alert rules allow fine-grained control over which alerts go where

Alert Categories Overview

TrueNAS 26 divides alerts into clearly defined categories:

Category	Examples	Typical Severity
Storage	Pool degraded, Disk SMART failure, Scrub errors	Critical / Warning
Network	Link down, LACP degraded, DNS unreachable	Warning / Notice
Hardware	CPU temperature, PSU failure, Fan speed	Critical / Warning
Services	SMB stopped, NFS unresponsive, iSCSI target offline	Warning
System	Update available, Certificate expiring, Boot pool	Notice / Info
Replication	Replication failed, Snapshot hold expired	Warning

Each category has its own severity levels: Critical, Warning, Notice, and Info. This hierarchy forms the foundation for alert rules.

Alert Identifiers and Context Information

Every alert in TrueNAS 26 carries a unique identifier, e.g., STORAGE_POOL_DEGRADED or HARDWARE_DISK_SMART_FAIL. These identifiers are stable across versions and suitable for integration into external monitoring systems.

Additionally, each alert provides structured context information:

{
  "id": "STORAGE_POOL_DEGRADED",
  "severity": "CRITICAL",
  "category": "Storage",
  "context": {
    "pool": "tank",
    "vdev": "mirror-0",
    "disk": "da3",
    "serial": "WDC-WD40EFRX-68WT0N0-WD-WCC4E1234567",
    "state": "DEGRADED",
    "healthy_disks": 1,
    "total_disks": 2
  },
  "timestamp": "2026-04-28T08:15:32Z"
}

This structured data enables automation — a script can automatically react based on the identifier and context data, such as triggering a replacement disk ordering process.

Configuring Alert Rules

Alert rules are the centerpiece of the new architecture. They define which alerts, under which conditions, are sent to which target.

Alert Rule Structure

In the TrueNAS web interface under Alerts > Alert Rules, create rules with the following parameters:

• Name: Descriptive rule name (e.g., “Critical Storage to On-Call”)
• Categories: One or more alert categories
• Severity: Minimum severity (e.g., Warning and above only)
• Targets: One or more notification targets
• Schedule: Optional time windows (e.g., outside business hours only)

Example: Multi-Tier Alert Strategy

A proven configuration for SMB environments:

Rule 1 — Critical alerts immediately:

• Categories: Storage, Hardware
• Severity: Critical
• Targets: PagerDuty (On-Call), Email to admin team
• Schedule: 24/7

Rule 2 — Warnings during business hours:

• Categories: All
• Severity: Warning
• Targets: Slack channel #infra-warnings
• Schedule: Mon–Fri, 08:00–18:00

Rule 3 — Daily summary:

• Categories: All
• Severity: Notice, Info
• Targets: Email digest to team lead
• Schedule: Daily at 08:00

Setting Up Email Integration

Email notification is the most established alert channel. In TrueNAS 26, configure SMTP delivery under System > General > Email:

SMTP Server:    mail.example.com
SMTP Port:      587
Security:       STARTTLS
Username:       alerts@example.com
Password:       ********
From Address:   truenas-alerts@example.com

Always test the configuration with the Send Test Email function. A common mistake: the SMTP server requires an authenticated sender address that matches the username.

Best Practice: Dedicated Alert Mailbox

Set up a dedicated mailbox for TrueNAS alerts and create filter rules there. This cleanly separates storage alerts from other email traffic and allows custom categorization within the mailbox.

Slack Integration

For teams using Slack as their communication platform, TrueNAS 26 offers native webhook integration:

1. In Slack, create a new webhook under Apps > Incoming Webhooks
2. Select the target channel (e.g., #truenas-alerts)
3. Copy the webhook URL
4. In TrueNAS: Alerts > Alert Targets > Add > Slack
5. Paste the webhook URL and set a descriptive name

Slack alerts automatically include formatted messages with severity colors:

• Red for Critical
• Orange for Warning
• Blue for Notice
• Gray for Info

PagerDuty Integration

For environments with genuine on-call operations, PagerDuty is the right choice. TrueNAS 26 supports the PagerDuty Events API v2:

1. In PagerDuty, create a new service with Events API v2 integration
2. Note the Integration Key (32-character hex string)
3. In TrueNAS: Alerts > Alert Targets > Add > PagerDuty
4. Enter the Integration Key

PagerDuty offers decisive advantages over simple notifications:

• Escalation policies: If an alert is not acknowledged within X minutes, the next person is notified
• On-call rotation: The on-duty admin is automatically notified
• Incident grouping: Multiple related alerts are merged into a single incident
• Auto-resolve: When TrueNAS clears the alert, the incident is automatically closed

Avoiding Alert Fatigue

Alert fatigue — desensitization to warnings — is one of the biggest risks in IT operations. When the team has learned to ignore notifications, the next critical alert is just background noise.

Strategies Against Alert Fatigue

1. Use severity consistently Not every alert needs to trigger an immediate notification. Use the severity levels:

• Critical: Immediate action required (pager)
• Warning: Action within hours (Slack/Email)
• Notice: Informational, check when convenient (digest)
• Info: Dashboard visibility only

2. Suppress known states When replacing a disk and the pool is temporarily degraded, suppress the alert temporarily:

# Temporarily silence alert (via midclt)
midclt call alert.dismiss "STORAGE_POOL_DEGRADED" '{"pool": "tank"}'

3. Adjust thresholds Default thresholds do not fit every environment. A NAS in a climate-controlled data center needs different temperature limits than one in a server closet:

• CPU temperature: Default 80 C — adjust to 75 C or 85 C as needed
• Pool utilization: Default 80% — for large pools, consider raising to 85%
• SMART warnings: Reallocated Sectors > 0 is the default alert; for older disks with a few sectors, consider setting to > 5

4. Leverage aggregation Instead of sending separate alerts for every single disk in a pool, TrueNAS 26 automatically groups related alerts. Use this feature consistently and configure aggregation windows:

Aggregation Window:    5 minutes
Max Alerts per Digest: 10
Repeat Interval:       4 hours

5. Regular alert review Schedule a monthly review session: Which alerts came most frequently? Which were never addressed? Which are missing? Adjust rules and thresholds accordingly.

Integration with External Monitoring Systems

TrueNAS alerts can be integrated into existing monitoring infrastructure through various methods:

SNMP Traps

Enable the SNMP service in TrueNAS and configure trap receivers:

Services > SNMP > SNMP Traps
Trap Host:      monitoring.example.com
Community:      truenas-traps
Port:           162

API Polling

The TrueNAS API provides an endpoint for active alerts:

# Query active alerts via the API
curl -s -X GET "https://truenas.example.com/api/v2.0/alert/list" \
  -H "Authorization: Bearer $API_KEY" | jq '.[] | {id, level, formatted}'

Syslog Forwarding

For SIEM integration, forward alerts to a central syslog server:

System > General > Syslog
Remote Syslog Server:  syslog.example.com:514
Transport:             TLS
Level:                 Warning

Practical Example: Alert Configuration for a Mid-Size Business

A typical configuration for a company with 50 employees, two TrueNAS systems, and a three-person IT team:

1. PagerDuty for Critical alerts (pool degraded, disk failure, hardware failure) — routes to the on-call admin
2. Slack #storage-alerts for Warning alerts — the entire IT team sees them
3. Daily email summary for Notice and Info — goes to the IT manager
4. SNMP integration into existing monitoring (Zabbix/Checkmk) — for historical data and trend analysis

Conclusion

The overhauled alert system in TrueNAS 26 provides the tools to precisely control notifications. The key to success lies not in the technology, but in the discipline: define clear severity rules, use different channels for different urgencies, and conduct regular reviews. This ensures your team takes the 3 AM critical alert seriously — because it does not get lost in the noise.