Ceph Recovery: 3 Typical Mistakes and How to Avoid Them

Ceph is one of the most robust storage systems we operate in customer environments. That robustness is also its trap: because the cluster silently absorbs almost every fault, the remaining failure scenarios feel especially threatening once they do occur. In most recovery situations we see in the field, the incident was not escalated by the original defect, but by the reaction to it.

This article describes the three mistakes we encounter most often in emergencies and shows what a structured process looks like instead. It is aimed at administrators who run a production Ceph cluster, ideally in combination with Proxmox VE or as a standalone storage backend.

Why recovery mistakes keep following the same three patterns

Ceph is a self-healing system. When an OSD fails, the cluster automatically redistributes the affected placement groups (PGs) and restores the configured replication level. The catch: this process is resource-intensive and takes hours to days depending on data volume, network and OSD count. During this window the cluster is degraded, but still fully operational.

That window is exactly when the classic mistakes happen:

1. An administrator restarts OSDs because “something feels off”.
2. A PG stalls in active+recovery_wait+degraded and gets set to active+clean manually, or ceph pg force-recovery gets fired off without understanding the cause.
3. A faulty OSD is removed with ceph osd rm before the data has been cleanly redistributed.

Each of these three moves can turn a tolerable incident into data loss. The good news: all three are avoidable if you know the official recovery flow and give the cluster the time it needs.

Mistake 1: Restarting OSDs during backfill

The most common reflex during a running recovery is that an administrator “preventively” restarts individual OSDs because they show high I/O, respond slowly, or appear as particularly active in ceph -s. That is exactly the wrong move during the backfill phase.

When Ceph redistributes data after an OSD failure, the OSDs under heavy load are precisely those that hold or are about to receive copies of the affected PGs. Restarting these OSDs does three things at once:

• The running backfill operation for all PGs on this OSD is aborted and partly has to restart from scratch.
• The cluster briefly marks the OSD as down, which triggers new peering rounds on all affected PGs.
• Other OSDs pick up the additional load, which can cascade.

What actually helps: Before intervening, check how far the recovery process has progressed and whether there really is a problem.

# Overall cluster status
ceph -s

# Recovery operation progress
ceph progress

# Active backfill/recovery per PG
ceph pg dump pgs_brief | grep -E "backfill|recover"

# Per-OSD I/O load
ceph osd perf

If high latencies are visible, the right lever is usually not a restart but throttling the recovery bandwidth:

ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 1
ceph config set osd osd_recovery_op_priority 1

These values reduce parallel load and extend the recovery, but keep client I/O stable.

Mistake 2: Setting PG states manually without ceph-mgr

The second typical mistake concerns stuck PGs. When a PG lingers in states such as active+recovery_wait+degraded, incomplete or stale, some administrators reach for ceph pg force-recovery, ceph pg force-backfill or, in extreme cases, ceph pg mark_unfound_lost. These commands are powerful and are legitimately needed in emergencies, but without a clean analysis they cause more damage than good.

The core issue: the Ceph manager (ceph-mgr) coordinates recovery priorities, progress reporting, and the interaction between mons and OSDs. Overriding PG states manually or firing force commands while mgr is not running cleanly bypasses this coordination and produces contradictory internal states.

The correct flow looks like this:

Step	Command	Purpose
1	ceph mgr stat	Is an active manager present?
2	ceph health detail	Which PGs are affected, in which state?
3	ceph pg <pgid> query	Detailed analysis of a single PG
4	ceph osd tree	Which OSDs are up/in, which are not?
5	Fix root cause	Network, disk, auth, clock skew
6	Only then force commands	Only after support/docs confirmation

ceph pg <pgid> query returns, among other things, the recovery_state, which reveals whether a PG is waiting on a specific OSD, whether objects are marked unfound, or whether the PG is stuck due to a peering error. Without that information, every action is a blind flight.

A frequent misconception: unfound does not mean “lost”. It means Ceph currently cannot locate the object on any reachable OSD. In many cases the object reappears as soon as a temporarily failed OSD comes back. mark_unfound_lost should always be the last step, because it permanently removes data from the PG.

Mistake 3: Removing OSDs without a drain

The third pattern: an OSD is acting up, shows SMART errors, or is flagged as a wobbly candidate, and the administrator removes it with a quick sequence of ceph osd out and ceph osd rm without letting the cluster drain cleanly first. If a second OSD fails at the same time or a previous recovery has not finished, this can push redundancy below the configured minimum (min_size).

The correct flow for a planned OSD removal:

# 1. Take the OSD out of the CRUSH data flow, but keep it in the cluster
ceph osd out <osd-id>

# 2. Wait until all PGs are cleanly redistributed
watch -n 5 'ceph -s'
# Target: HEALTH_OK, no PGs in backfill/recovery

# 3. Stop the OSD service (node-dependent)
systemctl stop ceph-osd@<osd-id>

# 4. Mark the OSD as down (safety step)
ceph osd down <osd-id>

# 5. Remove from CRUSH map and auth
ceph osd crush remove osd.<osd-id>
ceph auth del osd.<osd-id>
ceph osd rm <osd-id>

The decisive step is number 2: wait. Redistribution can take many hours on large disks and busy networks. Skipping this step actively removes redundancy from the system and risks immediate data loss on the next failure.

For an OSD that has already failed (down/out, unreachable), the drain step is not needed because Ceph is already redistributing the data. But the same rule applies: wait until the cluster reports HEALTH_OK again before performing the remaining administrative steps.

Recovery monitoring: the commands that really help

During a running recovery, situational awareness matters more than activism. These commands give you a solid read on whether the cluster is making progress:

# Live status overview with progress bar
ceph -s
ceph progress

# Per-PG progress
ceph pg dump pgs_brief

# Recovery rate in objects and MB/s
ceph -s | grep -E "recovery|backfill"

# Where is it stuck?
ceph health detail

# History of state changes
ceph osd tree
ceph osd df tree

A good indicator for “it is working” is a continuously decreasing number of degraded objects and misplaced objects in ceph -s. If that number stagnates completely for 15 to 30 minutes, it is worth a deep dive via ceph health detail and ceph pg <pgid> query.

For long-term observation we recommend the built-in Ceph dashboard and integration into an existing monitoring stack (Prometheus/Grafana via the Ceph exporter module). During an acute recovery, though, the CLI is faster and more reliable than any GUI.

When to call for help

There are situations where picking up the phone is the right next step. From our experience, those typically include:

• PGs in state incomplete or down (possible peering problem, potentially data-relevant)
• unfound objects that do not return after a longer wait
• Cluster in HEALTH_ERR with an unclear cause
• Recovery that makes no progress over days
• Suspected data loss from combined failure of multiple OSDs
• Ceph inside a Proxmox environment with production VMs whose downtime is costlier than the support call

For these scenarios we offer Ceph recovery support as part of our Proxmox and virtualization consulting as well as our Linux system administration.

Conclusion

Ceph forgives many mistakes, but not all of them. The three most common recovery traps can be avoided if you give the cluster time to do its job: do not restart OSDs mid-backfill, do not override PG states without analysis, do not remove OSDs without a clean drain. In most cases the right action is to wait, observe and throttle, not to intervene.

DATAZONE supports you in the design, operation and emergency recovery of your Ceph cluster, whether under Proxmox VE, as a standalone storage backend, or in combination with TrueNAS. If you are in an acute recovery situation or want to harden your cluster preventively, get in touch with us.