Live Cross-Region Data Migration Without Perceived Downtime: A Four-Phase Protocol for Multi-Service Cloud Systems

Source abstract

Relocating live data between geographic regions is difficult when one user-visible entity spans many independently owned services and storage backends. Copying the data is not enough: concurrent writes, deletes, and stale routing caches can lose updates, resurrect deleted data, or split write authority at cutover. This paper presents a four-phase migration protocol, deployed on a large-scale messaging platform, that keeps such migrations imperceptible to users. The protocol fences operations by class rather than by phase, so that only the final delta catch-up blocks writes; carries deletes as content-stripped tombstones, so that a paginated copy cannot resurrect deleted data; enforces locks through routing metadata that every request already reads; and cuts over through a single atomic routing update gated on end-to-end validation. We state a cutover-completeness invariant and four safety propositions with proof sketches. Over 30 days and 6–10 million migrations per day, the write lock lasted 13 s at the median and 19 s at the 99th percentile; 99.9% of migrations succeeded on the first attempt and all succeeded after retry; and fewer than one in 100 million message sends encountered the write lock, each succeeding on client retry. A trace-driven counterfactual analysis shows that blocking writes throughout historical copy would raise the 99.99th-percentile write-blocked window from 159 seconds to at least 873 minutes. Pre-cutover validation detected and rolled back the rare migrations in which a service’s export or import lagged behind its own schema changes.