Distributed training of LLMs in bandwidth-constrained settings is an evolving frontier. Covenant 72B [1] recently demonstrated that performant LLMs can be trained over internet-grade links, however, the low-bandwidth connections were confined to the outermost data-parallel dimension. Unlocking true scale and robustness to hardware topology requires handling low-bandwidth communication between pipeline stages as well. Our new platform Crucible composes SparseLoco with subspace network compression at model pipeline boundaries, enabling model replicas themselves to be distributed across clusters. This advancement also introduces new failure modes. In this blog post we detail some of the ways we have been exploring to ensure our training runs are robust to hardware faults within pipeline stages.
Fault tolerance in SparseLoco
Our implementation performs outermost data-parallel aggregation via all-gather through bucket storage [2] rather than a collective all-reduce, and this topology is naturally fault tolerant: each peer independently downloads and aggregates whatever pseudo-gradients have arrived without the need to rebuild the communication collective. A missing contribution from a failed stage can simply be ignored, with the remaining peers signaled to gather only successful uploads, while training proceeds uninterrupted for the surviving participants. This framework is only possible because of SparseLoCo's aggressive compression.
Figure 1. SparseLoCo [3] robustness at the communication round. A missing contribution is omitted from the agreed set; the available updates remain accessible through object storage
Fault tolerance in low-bandwidth pipelining
Crucible composes a SparseLoCo implementation with pipeline parallelism over low-bandwidth connections using subspace networks [4]; low-rank compressions of the residual stream at pipeline boundaries which enable a model to be pipelined across non-colocated GPUs. Sharding a model across disjoint hardware inevitably increases exposure to faults within the pipeline. Without mitigation, a single failed stage stalls the entire pipeline, leaving the entire replica idle. This can be particularly problematic as an entire replica would need to wait until the missing stage is replaced and then until the next outer round is completed to resume processing, leaving all remaining stages idle on the replica for significant periods of time.
One solution to this problem in the literature is SWARM [5] which allows having a pool of computational units at each pipeline stage and simply rerouting when one of the pool members is missing. Consider a failed stage on replica . In Crucible a variant of this would route activations from stage on replica to other stages in the other remaining replicas. However, this can lead to lower throughput and greatly increases system complexity. We have thus been investigating a simpler approach that leverages the structure of the transformer based on a variant of SkipPipe [6] adapted to SparseLoCo and low rank compression between the model pipeline stages. This explores temporarily rerouting forward and backward passes around downed pipeline stages, allowing the replica to continue processing tokens and contributing pseudogradients to the run.
The main idea leverages some fundamental attributes of the transformer; the residual connections running throughout the model and the homogeneity of transformer blocks. A temporary rerouting therefore essentially means omitting residual stream modifications by whichever transformer blocks are part of the downed stage, but this does not fundamentally corrupt the hidden states.
Figure 2. A temporary inner-stage failure bypasses that stage within the affected replica. Other replicas keep their full paths. The bypass requires compatible tensor shapes and projection interfaces; it does not supply the missing stage’s computation.
Experimental setup
In order to investigate the viability of stage-skipping during node failures, we experimented with simulating stage failures within Crucible training runs. We consider a setting consisting of 8 replicas with each model split over 4 stages, and simulate temporary failures on inner pipeline stages by temporarily rerouting the forward and backward pass around pipeline stages. The probability of a replica experiencing a failure on any given global step is . When a failure occurs, one of the replica's inner stages is chosen at random, the forward and backward passes are rerouted around this stage for global steps, as in Figure 2. Each replica can only ever experience a single stage failure at a time, however, multiple different replicas can experience stage failures simultaneously.
While a stage is down, the replica continues training with the failed stage removed from the pipeline: activations from the preceding stage are passed directly to the following stage on the forward pass, and gradients flow back along the same shortcut on the backward pass. The healthy stages take their inner optimizer steps as normal, while the downed stage performs no computation. At the outer step, all healthy stages within the replica broadcast their pseudogradients to other replicas as normal, and only the downed stage is omitted.
During a simulated stage failure, the downed stage acts as a passive participant in the pseudogradient all-gather; it does not contribute but continues to apply pseudogradients to stay in sync with the equivalent stage in other replicas. When the failed stage rejoins the replica after global steps, its optimizer moments and error buffer are reset, resembling a node that has been totally lost and then restored with the current global weights, but none of its local optimizer state.
Therefore this simulation captures the following algorithmic effects of inner-stage failures and recoveries, which are: (a) the missed pseudogradients from the failed stage (b) the impact on the healthy stage’s weights from training under temporary forward/backward pass rerouting (c) the effect of losing the error buffer and optimizer moments during a run due to hardware faults.
Robustness under stage failures in Crucible
Figure 3. Relative validation-loss degradation under simulated inner-stage failures, with . The horizontal axis expresses the per-replica failure probability as a percentage. Solid curves show stage skipping; dashed curves show dropping tokens assigned to the affected replica. Each curve uses its own no-failure baseline.
We use a 178M model with Chinchilla-optimal token budget for these experiments. Figure 3 shows how much the final validation loss changes as stage failures become more frequent. All curves start at zero because each is normalized to its own no-failure run. Moving right increases the chance of a new failure on an eligible replica; each failure then removes that stage for six global steps whilst the replica continues training with rerouted forward and backward passes. The vertical axis measures the resulting percentage increase in loss, not the fraction of failed workers or the fraction of training time lost.
We experimented with increasing the failure rate, under both uncompressed pipeline boundaries and subspace network compression at a factor of 8x compression. The compressed configuration shows a smaller relative loss increase than the uncompressed case, a gap that increases with the simulated failure probability. At the highest tested probability, the increases are approximately 0.87% and 1.20%, respectively. These are relative changes against each configuration’s own no-failure baseline, not a comparison of their absolute losses.
We observe that overall even at relatively high failure probability such as 10%+ (which would imply extremely frequent node failures) the relative validation-loss increase remains around 1% in SparseLoco without Pipeline Compression. When utilizing the fixed Subspace projections used in Crucible we actually observe significantly more robustness to these node failures.
Skipping stages versus dropping tokens
A simple alternative is to stop processing the tokens assigned to a degraded replica until its failed stage recovers. This drop-token baseline leaves the remaining replicas training normally, but consumes fewer tokens within the same scheduled run. In Figure 3, dropping tokens increases validation loss by about 3.4–3.6% at , compared with about 0.6–0.9% when the failed stage is bypassed. The comparison shows the benefit of continuing to process tokens through the healthy stages rather than leaving the entire replica idle.
Alternatively, the drop-token run could be extended to make up the missing tokens. That would hold the token budget fixed but require additional training time and compute; the plotted results compare the original schedule, not this extended-run alternative.
Why shared projections may help
Subspace Networks communicate through low-dimensional representations. One possible reason fixed, shared projectors help during a bypass is that they give different stage boundaries a common coordinate system and distribute information across the projected coordinates, potentially reducing sensitivity to individual components of the preceding stage’s representation.
To investigate the role of fixed projections, we compare SSN with MAPL [7], where instead of using a fixed projection for all pipeline boundaries, the compression matrix at each boundary is learned during training under some orthonormality constraints. The table below reports the percentage increase in validation loss relative to each configuration’s own no-failure baseline.
| Failure probability per replica | Fixed SSN | MAPL | Uncompressed |
|---|---|---|---|
| 10% (p = 0.10) | 0.6% | 0.9% | 0.9% |
| 20% (p = 0.20) | 0.9% | 1.1% | 1.2% |
Table 1. Percentage increase in validation loss relative to each configuration’s no-failure baseline, with stage skipping enabled.
Fixed SSN exhibits the smallest loss increase at both failure probabilities. MAPL instead shows degradation close to the uncompressed baseline, suggesting that compression alone does not explain the robustness benefit. These results support the importance of fixed projections and are consistent with the hypothesis that a stable, shared subspace helps preserve compatibility when intermediate stages are bypassed.
Why fault tolerance matters at scale
Even a small per-replica failure probability can become disruptive across a large training job. As an illustration, 0.1% per eligible replica per global step means one failure per 1,000 eligible steps on average for that replica. With eight healthy replicas, and assuming independent failures, the chance that at least one fails on the next global step is about 0.8%: . This illustrates why a job that must stop for every failed participant becomes increasingly fragile as it grows.
This illustrative 0.1% rate is below the nonzero probabilities tested in Figure 3. The experiment measures risk per eligible global step; translating it into an hourly interruption rate requires the duration of a global step and the actual failure process. AWS Spot Instance Advisor, for example, reports interruption frequency over the trailing month [8], so its percentages are not directly comparable with this experiment.
This matters economically because a larger supply of compute is available if every machine does not have to remain continuously available for the whole run. Spot instances are one example: AWS makes spare capacity available at discounts of up to 90% relative to On-Demand prices, but can reclaim it [9]. For stop or termination, interruption notices are issued on a best-effort basis two minutes beforehand [10]. A training system that can continue through a missing contribution or temporarily bypass an inner stage could use some of that interruptible capacity without repeatedly idling all of its healthy workers.
The same principle applies beyond a single cloud. Crucible can aim to aggregate workers across providers and locations, including machines available only for part of a long run. Tolerating departures and rejoining workers widens the pool we can draw from and makes it less dependent on one uninterrupted allocation. Fault tolerance is therefore critical to globally distributed pre-training: adding more compute is useful only if its interruptions do not erase the benefit. The experiments here support that direction, while production savings still depend on replacement time, checkpoint and rejoin costs, bandwidth, and correlated failures.