Large-scale pre-training has traditionally assumed one place: one datacenter, one tightly coupled cluster, and one network fast enough that communication can be treated as infrastructure rather than part of the algorithm.

Templar's mission is to pioneer AI infrastructure that is robust to hardware topology. Useful GPUs are scattered across clouds and regions. The hard problem is turning this poorly connected hardware into one dependable and economically competitive training system without degrading the final model performance.

Crucible is our new pre-training system for assembling globally distributed compute into concurrent, large-scale training jobs. It combines two lines of our research: SparseLoCo for communication between data-parallel replicas, and low-bandwidth pipeline parallelism based on Subspace Networks and our work on heterogeneous low-bandwidth pre-training alongside our cloud platform (Basilica), which aggregates and validates globally distributed GPUs and mediates interaction between them.

Our team previously established the research foundation for Crucible with Covenant-72B, a 72-billion-parameter model trained across globally distributed, low-bandwidth workers. To the best of our knowledge, it remains the largest publicly reported pre-training run of its kind and maintains learning performance close to centralized baselines. Crucible turns those results into a repeatable deployment system. It combines our communication-efficient training methods (now extended to allow more participants) with Basilica’s scalable GPU pool. On top of that pool it adds network-aware worker selection. It also builds and orchestrates the training pipelines and runs concurrent jobs.

In one of our first end-to-end runs, Test Run 003, Crucible processed 50.53B tokens for an 8B dense model at an estimated $0.1202 per million tokens on our distributed cloud infrastructure using 48 A100 GPUs. Its end-to-end effective MFU was 48.3%, implying a 25.6% low-bandwidth overhead relative to the reported 65% MFU ceiling for A100 pretraining (Team PyTorch at IBM and Meta, 2024).

This overhead can be greatly improved through future engineering and research efforts our team is undertaking. Even with the current modest overhead, the economics are already compelling. Using publicly listed AWS A100 Capacity Blocks prices as a reference, Crucible’s estimated GPU rental cost per token was approximately 29% lower even after accounting for the reduction in throughput that comes from low-bandwidth links. The saving comes from not needing to reserve one tightly connected block of A100s at once: Crucible can assemble appropriately priced accelerators across locations, then use compression and orchestration to keep them productive over lower-bandwidth links at a net gain in cost effectiveness.

Loss tracked our internal centralized training benchmark closely over the run, with minimal observed degradation, as shown in our previous publications on SparseLoco and the follow-on work with pipelineing.  To the best of our knowledge, we have also not seen a stronger publicly visible throughput profile and model performance from a live deployment in this class of globally distributed pre-training systems. Since our system is built on top of a live cloud platform we are also able to show a first clear economic demonstration in a live system that globally distributed pre-training can be cost-competitive after its network overhead is included.

Figure 1. Crucible groups globally distributed workers into multi-stage pipeline replicas. Subspace-compressed activations travel over the low-bandwidth links between adjacent stages, while SparseLoCo pseudo-gradients synchronize across replicas through R2 under a shared orchestrator.

Training over the internet

Training at scale typically involves data parallelism, where several identical copies of a model (replicas) train simultaneously, each on its own shard of the dataset, and synchronize their gradients before every optimizer step, keeping all replicas identical throughout the run.

Hosting a full replica of the model at scale requires many GPUs, beyond what a typical server can hold. A replica must therefore typically span several machines communicating over the internet. Crucible does this with pipeline parallelism (PP): the model's layers are divided into consecutive stages, one per machine, so that each server hosts only a fraction of the model, a stage in the pipeline. Each stage communicates only its activations—its intermediate outputs— and the corresponding gradient-activations with its neighboring stages, making PP an efficient protocol for partitioning a large model across machines connected by limited-bandwidth links.

Two communication paths therefore cross the open internet—between replicas, and between adjacent stages:

1. Across replicas: Crucible uses SparseLoCo, a communication-efficient data-parallel optimizer that replaces per-step gradient synchronization with infrequent, compressed exchanges. Crucible further reduces the cross-replica communication cost by streaming each update in fragments and overlapping the transfers with ongoing computation, hiding cross-replica communication behind computation, as described in the next section.
2. Inside a replica: Cross-stage communication is the critical path: every microbatch crosses every stage boundary, forward and backward. These transfers are activation-sized—far smaller than the model—but at scale the activations of a full microbatch are still substantial, particularly over slow internet links. When adjacent stages reside on different machines, Crucible compresses each boundary tensor into a lower dimensional representation and reconstructs it on the receiving side, reducing the traffic to a fraction of its original size. The compression rate is configurable per link: stages with high bandwidth, e.g., co-located in the same datacenter, can communicate uncompressed, while stages connected over slower links apply as much compression as their bandwidth requires to sustain pipeline throughput. Each stage is typically hosted by a single server with fast, interconnected GPUs utilizing standard parallelization techniques such as FSDP or tensor parallelism.

Crucible's capacity to incorporate low-bandwidth connections on both these paths means the only requirements for participation are an internet-grade connection and enough GPU capacity to host a single stage. This is what enables Crucible to combine accelerators from different providers in a single training run.

Overlap-streamed SparseLoCo

Conventional data-parallel training synchronizes a model-sized gradient at every optimizer step. SparseLoCo instead lets each replica take multiple local steps and then share only a compressed pseudo-gradient. Because this cross-replica communication is both infrequent and small, replicas synchronize quickly even over ordinary internet connections.

Figure 2. Streaming staggers fragment captures across the local-training cycle. For each fragment, R2 upload and peer download run in the background while the next $\tau$ optimizer steps proceed; only transfer time beyond that overlap window becomes a blocking tail.

Crucible also keeps training through node failures. Conventional data-parallel training synchronizes through collective operations such as all-reduce, in which every participant sits on the critical path: a single node failure breaks the collective, and training halts until a new process group is formed and the communication restarts. Crucible's replicas never communicate directly. Each uploads its compressed fragments to R2 and independently fetches those of its peers, so a failed replica does not interrupt the round: if it fails after uploading, its update is still consumed by the others; if it fails beforehand, the remaining replicas aggregate without it and training continues without repeating the communication.

Formally, let $\theta^{\mathrm{anchor}}_r$ be replica $r$’s last accepted model and $\theta^{\mathrm{local}}_r$ its model after multiple local optimization steps. The pseudo-gradient is

SparseLoCo folds this update into an error-feedback buffer $e_r$, keeps the largest coordinates within each tensor chunk, and quantizes the retained values to two bits:

The communicated update is typically 1–3% dense—equivalently, 97–99% sparse—with the 2-bit quantization, yielding up to 171x compression over the model size. Sparsified information is also not lost: error feedback accumulates the unsent coordinates so they can enter a later update. Notably, despite this aggressive compression, SparseLoCo consistently outperforms its dense-communication baseline, DiLoCo.

Streaming

In a conventional outer round—where replicas communicate—each replica forms a whole-model pseudo-gradient after $H$ local optimizer steps and exchanges it all at once. To avoid concentrating communication in a single large burst, Crucible partitions the model into $P$ non-overlapping fragments and captures and transfers one fragment at a time throughout each $H$-step cycle. With $q=\lfloor H/P\rfloor$, fragment $f$ is captured after inner step

At $c_f$, Crucible forms and compresses only that fragment’s pseudo-gradient relative to its last consensus anchor. Streaming therefore spreads cross-replica communication across the cycle, rather than concentrating it in one whole-model exchange every $H$ steps.

Overlap

As soon as a fragment is captured, a dedicated background I/O loop begins uploading it to R2 while concurrently fetching the corresponding fragment from the other expected replicas. The training thread continues for another $\tau$ local optimizer steps before consuming the aggregate. This overlap therefore enables training and resource utilization while the communication takes place. Here $\tau$ is measured in inner optimizer steps—not outer rounds—and must satisfy $0\leq\tau<H$.

If $s_f$ is the fragment’s capture step, Crucible schedules its application at

If the transfer has completed by step $a_f$, applying the update incurs essentially no network wait; if it is still in flight, training blocks only for the remaining transfer time. Crucible then applies the accepted aggregate at the fragment’s consensus anchor and blends in the local drift accumulated during those $\tau$ steps:

The implementation does not claim to hide every communication cost. SparseLoCo compression, intra-replica shard assembly, any residual wait, and the outer update remain on the synchronous path.

What overlaps with subsequent local training is specifically the fragment’s upload and concurrent peer downloads. The design builds on Streaming DiLoCo, but composes its fragment schedule with SparseLoCo compression and object-store transport. Notably, overlapping can degrade performance when the number of inner steps tau becomes too large; thus, the basic SparseLoCo method (compression and infrequent communication) still carries the largest burden in reducing communication overhead.

Our ablations show that SparseLoCo holds its accuracy when communication is fragmented and overlapped with local training (assuming tau stays small). Validation loss stays close to the corresponding no-streaming baseline across every tested configuration. Even with streaming and overlap enabled, every configuration continues to outperform the dense-communication DiLoCo baseline (Table 1).

Table 1. SparseLoCo streaming ablations for the 512M model. Percentages show the relative change in validation loss from the corresponding SparseLoCo baseline at the same training budget. At 10.26B tokens, the largest observed gap is 0.33%. At the longer 41.02B-token budget the gap disappears, and validation loss is 0.13% lower than the no-streaming baseline.

Subspace Networks for low-bandwidth pipelining

SparseLoCo reduces communication across replicas. It does not address the communication inside a replica. Replicas typically utilize pipeline parallelism, where the critical path is every microbatch crossing every stage boundary.

For that path, Crucible uses the compressed pipeline construction from our heterogeneous low-bandwidth pre-training paper validated with SparseLoco, based on Subspace Networks.

Let a pipeline-boundary hidden state be $X\in\mathbb R^{b\times L\times d}$, for microbatch size $b$, sequence length $L$, and hidden dimension $d$, and let $U\in\mathbb R^{d\times k}$ have orthonormal columns with $k\ll d$. After removing a fixed token-conditioned component $T_\perp[x]$ and positional component $P$, the sender projects the residual into the shared subspace:

The receiving stage reconstructs an approximation in the model dimension:

During backpropagation, the boundary gradient is compressed with the same basis:

The activation payload scales with $k/d$. A compression ratio of $k/d=1/8$, for example, gives a nominal 87.5% reduction in boundary tensor size before protocol overhead. Each boundary may use a different compression ratio based on its bandwidth. Crucible packs the compressed tensor and the token identifiers needed for reconstruction into one message per microbatch in each direction.

The current implementation uses a deterministic orthonormal basis shared by the stages. That creates a stable communication contract: each worker knows the full model dimension $d$, communicated dimension $k$, projection seed, and which contiguous layers it owns before the first microbatch is sent. The full construction and experiments are in our paper, Heterogeneous Low-Bandwidth Pre-Training of LLMs.

Orchestrating the GPU pool

Crucible’s orchestrator turns an available pool of GPUs into a concrete training topology. It tracks which workers are live, assigns them to pipeline stages, and publishes a run-specific configuration so every participant agrees on stage ownership, compression settings, and the synchronization schedule. Updates, checkpoints, and control state are isolated by run, allowing multiple jobs to share the infrastructure without sharing state.

Bandwidth is part of this placement decision. Crucible profiles the available pool and forms replicas whose adjacent links can support the selected microbatch and compression settings, while ensuring that every replica can participate in the object-storage exchange used by SparseLoCo. This is what turns the communication methods described above into a deployable system across heterogeneous providers.

Economic Validation of Crucible

We have completed three end-to-end Crucible test runs. Our latest TEST_RUN_003, an 8B dense Llama-style model trained with SparseLoCo and four-stage low-bandwidth pipelines, used 48 A100 80 GB GPU workers organized as 12 replicas × 4 pipeline stages. The GPUs are rented through our globally distributed cloud platform (basilica) Across the run, the public run dashboard reports:

At 130.1K tokens/s, 50.53B tokens corresponds to approximately 107.9 hours, or 4.50 days. Multiplying the dashboard’s unit cost by the token count gives an estimated GPU rental total of approximately $6,074.

What MFU and the “low-bandwidth tax” mean here

Model FLOPs utilization (MFU) compares achieved model throughput with the theoretical tensor-core throughput of the GPUs:

MFU is GPU-, precision-, and FLOP-convention-dependent. The 48.3% number is Crucible’s end-to-end effective/global MFU: it includes the time spent on pipeline communication and outer-round synchronization.

For Test Run 003, per-stage pure-compute MFU was not independently instrumented. The dashboard uses a configured 65% compute ceiling and defines the low-bandwidth overhead as the relative gap from that ceiling:

Using the rounded display values, $1-48.3/65\approx25.7\%$; the dashboard shows 25.6% from the unrounded series. This is an operational estimate of the combined communication and orchestration penalty. It is not a direct measurement of time spent on the network, and it is not a second MFU value.

An AWS price comparison

The dashboard's $0.1202/Mtok estimate is derived from observed end-to-end throughput and the rental price of the participating GPUs. Comparing to AWS: it currently lists a p4de.24xlarge Capacity Block at $17.712/hour in US East (N. Virginia) and US West (Oregon). Each instance contains eight A100 80 GB GPUs, so matching 48 GPUs requires six instances, or $106.272/hour. Assuming the 65% MFU ceiling for a tightly interconnected cluster, scaling Test Run 003's measured 130.1K tokens/s by the MFU ratio (65 / 48.3) gives an estimated AWS throughput of 175.1K tokens/s. Then:

On this basis, Crucible's observed cost-per-token is about 29% lower, after accounting for the low-bandwidth tax that comes from training across non-colocated hardware. Breaking this down, our hardware costs roughly $1.17 per GPU-hour against AWS's $2.21, a 1.89× price gap. The cheaper hardware comes at the cost of low-bandwidth communications, and a resultant decrease in MFU from the 65% compute ceiling to the measured 48.3%, a 1.35× reduction in token throughput. These factors combine for a net 1.40× cost advantage per token. For the AWS cluster to achieve a cost efficiency of $0.1202/Mtok achieved by Crucible and Test Run 003, it would need to sustain 245.6K tokens/s at roughly 91% MFU, well beyond what is realisable in practice.

Note that the 65% ceiling is configured, not measured. However we provide this as a strongly conservative comparison since well-tuned dense A100 runs typically land below 65% MFU. Additionally estimating the AWS prices using Capacity Blocks also errs on the side of conservatism, being a cheaper option than on-demand rates. Conversely, the comparison does not price R2 storage and operations expenses required for Crucible orchestration and synchronisation, and AWS rates can change. AWS Capacity Blocks pricing.

What comes next

Test Run 003 gives us our first direct measurement of globally distributed pre-training unit economics at this topology: 48 pipeline workers, 12 independent replicas, and 50.53B tokens, with cost and throughput numbers that include the low-bandwidth overhead rather than hiding it behind a compute-only benchmark.

There is significant work ahead. Distributed training at scale over varying topology is a frontier with both new challenges and new opportunities. Challenges in the form of network outages and less reliable hardware that will need to be handled with novel fault tolerance and recovery procedures. Opportunities can be found from tolerating participant churn gracefully, and therefore leveraging compute that is flexible not only in space but in time through the use of spot instances.

Our next systems milestone is to scale the same composition to larger models. Beyond that, the goal is a sustained, long-duration pre-training run in which throughput and loss, along with recovery and unit economics, are measured over the full training horizon rather than a test window.

Crucible is the system that lets us run those experiments at the scale and on the networks that matter.

Read the research

• Overcoming the Communication–Performance Tradeoff in LLM Pretraining (SparseLoCo)
• Heterogeneous Low-Bandwidth Pre-Training of LLMs
• Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
• Streaming DiLoCo with Overlapping Communication
• Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet