Technical notes from building pre-training and post-training systems across open, heterogeneous compute.
Our new scalable pre-training platform processed 50B tokens through an 8B model over globally distributed GPUs at a cost of $0.1202 per million tokens, 29% below a centralized AWS A100 cluster.
PULSELoCo applies the PULSE compute-visibility rule to DiLoCo-style pseudo-gradients: it matches DiLoCo learning behavior while sending over 17x less data than DiLoCo and about 138x less than DDP on Qwen2.5-7B.
The architecture, verification pipeline, and incentives behind one of the first open decentralized RL networks with real-time transparency — miners generate rollouts, validators verify them with the grail proof, and a trainer learns from a live network.
