Technical reports from building pre-training and post-training systems across open, heterogeneous compute.
A 72B-parameter language model trained with open, permissionless participation over commodity internet links, using Gauntlet incentives and SparseLoCo compression to keep unreliable peers useful.
RL fine-tuning changes only about 1% of model weights per step. PULSE exploits that sparsity to synchronize policy weights losslessly with over 100x less bandwidth.
A framework for letting resource-limited participants form a SparseLoCo replica through compressed pipeline parallelism while better-connected replicas stay full precision.
SparseLoCo replaces dense pseudo-gradient exchange with chunked TOP-k sparsification, local error feedback, and 2-bit values while matching or improving DiLoCo-style pre-training.
Gauntlet scores pseudo-gradient contributions from untrusted peers with fast reliability checks, loss-before/after validation, and persistent OpenSkill ratings.
Every paper above started as an open problem. We hire researchers and engineers to work on the next one: optimizers, incentives, and systems for training frontier models over the open internet. We also collaborate with external researchers and labs, and offer research compute grants for promising work on decentralized intelligence.