Diffusion Training from Scratch on a Micro-Budget

The paper titled "Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget" by Vikash Sehwag and colleagues addresses the challenges of training large-scale text-to-image (T2I) generative models, particularly in the context of high computational costs that favor well-resourced developers. The authors propose a novel approach to significantly reduce these costs by implementing a strategy that randomly masks up to 75% of image patches during training. This deferred masking technique, which preprocesses patches using a patch-mixer, minimizes performance degradation compared to traditional model downscaling methods. Additionally, the authors leverage advancements in transformer architecture, including mixture-of-experts layers, to enhance performance. They highlight the effectiveness of using synthetic images in training on a micro-budget. The study demonstrates that with only 37 million publicly available real and synthetic images, they successfully trained a 1.16 billion parameter sparse transformer for a mere $1,890, achieving a Fréchet Inception Distance (FID) of 12.7 in zero-shot generation on the COCO dataset. This model's performance is notably competitive, costing 118 times less than stable diffusion models and 14 times less than the current leading approach, which costs $28,400. The authors aim to release their end-to-end training pipeline to democratize access to large-scale diffusion model training for those with limited budgets.