The problem of 'model collapse': how a lack of human data limits AI progress

Research indicates that the reliance on synthetic data to train artificial intelligence (AI) models poses significant risks, potentially leading to nonsensical outcomes. Major AI companies, including OpenAI and Microsoft, have explored using synthetic data as they exhaust available human-generated data. A study published in Nature highlights that synthetic data can cause rapid degradation of AI models, with one experiment showing a model's output devolving into irrelevant topics after just a few generations of training. The study emphasizes that the accumulation of errors in successive training cycles can lead to a phenomenon known as "model collapse," where the model loses its utility due to overwhelming inaccuracies.

The research found that early signs of collapse involve a loss of variance, resulting in overrepresentation of majority subpopulations while minority groups are neglected. As the collapse progresses, the output may become increasingly nonsensical. The study also noted that models trained on their own outputs tend to produce repetitive phrases and distorted representations of data. To mitigate these issues, some companies are implementing techniques like watermarking AI-generated content to prevent it from being used in training datasets. However, effective coordination among tech companies remains a challenge. The findings suggest that companies that have sourced diverse training data from the pre-AI internet may have a competitive advantage in developing more accurate generative AI models.