DeepSeek and the Effects of GPU Export Controls
DeepSeek launched its V3 model, trained on 2,048 H800 GPUs for $5.5 million, emphasizing efficiency and innovation due to U.S. export controls, while exploring advancements beyond transformer architectures.
Read original article
DeepSeek recently launched its V3 model, which was trained using 2,048 H800 GPUs at a cost of $5.5 million, significantly less than the estimated $40 million spent on OpenAI's GPT-4. Despite using fewer resources, DeepSeek claims their model meets or exceeds benchmarks set by leading AI models. The U.S. export controls on high-end GPUs to China forced DeepSeek to innovate with the limited H800s, leading to a focus on architectural efficiency rather than sheer computational power. This included advancements in FP8 precision training and novel training frameworks. DeepSeek is backed by High-Flyer, a quant fund, and its CEO emphasizes a long-term vision for artificial general intelligence (AGI) rather than immediate profits. While DeepSeek's achievements are notable, experts caution against overinterpreting the results, suggesting that the path to improved AI may not solely rely on increased hardware resources. The company is also exploring ways to overcome the limitations of transformer architectures, indicating potential future advancements in AI training methodologies.
- DeepSeek's V3 model was trained on 2,048 H800 GPUs at a fraction of the cost of leading models.
- U.S. export controls on GPUs led to innovative training methods focused on efficiency.
- DeepSeek is backed by High-Flyer, emphasizing foundational research over quick profits.
- The results suggest that significant AI advancements can be made without extensive hardware resources.
- Future developments may focus on overcoming transformer architecture limitations.
Related
DeepSeek-V3, ultra-large open-source AI, outperforms Llama and Qwen on launch
Chinese AI startup DeepSeek launched DeepSeek-V3, a 671 billion parameter model outperforming major competitors. It features cost-effective training, innovative architecture, and is available for testing and commercial use.
DeepSeek's new AI model appears to be one of the best 'open' challengers yet
DeepSeek, a Chinese AI firm, launched DeepSeek V3, an open-source model with 671 billion parameters, excelling in text tasks and outperforming competitors, though limited by regulatory constraints.
Interesting Interview with DeepSeek's CEO
Deepseek, a Chinese AI startup, has surpassed OpenAI's models in reasoning benchmarks, focusing on foundational AI technology, open-source models, and low-cost APIs, while aiming for artificial general intelligence.
DeepSeek v3: The Six Million Dollar Model
DeepSeek v3 is an affordable AI model with 37 billion active parameters, showing competitive benchmarks but underperforming in output diversity and coherence. Its real-world effectiveness remains to be evaluated.
Notes on the New Deepseek v3
Deepseek v3, a leading open-source model with 607 billion parameters, excels in reasoning and math tasks, outperforming competitors while being cost-effective, trained on 14.8 trillion data points for $6 million.
12 commentsEven so, if we look at Groq / Cerebras, the fastest LLM inference companies:
They're both based on architectures that are 7nm+, and so architectures that China can produce locally despite the export restrictions.
Ultimately, the export controls are mainly just inconvenience. Not a real blocker.
The Chinese don't need to achieve state of the art chip manufacturing to achieve SOTA AI outcomes.
They just need to make custom silicon specialized for the kinds of AI algorithms they want to scale.
Of course, at scale, that's going to mean that the US should eventually have both lower production costs, and energy use in consumer use of AI models, and that Chinese products will likely be more dependent on the cloud for at least the near future.
The whole strategy seems ultimately meh in a long term sense... Mainly good for building up a sense of mutual enmity and dividing the world ... Which is also going to result in higher cost of living around the world as trade falters.
Sad stuff.
China can produce much cheaper electronics that can compete even when they aren't as powerful as NVIDIA's
> Novel training frameworks
Where can one find more information about these? I keep seeing hand-wavy language like this w.r.t. DeepSeek’s innovation
How is that useful?
I would assume nothing, similarly to how exports of western tech from western countries somehow magically exploded overnight to Russia's neighbors and everyone is pretending not to notice because it makes money.
Should have been obvious but now somehow isn't?
Related
DeepSeek-V3, ultra-large open-source AI, outperforms Llama and Qwen on launch
Chinese AI startup DeepSeek launched DeepSeek-V3, a 671 billion parameter model outperforming major competitors. It features cost-effective training, innovative architecture, and is available for testing and commercial use.
DeepSeek's new AI model appears to be one of the best 'open' challengers yet
DeepSeek, a Chinese AI firm, launched DeepSeek V3, an open-source model with 671 billion parameters, excelling in text tasks and outperforming competitors, though limited by regulatory constraints.
Interesting Interview with DeepSeek's CEO
Deepseek, a Chinese AI startup, has surpassed OpenAI's models in reasoning benchmarks, focusing on foundational AI technology, open-source models, and low-cost APIs, while aiming for artificial general intelligence.
DeepSeek v3: The Six Million Dollar Model
DeepSeek v3 is an affordable AI model with 37 billion active parameters, showing competitive benchmarks but underperforming in output diversity and coherence. Its real-world effectiveness remains to be evaluated.
Notes on the New Deepseek v3
Deepseek v3, a leading open-source model with 607 billion parameters, excels in reasoning and math tasks, outperforming competitors while being cost-effective, trained on 14.8 trillion data points for $6 million.