Nvidia GPU roadmap confirms it: Moore's Law is dead and buried

The title is usual clickbaity Register but the content is good enough.

TL;DW:

  * The improvements don't come from transistor density but other tricks like putting 2 chips together, using smaller 4bit format, more on-chip memory, 2x memory bandwidth
  * And Nvidia is packing more GPUs in the same rack and consuming unheard of amounts of power, with a huge toll on datacenter infra

And it seems all these have growth limits. No more 2x every year or even every other year.

No surprise there, that Moore's Law requires a healthy competition environment for it to be likely to take effect and it's been a while since Nvidia had actual competition.

Of course it's much more complex than that. The nature of the problems that tech now has to solve is different and as stated in the article, Nvidia hit many roadblocks, but I still think if it had healthy competition, other brands would step in and make it more likely for a creative solution to manifest itself.

An earlier big data mantra said to move computation to where the data is - as data is too big to move around. Then I guess here, it could turn out, computation should be done where energy is, as energy is potentially even harder to move around than data? Tens of gigabytes per second is easy through one tiny fiber. A bundle is easily laid by an ordinary excavator. In contrast a transmission line for a gigawatt for a few hundred kilometers takes billions.

With proper engineering with nothing fundamentally new, one could get proper integrated water cooled racks with very high computing density and power usage. Then put those racks in a building close to a nuclear powerplant (outside the security zone perhaps though). Or a wind or solar park + batteries. Can you circumvent the costs associated with using the public power grid?

Extremely simplified, you only need fiber in and out of this power+computing facility. Ultimately you could do this in space with solar power and laser up/downlink. Cooling might be problematic though.

France at least used to have a dedicated nuclear plant just for uranium enrichment. Before electricity, a lot of industry centered to places near running water for sawmills, flour mills, metal hammering and so on. With electricity these were in many cases decoupled. But things that get energy intense enough and don't need material transport or local labor, it might make sense to locate close to energy sources again.

> Anytime Meta, Oracle, Microsoft, or anyone else announces another AI bit barn, a juicy power purchase agreement usually follows. Meta's mega DC being birthed in the bayou was announced alongside a 2.2GW gas generator plant — so much for those sustainability and carbon neutrality pledges.

Ouch, I grew up where Meta created its first data center and while it is very efficient (https://engineering.fb.com/2011/04/14/core-infra/designing-a...) I remember a more recent article about how they are zero net emissions (https://tech.facebook.com/engineering/2021/11/10-years-world...). I wonder if they have kept that in this AI rush (along with most of the datacenter companies).

EDIT: a direct link to net zero for Meta: https://tech.facebook.com/ideas/2020/9/facebooks-path-to-net...

Intel made 3.7 GHz CPUs 20 years ago. Modern CPUs run at the same frequency.

So was Moore's law dead 20 years ago? Clearly not.

Right now Nvidia's focus is on making bigger silicon with faster interconnects. But pretty soon that will stop working and then the focus will shift again. It's still early days for AI. People have only just started working on making dedicated AI chips. Presuming that performance per watt or per mm^2 of silicon cannot go up from here seems silly.

The economist wrote an article "Jensen Huang says Moore’s law is dead. Not quite yet" (https://www.economist.com/science-and-technology/2023/12/13/...) which refutes this claim quite nicely. It's two years old but still pretty accurate.

The article mentions 600kW power to a single rack might be a thing soon. Surely not air-cooled ?

This seems wild to me. I used to warm myself next to 4kW racks of telco gear when I was still doing overnight work (20 ish years ago) and thought this was a lot of power to be using in such a small space…

If price per compute keeps going down we effectively keep having moore’s law for parallel compute like GPUs. (Just get better at making ML models that don’t have as big of a communication bottleneck.)

Maybe we are near the limits of the silicon. I wonder if changing to some other medium, carbon, optical circuits etc., wouldn't allow us to benefit from Moore's law again.

Nvidia just wants you to accept higher prices for less computing power.

That’s why Jensen is always saying Moores Law is dead - so you buy less of his sand for more bucks.

Isn't Moore's law about flops/watt nowadays? Yes that skips the Nvidia part which is $/(flops/watt) but they should just price it at a level where they maximise profits (which is also a function of competitive environment, so not relevant for Moore's law).

The reason it's about flops/watt these days is that potentially the demand for "intelligence" limitless.

I have been reading Moore's Law is dead articles for the last 30 years. You sort of become immune to the "sky is falling" narrative after a while.

Tech leaders love this narrative:

2005: Intel CEO https://hothardware.com/news/moores-law-is-dead-says-gordon-...

2009: Sandisk CEO https://archive.nytimes.com/bits.blogs.nytimes.com/2009/05/2...

2010: NVIDIA VP https://finance.yahoo.com/news/2010-05-03-nvidia-vp-says-moo...

2016: Intel CEO https://www.nytimes.com/2016/05/05/technology/moores-law-run...

2017: NVIDIA CEO https://www.extremetech.com/cars/256558-nvidias-ceo-declares...

2019: NVIDIA CEO https://www.cnet.com/tech/computing/moores-law-is-dead-nvidi...

2022: NVIDIA CEO https://www.barrons.com/articles/nvidia-graphic-card-prices-...

The only time this felt true was on the desktop/Windows/Linux for CPUs when Intel had an uncontested monopoly from 2009 - 2016:

https://cdn.arstechnica.net/wp-content/uploads/2020/11/CPU-p...

But it was broken by AMD's Ryzen architecture on one side and Apple's M series on the other.

TL;DR it’s dead in the sense, we’re falling short of doubling the transistor count every 18-24 months.

That said, in terms of horizontal expansion (tripling or quadrupling the number of computational cores), surely the main challenge must be the lack of bandwidth to the memory to power such insane data transfer rates.

You could of course dedicate some amount of memory to each compute node, then the problem is one of quickly splitting the workload into chunks and sending them their separate way with no need to move them around. In a way, LLM and neural networks do function that way. Maybe it is time to build hardware that maps 1:1 to the software architecture.

I don't know why, but the idea of a 72 GPU rack provokes a kind of existential dread in me.

Surely there will be a sizeable increase in performance from the die shrink from 4NP to 2NP.