Grace Hopper, Nvidia's Halfway APU

If AI remains in the cloud, nvidia wins. But I can’t help but think that if AI becomes “self-hosted”, if we return to a world where people own their own machines, AMDs APUs and interconnect technology will be absolutely dominant. Training may still be Nvidias wheelhouse, but for a single device able to do all the things (inference, rendering, and computing), AMD, at least currently, would seem to be the winner. I’d love someone more knowledgeable in AI scaling to correct me here though.

Maybe that’s all far enough afield to make the current state of things irrelevant?

I am really surprised to see the performance of the CPU and especially the latency characteristics are so poor. The article alludes to the design likely being tuned for specific workloads, which seems like a good explanation. But I can't help wonder if throughput at the cost of high memory latency is just not a good strategy for CPUs even with the excellent branch predictors and clever OOO work that modern CPUs bring to the table. Is this a bad take? Are we just not seeing the intended use-case where this thing really shines compared to anything else?

Irrelevant, but the intro reminded me that nvidia also used to dabble in chipsets like nforce, back when there was supplier variety in such.

In my tests of a Supermicro ARS-111GL-NHR with a Nvidia GH200 chipset, I found that my benchmarks performed far better with the RHEL 9 aarch64+64k kernel versus the standard aarch64 kernel. Particularly with LLM workloads. Which kernel was used in these tests?

Side note: The acronym APU was used in the title but not once defined or referenced in the article?

Somewhat tangential, but did Nvidia ever confirm if they cancelled their project to develop custom cores implementing the ARM instruction set (Project Denver, and later Carmel)?

It’s interesting to me that they’ve settled on using standard Neoverse cores, when almost everything else is custom designed and tuned for the expected workloads.

> The downside is Genoa-X has more than 1 GB of last level cache, and a single core only allocates into 96 MB of it.

I wonder if AMD could license the IBM Telum cache implementation where one core complex could offer unused cache lines to other cores, increasing overall occupancy.

Would be quite neat, even if cross-complex bandwidth and latency is not awesome, it still should be better than hitting DRAM.

> The first signs of trouble appeared when vi, a simple text editor, took more than several seconds to load.

Can it run vi?

It always made sense to have a single chip instead of 2, I just want to buy a single package with both things on the same die.

That might make things much simpler for people who write kernel, drivers and video games.

The history of CPU and GPU prevented that, it was always more profitable for CPU and GPU vendors to sell them separately.

Having 2 specialized chips makes more sense because it's flexible, but since frequencies are stagnating, having more cores make sense, and AI means massively parallel things are not only for graphics.

Smartphones are much modern in that regard. Nobody upgrades their GPU or CPU anymore, might as well have a single, soldered product that last a long time instead.

That may not be the end of building your own computer, but I just hope it will make things simpler and in a smaller package.

The article talks about the difference in the pre-fetcher between the two neoverse setups (Graviton and Grace Hopper). However isn’t the prefetcher part of the core design in neoverse? How would they differ?

This is good for datacenters, but.. NVidia stopped doing anything for consumers market.

Yeah so I also benchmarked GH200 yesterday and I am also a bit puzzled TBH:

https://github.com/mag-/gpu_benchmark

I’m torn: NVIDIA has a fucking insane braintrust of some of the most elite hackers in both software and extreme cutting edge digital logic. You do not want to meet an NVIDIA greybeard in a dark alley, they will fuck you up.

But this bullshit with Jensen signing girls’ breasts like he’s Robert Plant and telling young people to learn prompt engineering instead of C++ and generally pulling a pump and dump shamelessly while wearing a leather jacket?

Fuck that: if LLMs could write cuDNN-caliber kernels that’s how you would do it.

It’s ok in my book to live the rockstar life for the 15 minutes until someone other than Lisa Su ships an FMA unit.

The 3T cap and the forward PE and the market manipulation and the dated signature apparel are still cringe and if I had the capital and trading facility to LEAP DOOM the stock? I’d want as much as there is.

The fact that your CPU sucks ass just proves this isn’t about real competition just now.