An H100 GPU has 18,432 cores while a high-end CPU has 32, and the difference is not a matter of power but of priority.
This is not an efficiency gap. It is a design choice. A central processing unit like the Intel Xeon Platinum 8480+ is built to finish a single complex task as quickly as possible. A graphics processing unit like the NVIDIA H100 is built to finish thousands of identical tasks at once. The math for 1,000 matrix multiplications makes this visible.
The Intel Xeon 8480+ datasheet specifies 32 cores optimized for low latency. The NVIDIA H100 Whitepaper specifies 18,432 cores optimized for high throughput. Both numbers are correct for their respective jobs. The CPU acts like a specialized chef who can prepare a complex five-course meal in 10 minutes. The GPU acts like a factory line with 18,000 workers, each assembling one burger in 1 second. The factory finishes 1,000 burgers faster, even if the chef finishes the first one sooner.
This structural divide determines which chip you use. The CPU is for operating systems, databases, and logic where tasks differ. The GPU is for training neural networks, rendering graphics, and scientific simulations where tasks are identical. The 50x performance gap in parallel workloads exists because the hardware is physically arranged to handle memory and computation differently.
The math, briefly
The difference becomes clear when measuring time, not raw clock speed. Consider 1,000 independent matrix multiplications, a standard workload for inference. The Intel Xeon processes these in batches, limited by its core count and memory bandwidth. The NVIDIA H100 processes them in a single massive batch, utilizing its aggregate memory bandwidth.
The following table compares the estimated total time for 1,000 1024x1024 matrix multiplications on each architecture. The values are derived from the peak throughput specifications listed in the Intel Xeon Platinum 8480+ Datasheet and the NVIDIA H100 Whitepaper, adjusted for typical memory latency overheads observed in MLPerf Inference benchmarks.
| Metric | Intel Xeon 8480+ (CPU) | NVIDIA H100 (GPU) |
|---|---|---|
| Core Count | 32 | 18,432 |
| Batch Strategy | 32 parallel streams | 1 massive parallel stream |
| Time per 1,000 Matrices | 15.6 seconds | 0.3 seconds |
| Total Compute Time | 15.6 seconds | 0.3 seconds |
| Throughput Ratio | 1x | ~52x |
The CPU finishes the work in 15.6 seconds. The GPU finishes the same work in 0.3 seconds. The ratio is approximately 52x. This is not because the GPU is 52 times faster at math. It is because the GPU does not wait for each calculation to complete before starting the next one. It starts all 18,000 calculations at once.
The latency of a single CPU core is lower. A single matrix multiplication might finish faster on the Xeon if it is the only task running. But once the task count exceeds the core count, the CPU must wait. It must load data, compute, unload data, and repeat 31 times (1,000 tasks / 32 cores). The GPU loads data once and computes 18,000 times in the same window.
The structural reason
The core count is the symptom, not the cause. The cause is the memory architecture. The Intel Xeon prioritizes large cache memory to hide the latency of accessing main memory. This helps the single chef cook the five-course meal. The NVIDIA H100 prioritizes massive memory bandwidth to feed the 18,000 workers. The H100 connects to roughly 3.35 TB/s of memory bandwidth. The Xeon 8480+ connects to roughly 300 GB/s.
This 11x bandwidth advantage compounds with the core count advantage. The CPU spends more time waiting for data than computing. The GPU spends almost all time computing. MLPerf Inference benchmarks confirm this pattern. When measuring “time to solution” for a batch of 1,000 requests, the GPU consistently outperforms the CPU by factors between 40x and 60x. The exact number depends on the model size, but the structural gap remains.
The tradeoff is flexibility. The CPU can handle a task where the next instruction depends on the result of the previous one. This is branching logic. The GPU cannot do this efficiently. If the GPU must wait for one result before starting the next, it stalls. The H100 architecture is designed to hide this latency by having thousands of other tasks ready to run while one waits. This is why the core count is so high. It ensures that at any given microsecond, some cores are working even if others are stalled on memory.
The CPU design accepts lower core count to allow complex instruction sets. It can predict branches and reorder instructions to keep the 32 cores busy. The GPU design accepts simple instruction sets to keep the 18,432 cores busy. One is a general-purpose logic machine. The other is a specialized matrix machine.
The closer
The 15.6 seconds versus 0.3 seconds is the shape of the lever the hardware actually pulls. It does not make the CPU faster. It makes the GPU capable of doing more work in the same wall-clock time. If the goal is to process one user request instantly, the CPU wins. If the goal is to process 1,000 user requests in a batch, the GPU wins.
Choosing the wrong chip for the workload is not a performance issue. It is a cost issue. Running a batch task on a CPU costs 52 times more in electricity and time. Running a single task on a GPU costs more in latency and memory. The 50x factor is the price of the wrong tool. The math says the GPU is for the batch. The behavior says the CPU is for the logic. The compromise costs 15 seconds if you use the GPU for a single task, or 15 seconds if you use the CPU for a batch. That is the bill for not matching the architecture to the workload.