AI Hardware

A flaming jalapeño character standing on NVIDIA and AMD GPUs

Know Your Enemy, Know Yourself, Part 7: A Model Trained on GPUs Builds a Better Chip Than GPUs

The era of using AI to build chips that run AI has arrived. Drawing on publicly released information about Jalapeño, OpenAI’s custom chip, this article explores what that means.

KV cache offload from GPU HBM to CXL memory, followed by memory pooling across servers and memory tiering

Memory in the AI Era, Part 5: Exploring CXL Workloads

We examine how CXL is used in real workloads through KV cache offload, memory pooling, and memory tiering.

The three datacenter-CPU powers of the Agentic AI era — Intel, AMD, NVIDIA

Know Your Enemy, Know Yourself, Part 6: The Agentic AI Era — The Revival of the CPU and the Dawn of the CPU Three Kingdoms

We analyze why the CPU became the bottleneck of inference infrastructure in Agentic AI workloads, walk through the latest datacenter CPU lineups from the three CPU vendors, and explore why the CPU has risen to prominence again in the Agentic AI era.

Where CXL sits in the memory hierarchy — a memory pool next to the CPU and accelerator coherency

Memory in the AI Era, Part 4: Understanding CXL

Next to the GPU, HBM and HBF fill the gap — but there’s another empty seat next to the CPU. We look at CXL, the new interface that fills the awkward gap between PCIe and DDR: its basic structure, device types, and the CXL product blueprints the big three memory vendors are drawing.

HBF commercialization challenges cover image

Memory in the AI Era, Part 3: The Remaining Challenges of HBF

HBF clearly has its place, but it still has gaps to fill before it can claim a spot in the memory hierarchy pyramid. We walk through the latest LLM model and inference workload trends, how Flash memory is actually used in LLM serving today, and the remaining challenges HBF has to solve.

Memory in the AI Era, Part 2: Where Does HBF Actually Fit?

Centered on SK hynix’s H³ architecture, we explore workloads that can overcome HBF’s weaknesses.

Know Your Enemy, Know Yourself Part 5: Cerebras and the Wafer-Scale Engine

This post explains Cerebras’s recent momentum, WSE-3 architecture, and the trade-offs of wafer-scale chips in a beginner-friendly way, following reports of a large OpenAI deal.

Memory hierarchy pyramid — SRAM, DRAM, HBM, HBF, SSD

Memory in the AI Era, Part 1: Understanding HBF

Why are there so many types of memory, and where does HBF fit in? From SRAM to HBF, we explore the physical principles behind the memory hierarchy and the technical architecture of HBF.

Know Your Enemy, Know Yourself, Supplement: Pallas Programming Model

Learn about Pallas programming model that enables writing custom kernels on TPU.

Know Your Enemy, Know Yourself, Part 4: Memory Capacity Bottleneck and NVIDIA ICMS

We explore the technical principles behind NVIDIA’s ICMS — a new storage tier designed to solve the KV cache capacity bottleneck in LLMs — and the Bluefield-4 DPU that manages it.