AI Inference Hardware
Definition
Inference hardware is silicon built to serve trained models rather than train them. The design target has moved: the binding constraint on a frontier lab is neither budget nor floorspace but datacenter power, so the metric that matters is tokens per megawatt — throughput per watt, not peak FLOPs.
Jensen Huang put the economics plainly at Computex 2026: “If you have 1 gigawatt of power, then throughput per watt is revenue.” Nvidia repeated it at Hot Chips 2026 — “the data center is power limited today.”
Jalapeño is OpenAI’s inference ASIC, announced at Hot Chips 2026 and benchmarked in-lab by SemiAnalysis. Despite press framing, it is not specialised to OpenAI’s own models: it is a generalised inference chip that runs DeepSeek R1, Kimi-K2.5, GPT-OSS — and, as a joke, Doom, ported with Codex prompts alone.
Core Ideas
Perf/W is the design axis
Jalapeño beats Blackwell on perf/W across almost all scenarios without being tuned to any point on the curve, excelling at both low-latency and high-throughput ends. At concurrency 1 it exceeds 700 tokens/sec/user on DeepSeek R1; GPT-OSS runs around 1,400 tok/sec/user. GSM8k evals come out on par with Nvidia silicon.
Notably it does this with single-token prediction, no speculative decoding, and no prefill-decode disaggregation, while the chips it is compared against use their best multi-token-prediction configs.
The honest caveats
SemiAnalysis flags three:
- All numbers come from OpenAI. The InferenceX runs were verified in person, but not the full suite, and AgentX — the long-context, multi-turn suite that reflects real cache behaviour — was not seen. A chip that looks good on single-turn 8k1k may not stress routers, prefix cache, or offload infrastructure the same way.
- Blackwell is the wrong comparison. Jalapeño’s real competitor is Rubin, which also uses HBM4. Vera Rubin NVL72 delivers 5.4x the perf/MW of GB200 NVL72.
- The tested models are not on the open frontier — the newer and larger the model, the harder the bring-up.
Specs, and the pace
The B0 stepping delivers 13.4 PFLOPs of MXFP4 on a single reticle-sized die on TSMC N3P, against 17.5 PFLOPs dense NVFP4 for a comparable Rubin die — but at 700W TDP versus Rubin’s 900–1,150W. Jalapeño has the highest HBM bandwidth per watt and highest FLOPs per watt of any accelerator compared. HBM4 at 10Gbps pin speed gives 15.4TB/s per package, edging Rubin’s 9.6Gbps.
The timeline is the real story. CoWoS tape-out November 2025; 9 months later, with 3 months of silicon bring-up and a software stack started from zero, the results above. Rubin taped out a month earlier and has only CoreWeave engineering-sample numbers. Production ramps through 2027.
The CUDA moat argument
OpenAI’s software bring-up outran Nvidia’s, which is what a moat is supposed to prevent. Three things did it:
- Hardware/software co-design — the area where a frontier-lab ASIC team beats merchant silicon.
- Starting from scratch, counter-intuitively an advantage: clean-sheet architecture with no backwards compatibility to preserve.
- The models writing the chip. AI assistance cut SIMD area 8% and matrix-engine area 10% during design. Codex wrote functional, efficient kernels with no intervention from the kernel team — OpenAI had no internal MLA kernel implementation until they benchmarked DeepSeek, and Codex closed the gap. GPT 5.6 Sol, running on Nvidia GPUs, helped design the chip that threatens them.
Architecture and programming model
The matrix engine uses MXFP formats and a weight-stationary systolic array like a TPU, but supports smaller shapes — so no performance cliffs on awkward matmuls. 64-bit scalar cores, FP32/INT32 vector cores, tray-level redundancy, core- and channel-level yield harvesting. The design goal is eliminating KV-cache and weight movement plus fixed latencies, so small batches still approach peak FLOPs and bandwidth.
Kernels are written like assembly — hand-tuned, some ~3,000 lines, with correctness checks and a custom sanitizer, served by an engine called Teacup. Programming is via Gluon, OpenAI’s kernel language built on Triton: it keeps Triton’s SPMD model but exposes low-level abstractions, the most distinctive being the layout — a mapping between hardware resource and tensor element, formalised as Linear Layouts, an algebra that makes layout conversions provably correct and memory swizzling optimal.
To disaggregate or not
Prefill and decode stress hardware differently, so prefill-decode disaggregation (PDD) improves efficiency — at one chosen input/output ratio. OpenAI skipped it, and the reasoning generalises well beyond chips:
Production traffic does not stay at that ratio. Sequence lengths, concurrency, cache-hit rates, acceptance rates, and latency targets all move through the day.
Once devices are split into pools, an entire chip sits idle because it is in the wrong pool. Local utilisation looks better while global utilisation gets worse, and the operator must forever predict the split, hold spare capacity on both sides, and rebalance toward a moving target. Disaggregation also breaks locality: the prefill worker’s KV cache must cross the network to the decode worker, adding bandwidth, synchronisation, queueing, and a failure domain, with cost rising as input length grows.
The trade is real in both directions — moving KVs costs power and per-request latency but buys hardware utilisation. A fungible fleet shifts capacity between latency-sensitive requests and throughput batches; a fixed split strands hardware whenever the traffic mix changes.
Relationships
- Transformer (GPT Architecture) — the workload this silicon is shaped around
- AI Model Routing — the serving-layer counterpart to fleet fungibility
- Reasoning Effort Control — effort as a cost axis above the hardware one
- Observability — utilisation measured globally rather than per pool
- AI & Machine Learning — parent topic
References
- OpenAI Jalapeño: Better Than Nvidia Blackwell — SemiAnalysis, 2026-08-26