Inference chip claims are cheap; benchmark slides are not. On Tuesday 25 August 2026, at the Hot Chips conference in Silicon Valley, OpenAI put the first real numbers behind Jalapeño — the custom silicon it unveiled with Broadcom in June — and said the part returns answers faster, and burns less power doing it, than the best accelerators money can currently buy.
The headline from the talk was blunt. Measured on SemiAnalysis’s public InferenceX benchmark against an Nvidia Blackwell system, OpenAI’s inference chip handled 1.5 to 1.9 times more work per watt while cutting end-to-end latency by 1.7 to 3.6 times. On highly interactive workloads — the chatty, low-latency traffic that ChatGPT actually generates — Jalapeño ran 2.1 to 4.1 times faster than the systems it was measured against.
“The bottom line is that the results show a very, very significant performance advance over state of the art,” said Richard Ho, OpenAI’s head of hardware. That is a large claim about a part nobody outside OpenAI and Broadcom has been able to test, and it lands in the middle of the most expensive infrastructure race in technology. This article breaks down what was actually measured, how the inference chip achieves it, what OpenAI is still not saying, and what a credible custom accelerator means for anyone buying AI capacity.
Table of contents
- OpenAI Puts Its Inference Chip on the Benchmark Board
- The Inference Chip Numbers OpenAI Reported
- Why the Jalapeño Inference Chip Is Fast
- An Inference Chip Designed in Nine Months
- Jalapeño Against Nvidia Blackwell
- What OpenAI Has Not Disclosed About the Inference Chip
- When the Inference Chip Actually Ships
- What the Custom Inference Chip Race Means for Your Business
- Inference Chip FAQ: Your Questions Answered
- References
OpenAI Puts Its Inference Chip on the Benchmark Board
June’s announcement was a press release with adjectives. August’s Hot Chips session was the first time OpenAI showed its working.
From “significantly better” to a number
When OpenAI and Broadcom unveiled Jalapeño on 24 June 2026, the performance language was deliberately vague: the inference chip would deliver performance per watt “significantly better than current state-of-the-art,” with a detailed technical report promised later. Richard Ho said only that early testing suggested Jalapeño would “efficiently execute our most important workloads close to the hardware’s theoretical limits.”
Two months on, that report arrived. The Hot Chips talk is the first public data set for an inference chip that OpenAI intends to run a meaningful share of its own traffic on.
Why the benchmark choice matters
OpenAI did not invent a metric for itself. It ran InferenceX, SemiAnalysis’s open benchmark for AI inference, across three publicly available models: GPT-OSS 120B, DeepSeek R1 670B — a reasoning model trained with reinforcement learning — and Kimi K2.5 1T. Using open models on a third-party benchmark is a meaningfully harder test than a vendor-authored demo, because anyone with Blackwell hardware can reproduce the baseline side of the comparison.
What nobody can yet reproduce is the Jalapeño side. The inference chip has not been released for external testing, so every figure below is OpenAI’s own measurement of OpenAI’s own silicon.
| Metric | Reported range | What it measures |
|---|---|---|
| Work per watt | 1.5x to 1.9x | Throughput delivered per kilowatt drawn — the number that sets your cost per token |
| End-to-end latency | 1.7x to 3.6x lower | Total time from request to completed answer |
| Highly interactive workloads | 2.1x to 4.1x faster | Short-turnaround chat and agent traffic where users feel every millisecond |
| Models tested | 3 | GPT-OSS 120B, DeepSeek R1 670B, Kimi K2.5 1T |
The Inference Chip Numbers OpenAI Reported
Three multipliers came out of the session, and they are not interchangeable. Each one answers a different question about what an inference chip is for.
Tokens per user, and throughput per kilowatt
OpenAI’s framing was that Jalapeño registered both more tokens per user and more throughput per kilowatt than currently available state-of-the-art inference processors. Those two goals normally fight each other. You can push throughput up by batching aggressively, but every user in that batch waits longer. You can cut latency by shrinking batches, but then the silicon idles and your cost per token climbs.
The claim worth paying attention to is that Jalapeño moved both at once. An inference chip that improves throughput and latency together is not a tuning win; it is an architecture win.
The interactive number is the important one
The 2.1x to 4.1x figure on highly interactive workloads is the one with commercial teeth. Batch inference — summarising a document store overnight, running an evaluation sweep — tolerates latency. Interactive inference does not. Every agent step, every ChatGPT turn, every code completion is a user staring at a cursor.
That is precisely where a general-purpose GPU is least efficient, and precisely the traffic OpenAI runs most of. An inference chip tuned for that shape of workload attacks OpenAI’s single largest operating cost.
Here is how the best-case multipliers OpenAI reported compare with the baseline system, on a common scale.
Read the bottom of every range
Note the width of those ranges. A metric quoted as “1.7 to 3.6 times” means at least one of the three tested models saw less than half the headline benefit. Buyers should read 1.5x work per watt and 1.7x latency as the numbers to plan against, and treat the top of each range as the best case on the friendliest model.
That is not a criticism of the inference chip. It is how every accelerator behaves — performance is a property of the model, the sequence length and the serving configuration, not of the silicon alone.
Why the Jalapeño Inference Chip Is Fast
OpenAI’s architectural pitch is narrow on purpose. Jalapeño does not train models. It only serves them, and it was designed around the specific bottlenecks that make serving expensive.
Killing the prefill and communication stalls
Jalapeño is built to minimise delays during the prefill and communication phases of inference, which OpenAI identifies as the usual choke points. Prefill is the pass where the model reads your prompt before it can emit a single token; communication is the traffic between accelerators when a model is too large to sit on one device. Neither produces output, and both make users wait.
An inference chip that attacks those two phases specifically is optimising the part of the request users actually experience as lag.
Putting model state where it belongs
The second lever is explicit placement of model state and the KV cache to reduce data movement. In transformer serving, the key-value cache is the running memory of the conversation, and shuttling it between compute and off-chip memory burns time and power for no arithmetic.
OpenAI’s design balances compute, memory and networking so that data moves as little as possible. This is the least glamorous idea in the whole inference chip and probably the most consequential — most of the energy in modern accelerators is spent moving numbers, not multiplying them.
A blank-slate design, not a GPU derivative
OpenAI describes Jalapeño as a blank-slate design for modern large language model inference, combining the throughput of leading AI accelerators with latency closer to the fastest specialised inference systems. Broadcom contributed the silicon implementation and its Tomahawk Ethernet networking; Celestica handled system integration.
| Design decision | Bottleneck it targets | Effect users notice |
|---|---|---|
| Prefill-phase optimisation | Time spent reading the prompt before any token appears | Shorter wait before the answer starts |
| Communication-phase optimisation | Accelerator-to-accelerator traffic on sharded models | Less stalling on large models |
| Explicit KV cache placement | Repeated movement of conversation state | Steadier speed on long conversations |
| Inference-only scope | Silicon area spent on training features | More of the die doing useful serving work |
The thermal envelope is a strategy, not an accident
One reported design target is a thermal design power of 700W to 800W rather than pushing past a kilowatt. That keeps the inference chip inside air-cooled limits instead of requiring liquid cooling. For anyone who has costed a data centre retrofit, that constraint is worth more than a benchmark multiplier — it decides which existing halls the hardware can be racked in at all.
An Inference Chip Designed in Nine Months
The performance claim is contested until independent testing exists. The schedule claim is arguably the more remarkable story, and it is one Broadcom can corroborate.
Concept to tape-out in three quarters
Jalapeño went from initial design to manufacturing tape-out in nine months. For a reticle-sized ASIC on an advanced node, that is close to unprecedented. Architecture, register-transfer level design, verification, timing closure, physical design and packaging qualification each normally consume a quarter or more; eighteen to twenty-four months from concept to tape-out is considered a fast schedule for a frontier-class accelerator.
The models helped build the inference chip
OpenAI says its own models accelerated parts of the design and optimisation work. Greg Brockman told CNBC that the chips were designed end to end in nine months with help from those models, adding that “the degree to which our models have been able to accelerate it was very surprising to us.”
This is the recursive loop the industry has been predicting out loud for two years, showing up in a shipping product: a company using its language models to compress the design cycle of the inference chip that will then serve those same models more cheaply. Whether the speed-up came mainly from verification, code generation or physical design is not broken out.
Engineering samples are already doing real work
OpenAI has said engineering samples were running production workloads, including GPT-5.3-Codex-Spark, at target frequency and power levels. Silicon hitting its frequency and power targets on first engineering samples is the strongest available signal that an inference chip programme is not in trouble.
Jalapeño Against Nvidia Blackwell
Every number in the Hot Chips deck is relative to somebody else’s hardware, so it is worth being precise about what the comparison does and does not establish.
What the comparison establishes
Broadcom president and chief executive Hock Tan has described Jalapeño as comparable to Nvidia’s Blackwell chips and Alphabet’s tensor processing units, and has pointed to roughly 50% better cost efficiency against standard AI GPUs. OpenAI’s InferenceX results are consistent with that framing on inference workloads specifically.
Richard Simon, chief technology officer at T-Systems International, summarised the enterprise read: the inference chip promises “cost efficiency per inference token and better performance per watt, reduced latency and faster responses.”
What it does not establish
It does not establish that Jalapeño beats Blackwell at anything other than inference, and OpenAI has not claimed it does. Jalapeño cannot train models. Nvidia’s position in training is untouched by these results, and training demand is what has driven most GPU spending to date.
Nor does it establish a price-performance figure you could put in a budget. Cost per token depends on silicon cost, memory cost, rack density, utilisation and power price — none of which OpenAI disclosed.
| Factor | Jalapeño | Blackwell-class GPU |
|---|---|---|
| Workload scope | Inference only | Training and inference |
| Availability to buyers | Not sold externally; OpenAI infrastructure | Purchasable and rentable worldwide |
| Independent benchmarks | None yet published | Extensive third-party results |
| Software ecosystem | Internal stack | Mature, broadly supported toolchain |
| Reported cooling requirement | Targeted at 700W-800W, air-cooled | Commonly liquid-cooled at the top bins |
The margin story behind the silicon
There is a financial motive underneath the engineering one. Alexander Harrowell, senior principal analyst at Omdia, notes that Nvidia’s gross margins run around 75% to 78%, while ASIC outsourcers such as Broadcom operate nearer 30% to 35%. Building a custom inference chip lets a hyperscale buyer capture a chunk of that spread.
“AI as an application has been so demanding that it’s forced the industry to switch strategy to customisation,” Harrowell said. Quentin Reul, director of global AI strategy at expert.ai, makes the same point more directly: custom silicon reduces costs by bypassing third-party margins.
What OpenAI Has Not Disclosed About the Inference Chip
A responsible reading of the Hot Chips announcement means being equally clear about the gaps, and they are wide.
The specification sheet is mostly blank
OpenAI has confirmed that Jalapeño is a reticle-sized ASIC — the largest die a lithography tool can expose in a single shot. Almost everything else a hardware buyer would ask for is absent: no confirmed foundry or process node, no published memory capacity or bandwidth, no die photograph, no yield expectations and no volume ramp. Secondary write-ups have offered specifics on node and memory configuration, but OpenAI and Broadcom have not confirmed them, and they should be treated as unverified.
No independent testing, no external access
The finished part has not been released for external testing, so there is no independent corroboration of its efficacy. There is also no developer access model, no price, and no indication that Jalapeño will ever be sold to anyone. It is infrastructure OpenAI built for OpenAI.
| Detail | Status |
|---|---|
| InferenceX results on three open models | Disclosed by OpenAI at Hot Chips |
| Reticle-sized ASIC, inference-only | Confirmed |
| Nine-month concept-to-tape-out | Confirmed by OpenAI and Broadcom |
| Foundry, process node, memory configuration | Not confirmed |
| Independent third-party benchmarks | None |
| External availability or pricing | Not offered |
Why the gaps are normal, and still matter
None of this is unusual for first-party silicon. Google published very little about early tensor processing units, and hyperscalers rarely open their internal accelerators to reviewers. But it does mean the honest description of the current state of play is that OpenAI has published a strong self-reported result on a credible third-party benchmark — not that an independent lab has confirmed the inference chip beats Nvidia.
When the Inference Chip Actually Ships
The gap between a benchmark slide and a rack that serves your requests is measured in quarters.
Small volumes this year, scale next year
Richard Ho estimated that Jalapeño would deploy at the end of 2026 “in very small volumes,” with more significant deployment arriving in 2027. OpenAI has separately described gigawatt-scale deployment alongside Microsoft and other partners beginning at the end of 2026, and has framed Jalapeño as the first step in a multi-generation compute platform rather than a one-off part.
Analysts covering the segment expect custom ASICs to start taking substantial market share during 2027, which lines up with OpenAI’s own schedule.
| When | Milestone |
|---|---|
| 24 June 2026 | OpenAI and Broadcom unveil Jalapeño; no performance figures released |
| Mid 2026 | Engineering samples run production workloads at target frequency and power |
| 25 August 2026 | First InferenceX benchmark results presented at Hot Chips |
| End of 2026 | First deployments, in what Richard Ho called very small volumes |
| 2027 | Significant deployment; analysts expect ASICs to take real market share |
What “very small volumes” really signals
That phrase is doing a lot of work. It means the 2026 milestone is a proving deployment, not a capacity event, and that anyone modelling AI infrastructure supply should keep Nvidia at the centre of their 2026 assumptions. The inference chip changes OpenAI’s cost curve in 2027 and beyond, not this year.
What the Custom Inference Chip Race Means for Your Business
Almost no organisation reading this will ever buy a Jalapeño. The effects still reach you, through prices, latency and roadmaps.
Cheaper, faster inference eventually reaches the API
Greg Brockman framed the programme as part of “our long-term full-stack infrastructure strategy to make compute more abundant, resulting in AI which is faster, more reliable, more affordable.” If the inference chip delivers even the low end of its reported ranges at scale, OpenAI gains room to cut prices, raise rate limits or serve larger models at the same cost.
That matters most for latency-sensitive products. Voice interfaces, coding assistants and multi-step agents are the applications that stall at today’s response times, and they are exactly what this hardware is tuned for.
Do not re-architect around unshipped silicon
The practical advice is conservative. Build for portability rather than betting on one vendor’s roadmap: keep prompts, evaluation suites and orchestration layers independent of the model provider so you can move when the economics shift. If you are shaping that layer now, our guidance on AI strategy covers how to keep the option open without stalling delivery.
Watch the second-order effects on capacity
The most immediate impact of custom silicon is not price — it is supply. Every hyperscaler serving its own traffic on its own accelerators frees merchant GPU supply for everyone else. Combined with machine learning workloads that increasingly run on specialised parts, that is a genuine loosening of a market that has been capacity-constrained for three years.
Data centre operators face the mirror image of the same shift. A part designed to stay in a 700W to 800W envelope is a signal that at least one major buyer expects to deploy in air-cooled halls, which changes retrofit priorities for anyone planning data centre operations upgrades. Our AI models and tools hub tracks how these releases land for buyers.
The questions to ask your vendors
When your AI provider next quotes you a price or a latency target, the useful questions are simple. What hardware is my traffic served on today, and what will it be served on in twelve months? Are latency commitments contractual or aspirational? And if the inference chip underneath changes, does anything about my integration change with it? Vendors that can answer those clearly are ones you can plan around.
Inference Chip FAQ: Your Questions Answered
What is the Jalapeño inference chip?
Jalapeño is OpenAI’s first custom AI accelerator, developed with Broadcom and system integrator Celestica, and announced on 24 June 2026. It is an inference chip: it serves trained models rather than training them, and it is designed specifically for large language model workloads at scale.
How much faster is it, exactly?
On SemiAnalysis’s InferenceX benchmark across GPT-OSS 120B, DeepSeek R1 670B and Kimi K2.5 1T, OpenAI reported 1.5x to 1.9x more work per watt and 1.7x to 3.6x lower end-to-end latency than an Nvidia Blackwell system, rising to 2.1x to 4.1x faster on highly interactive workloads. These are OpenAI’s own measurements.
Has anyone independently verified the claims?
No. The part has not been released for external testing and there is no independent corroboration. The benchmark itself is public and third-party, which makes the baseline reproducible, but the Jalapeño results are not.
Can I buy or rent a Jalapeño?
Not currently. OpenAI has announced no external availability, pricing or developer access model for the inference chip. It is being built for OpenAI’s own infrastructure, deployed with Microsoft and other data centre partners.
Does this threaten Nvidia?
Partly, and only in one segment. Jalapeño does not train models, so Nvidia’s training business is unaffected. Analysts do expect custom ASICs to take meaningful inference market share from 2027, and margin pressure in inference is the realistic risk rather than displacement.
When will it affect what I pay for AI?
Not in 2026. Deployment begins at the end of this year in very small volumes, with meaningful scale in 2027. Any pass-through to API pricing or rate limits would follow that, and only if the reported efficiency gains hold at production scale.
References
OpenAI and Broadcom unveil LLM-optimized inference chip — OpenAI
OpenAI and Broadcom Unveil LLM-Optimized Intelligence Processor — Broadcom
OpenAI built a chip in nine months. Then it let AI rewrite the code. — The New Stack
The real heat behind OpenAI’s new Jalapeño chip — InformationWeek
Broadcom and OpenAI unveil custom-built Jalapeño inference processor — Tom’s Hardware
OpenAI and Broadcom reveal Jalapeno, first AI chip in partnership — CNBC
OpenAI, Broadcom debut custom Jalapeño chip for AI inference — SiliconANGLE
InferenceX open-source agentic inference benchmark — SemiAnalysis
Jalapeño in Nine Months: Did AI Just Break Chip Design Timelines? — Futurum Group
More AI coverage: explore Progressive Robot's AI Models, Tools & Releases hub — hands-on reviews, setup guides and benchmarks in one place.