OpenAI Jalapeño: First Benchmark Results Are In

OpenAI Jalapeño: First Benchmark Results Are In

At Hot Chips 2026, OpenAI finally showed its hand. Jalapeño — the company's first custom inference chip, co-developed with Broadcom — posted its first benchmark numbers, and they are not subtle. Across three open-weight models, Jalapeño delivered 1.5 to 1.9 times more AI work per watt at peak throughput and 1.7 to 3.6 times lower end-to-end latency than Nvidia's Blackwell-based comparison systems.

The results landed on SemiAnalysis' InferenceX benchmark, a public suite that measures the full lifecycle of serving an AI request — not just raw FLOPS, but how fast tokens actually reach users under realistic load.

What Was Tested

The comparison ran on three public models: GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T (Moonshot AI's trillion-parameter model). The test workloads were selected 8k-token prompts with 1k-token responses — single-turn scenarios that span high-throughput serving to low-latency interactive use.

The comparison system was Nvidia's GB200/GB300 Blackwell configuration. Results were normalized using each accelerator's published chip power rating.

Here is the headline data:

The last row is OpenAI's "more throughput at previous TBT" figure from the results appendix — ≈53.7x on GPT-OSS, ≈104.3x on DeepSeek R1, ≈56.1x on Kimi K2.5 (OpenAI).

For DeepSeek R1 specifically, SemiAnalysis measured over 700 tokens per second per user at concurrency level 1. That is a real number from verified benchmark runs in OpenAI's lab.

The 700W Trick

Jalapeño is rated at 700 watts, but its measured sustained power during these tests stayed at or below 550 watts. That gap matters. The chip draws power budgets that look competitive on paper, but in practice it sips less than its rating while delivering the performance numbers above.

OpenAI's head of hardware Richard Ho put it plainly: "Jalapeño can serve more AI work per unit of power, while also returning responses more quickly." The combination is the key — existing systems typically force a tradeoff between throughput and latency. Jalapeño claims to break that tradeoff in a single architecture.

How It Works

Jalapeño is not a general-purpose GPU adapted for inference. It is a ground-up design for LLM serving, built around three phases of the inference process:

Prefill — processing the input prompt, which is compute-intensive. Decode — generating tokens, which is memory-bandwidth-bound. Communication — moving data between chips, which often adds latency.

The design explicitly minimizes data movement. Model state, including the KV cache used during response generation, can be placed and kept local. The system activates the right combination of compute, memory, and networking for each phase rather than treating them uniformly.

Jalapeño Architecture (simplified)
┌─────────────────────────────────────────┐
│  Model State / KV Cache — kept local    │
├────────────────┬────────────────────────┤
│  Prefill Phase │  Compute-optimized     │
│  (prompt in)   │  paths                 │
├────────────────┼────────────────────────┤
│  Decode Phase  │  Memory-bandwidth-     │
│  (token gen)   │  optimized paths       │
├────────────────┴────────────────────────┤
│  Networking — Tomahawk silicon,         │
│  explicit placement                     │
└─────────────────────────────────────────┘

Broadcom's implementation includes Tomahawk networking silicon and custom interconnect. The goal is utilization "much closer to theoretical peak performance," which is a bold claim in a space where most accelerators waste significant cycles on data movement.

The Codex Loop

One detail stands out: OpenAI used its own AI to build Jalapeño's software stack. The team brought three open-weight models to high performance on Jalapeño within two months using Codex with GPT-Astra. For selected attention and mixture-of-experts blocks, AI-generated kernel implementations ran 1.5 to 1.8 times faster than human-expert-written code.

That is not a full-model speedup — it applies to selected blocks — but it points toward a feedback loop where models help build the hardware that runs them faster.

# Example: Jalapeño's serving stack loads models with
# explicit KV-cache placement per inference phase
import jalapeno_serve as js

config = js.InferenceConfig(
    model="deepseek-r1-670b",
    kv_cache_placement="local",     # keep cache on-chip
    prefill_policy="compute_opt",
    decode_policy="bandwidth_opt",
    max_concurrent_users=64,
)

server = js.InferenceServer(config)
server.start()  # serves from local state, no cross-chip KV movement

Important Caveats

The results are real but narrow. SemiAnalysis verified the runs, but several constraints apply:

• Short workloads only. The tests used 8k/1k single-turn scenarios. Longer-context, multi-turn agent workloads — the kind that stress routing and cache management — were not tested.
• Not against Vera Rubin. The comparison was against Blackwell. Nvidia's newer Vera Rubin platform, which uses HBM4 memory (NVIDIA specs) and ships in a similar timeframe, was not part of the benchmark.
• Engineering sample. Jalapeño is still at the engineering-sample stage. Production qualification is ongoing.
• Inference only. The chip does not train models. Nvidia remains unchallenged for training workloads.

Deployment Timeline

OpenAI plans to deploy Jalapeño in its compute infrastructure by the end of 2026, though "in very small volumes." Broadcom CEO Hock Tan told CNBC that meaningful scale arrives in 2027, with full production in the first half of 2028.

The company says generation 2 is deep in development and generation 3 is taking shape. Each generation will push efficiency further.

What This Means

The benchmark results establish OpenAI as a credible silicon player, not just a software company. But the immediate impact is economic, not architectural. If Jalapeño's per-watt advantage holds in production, it lowers the cost of every token served on ChatGPT, Codex, and the API.

For the rest of the industry, the signal is that vertical integration — designing the model, the serving software, and the chip together — produces measurable gains. OpenAI will continue deploying Nvidia accelerators alongside Jalapeño. The question is not whether custom silicon replaces GPUs, but how much inference work migrates to first-party chips over the next two years. If you are interested in NVIDIA's open model ecosystem, see our coverage of Nemotron 3 Ultra.

The real test comes in 2027, when Jalapeño faces Vera Rubin on production workloads with real traffic. Until then, these numbers are promising but controlled.

Sources

• OpenAI: Jalapeño's first results show industry-leading speed and efficiency in AI inference
• NVIDIA: NVIDIA Vera Rubin NVL72 specifications