NEEDLE: A Live Search Benchmark AI Agents Can't Cheat

Here's the problem nobody fixed until now: how do you benchmark a web search API when the thing being tested can read the answer key? A search agent comes with a fetch tool. If the gold labels live in a public dataset, the agent can download them mid-evaluation and skip the retrieval step entirely. Or worse — the answer is already baked into the model's parametric memory, so a correct response doesn't prove the search worked at all.

That's the trap Keenable is trying to design around with NEEDLE, a live, open-source search benchmark open-sourced on GitHub. Instead of freezing one fixed set of questions, it rebuilds its query set from fresh public sources — news queries regenerated hourly from RSS feeds and Google Trends, the other verticals daily. There is no stable test set left behind to overfit.

Why Static Benchmarks Lie

The overfitting problem isn't hypothetical. Epoch AI's SimpleQA Verified leaderboard got flagged as likely contaminated after Qwen3-Max-Instruct topped it. Shuffling the answer options in MMLU drops accuracy for every model tested. And most damningly, search agents will simply pull a benchmark's ground-truth labels straight off HuggingFace — the fetch tool makes it trivial: one wget and one grep, and the "search task" is done.

Even when nothing leaks at test time, memorization corrupts the measurement. BrowseComp builds questions where the answer is hard to find but easy to verify. A model that has already memorized the answer skips the "hard to find" part entirely — it knows what to search for and jumps straight to the answer.

Live benchmarks sidestep both failure modes. If the questions are newer than the model and keep changing, there's nothing to memorize and nothing to leak. It's already standard practice elsewhere — LiveBench refreshes monthly, LiveCodeBench only scores problems newer than a model's training cutoff, SWE-bench-Live rebuilds from fresh GitHub issues. Search just never got the same treatment until NEEDLE.

What NEEDLE Measures

NEEDLE stands for News, Everyday, Expert, Deep-tail, and Legal Evaluation — five verticals, each modeling a different agent intent:

• News projects the newest item from ~124 curated RSS feeds and Google Trends into a keyword query.
• Finance asks registry facts from Wikidata and GLEIF, plus single-quarter 10-Q figures from SEC XBRL.
• Scholar turns one paper into four query styles — a degraded title, a full-text-only detail, a natural-language clue, and a hedged tip-of-the-tongue description.
• Deep-tail samples rare-word queries from public agent-trajectory releases like DeepResearchGym, OpenResearcher, and LRAT.
• Legal pulls recent CourtListener opinions across 14 federal courts and eCFR sections.

Scoring splits the same way. News and deep-tail have no single correct result, so an LLM judge rates each result 0–4 and reports nDCG@5 with a duplicate-URL penalty. Finance reports answer-recall@5: does the fact reach the agent inside a top-5 snippet? Scholar and legal are known-item tasks scored by identifier match.

The Protocol Keeps Comparisons Honest

Every engine gets the same query text under one protocol. The runner issues one call at a time, so latency percentiles are comparable. Judging happens on the engine's own ranking, titles, and snippets — pages are never fetched and results are never re-ranked. Evidence is clipped to 2,000 characters for everyone, and the judge never sees the engine name.

There are 15 search APIs running against this in the current cycle, including Google (via Serper), Bing, Brave, and AI search startups like Exa, Tavily, and Parallel — plus Keenable itself.

Where the Field Actually Stands

The interesting number isn't any single leaderboard position — it's the ultimate ceiling. For each query, NEEDLE pools the results returned by every engine into a synthetic oracle engine, then orders that combined set by relevance. That creates an empirical upper bound based on what the entire field could retrieve. The gap to ultimate is an upper bound on agentic search quality today.

Published 7-day means for the window ending 2026-08-28:

• Finance is close to solved: Exa 0.910, Keenable 0.872, Perplexity 0.871, Google 0.847, against an ultimate of 0.965.
• Scholar spreads out — Keenable 0.774 down to Tavily 0.310 against a 0.869 ceiling, because title queries are answerable from metadata while body queries are not.
• Deep-tail is hardest and closest to real agent traffic: Exa leads at 0.557 of ultimate, Keenable 0.470, Bing just 0.199.

Latency matters here because agents call search dozens of times per task. Same window: Keenable-realtime at 193 ms p50 / 284 ms p95, versus Exa at 1,876 / 2,955 and Bing at 2,767 / 9,381.

Why This Matters for the Rest of Us

NEEDLE is an evaluation harness you can actually run — a Python CLI installed with uv sync, driven by a generate and run pair per benchmark:

# Install with uv
uv sync

# Generate a fresh news query set
uv run needle news generate --out queries.jsonl

# Evaluate all configured engines on those queries
uv run needle news run --queries queries.jsonl --out report.json

It needs an OpenRouter key for judging and one API key per engine tested, and it runs on a laptop or in CI. All runs execute in public GitHub Actions, with per-run artifacts shipped to a Hugging Face dataset.

The deeper point for developers: the gap between delivered and achievable quality widens exactly as queries approach how agents actually search. On rare-entity queries pulled from real agent logs, even the top engine reaches only 0.557 of what the pooled field could find. That gap is the reason search infrastructure — the layer every agent leans on constantly — is becoming one of the most valuable pieces of the AI stack. Whoever's index adapts fastest to how agents really query will own it.

It's the same pattern as other open evaluation efforts: a public, reproducible benchmark keeps a messy field honest. NEEDLE just adds the twist that the test itself won't sit still long enough to be gamed.

Sources:

• Keenable — NEEDLE: The benchmark your search engine can't memorize
• NEEDLE live dashboard
• GitHub — keenableai/needle