Every AI search tool finds something. They differ on whether it's true.

The moment that started this

We’ve been building a Slack bot for an investment team, a client that backs and works closely with early-stage startups. The bot answers questions about companies: mostly the team’s own portfolio and the industries around it. When someone asks about a company outside that known set, it needs to fall back to a live web search instead of just saying “no idea.” Simple enough - until we watched the first few of those fallback cases come back.

One of the search tools didn’t say “I couldn’t find this company.” It said something that read like a real company description: a plausible-sounding one-liner about what the company does, a category, even a vague value proposition. It was fabricated, fluently. And if you didn’t already know the company barely had a public footprint, you’d have believed it.

That’s the failure mode that made us stop trusting “accuracy” as the metric that mattered, and go build something else.

Why the obvious benchmark doesn’t catch this

Most comparisons of AI web-search tools measure the easy case: given a well-documented, well-indexed entity, which tool retrieves the right facts fastest and cheapest? That’s a real question, but it rewards exactly the case where these tools already agree with each other and rarely fail.

The case that actually costs you something is the opposite one: sparse-footprint entities. A company two weeks out of stealth, a vendor with no real website yet, anything the open web barely knows about. That’s precisely where tool behaviour diverges the most, and where the wrong answer is worst: a hallucinated-but-fluent answer is more dangerous than a plain “I don’t know,” because it doesn’t look wrong.

So we designed the test around that gap instead of around raw retrieval accuracy.

The benchmark: three questions, one live scenario, one instruction

We tested four providers: Gemini (a native LLM web search, the default everyone already has), and three purpose-built search APIs: Perplexity, Exa, and Parallel.ai. For every company in the test set, each provider got the same three structured questions, worded identically:

1. What is the official website domain of this company? The simplest possible lookup: if a tool can’t get this right, nothing harder is trustworthy either.
2. What does the company do? Product, customers, business model. The short factual “about” that goes into any company brief.
3. Why does this company matter? Market position, funding, differentiation. This one maps to a field the investment team’s own briefs carry. And, tellingly, it’s the field that was sitting empty for every single company in their dataset, because it demands synthesis, not lookup.

Then a fourth prompt that wasn’t a question so much as a live scenario: an open-ended “what can you tell me about this company?” It stands in for the exact moment the Slack bot’s internal data comes up empty and it has to decide what to say.

Every structured question carried the same instruction, deliberately: do not invent details. If you cannot find enough information, say exactly “UNKNOWN.” That single instruction is the whole experiment. It turns a retrieval benchmark into an honesty benchmark.

The adversarial twist

The test set was 12 real companies drawn from the client’s portfolio and network: 5 already-verified controls, and 7 with a known data gap. Ten of the twelve were ordinary, findable companies. The other two, both from the data-gap seven, were the hard cases the test was really about:

• A genuinely unfindable company: a real, operating business with essentially no public web footprint. The correct answer to every question about it is “unknown.”
• A name-collision company: a small startup that shares its name with an unrelated multi-billion-dollar hospital chain. The correct behaviour is to notice there are two candidates and answer about the right one.

Stealth and pre-launch companies are routine in this portfolio, so these weren’t contrived edge cases: they’re the normal case for this bot.

Two things kept the scoring clean. First, we wrote the ground truth before running any provider: researched by hand, without using any of the four tools under test, so the answer key couldn’t be contaminated by the things being graded. Second, all 192 answers (12 companies × 4 prompts × 4 providers) were scored blind: reviewers saw the answer text with the provider labels stripped, and graded each one on whether it found anything at all, whether it cited a source, and (the primary signal) whether it named the right entity with right facts and no fabrication.

What we found

Provider	Coverage	Right entity, right facts, no fabrication	Cost / query	Latency
Exa	97.9%	81.2%	$0.005	1.9s
Parallel.ai	95.8%	70.8%	$0.010	38.5s
Perplexity	87.5%	68.8%	$0.015	7.6s
Gemini	85.4%	52.1%	$0.037	7.3s

A few things jump out of that table. The default option (native LLM web search) was the least trustworthy of the four - 52.1%, barely better than a coin flip on whether an answer held up - and the most expensive per query. And the spread on the honesty axis (52% to 81%) is far wider than the spread on coverage (85% to 98%): every tool surfaces an answer; far fewer of those answers hold up.

But the table undersells where the gap actually lives. Broken out by question, “why does this company matter” was the worst-performing question for every provider (42-67%). That’s the highest-value field, the one that was empty across the entire portfolio. The thing most worth automating is the thing every tool is least trustworthy on.

And the adversarial cases are where the providers really separated:

• The unfindable company. Exa, Perplexity, and Parallel.ai all correctly answered “UNKNOWN” on the structured questions. Exa went further: on the open-ended prompt it explicitly named two similarly-named real companies, and stated that neither matched, actively recognising the trap. Gemini fell straight into it, confidently describing a real healthcare-AI company that simply wasn’t the company we asked about. A fabricated answer built from someone else’s facts.
• The name collision. Across all four providers, 11 of the 12 answers about this company described the wrong business: the hospital chain, or a generic policy concept. Exa was the only provider to name both companies, distinguish them, and surface the actual startup’s domain.

The best-performing tool won by being honest about not knowing, and, when it did answer, backing the answer with citations instead of asserting it flatly. That property has a name: explainability. In this test it was the entire difference between a tool you can put behind a fallback path and one you can’t.

Here’s the part we didn’t expect: we went in assuming calibrated honesty would cost extra. It didn’t. The most honest tool was also the cheapest ($0.005 a query, a seventh of the default’s cost) and the fastest (1.9 seconds, four times faster than the next usable option). But even if it hadn’t been - a cheap tool that hallucinates on your hardest queries isn’t actually cheap.

What we shipped

This wasn’t a spectator benchmark; a production decision hung on it. We built the enrichment path on Exa’s answer API, with per-query cost monitoring wired in from day one. It now backs both the bulk fill of those empty brief fields and the Slack bot’s live fallback. Perplexity stays on the bench as the documented backup, rather than a second production integration nobody needed yet.

One caveat: 12 companies and 192 scored answers is a directional test, not a statistically powered one. That was a deliberate scope decision: the goal was to catch a behavioural difference at the edge of the tools’ knowledge, and the difference turned out to be large enough that more samples would have been decoration.

The takeaway

If you’re evaluating AI search tools for anything where a confident wrong answer is worse than a slow right one (support bots, research assistants, anything customer- or decision-facing), don’t just benchmark on the easy, well-documented cases. Build your test set adversarially:

1. Include entities you know are sparse, ambiguous, or nearly undiscoverable: one unfindable company and one name collision gave us more signal than the ten findable ones combined.
2. Instruct every tool identically: don’t invent, say “unknown.”
3. Write the ground truth first, without the tools under test, and score blind.
4. Treat “I don’t know” as a correct answer when it’s true, and reward the tool that gives it.

We ran this test in a week, with 12 companies and a hand-written answer key. It changed which tool we put into production. Before you wire an AI search tool into anything that answers for you, it’s worth a week of yours.