We gave nine LLMs the same PDF. Two answered without reading it.

We build a product that sends the same question to several LLMs at once, so we hit an unglamorous problem early: when you attach a PDF, not every model that accepts it actually reads it.

Some refuse loudly. Some refuse quietly. And some answer anyway.

Here is what we measured, the exact error strings, and a script to reproduce it.

Method

• Route: OpenRouter, chat/completions, PDF as a file part with the file-parser plugin, pdf.engine: "native". This matters: results going direct to each provider’s own API may differ.
• Documents: a synthetic 27-page tender-style PDF (generated text, so we could verify answers), and a 1-page version for the capability check.
• Control question: “What is the subject of the contract?” — the answer appears on page one and nowhere else. A model that read the file gets it right; a model that didn’t produces a plausible paragraph about “the provision of services”.
• Date: 10–11 August 2026. Model behaviour changes; treat this as a snapshot, not a law of nature.
• Account setting that matters: our OpenRouter account enforces a data policy that excludes providers which don’t meet it. That alone removes some models entirely (see below).

Results

The last two are the finding. No error, no warning, a well-written paragraph describing what tender documents usually contain — for a document they never opened. If that answer reaches a user on its own, it is indistinguishable from a correct one.

Three things we didn’t expect

1. Each PDF page counts as an image, and there’s a cap.

Two real tender documents totalling 51 pages produced:

Too many images in request: 51, maximum allowed: 50.

Not “I read the first 50” — the entire request is rejected. One page over the line and there is no answer at all. We now count pages before choosing which models to call.

2. “Supports files” in the catalogue doesn’t mean you can use it.

x-ai/grok-4.3 declares file in its input_modalities on OpenRouter. Every one of its endpoints returns 404 No endpoints found that support file input. We had it in our file-capable list for days on the strength of that declaration, silently losing one juror on every document question.

We’d already been bitten by the same class of bug: a model listed as video-capable returned 404 on every video request for five days, because it was only served by a provider our data policy excludes. Declared capability is not available capability.

3. Reasoning models can spend the whole budget thinking.

With max_tokens: 1800, several models burned the entire budget on internal reasoning and returned an empty string — billed in full. google/gemini-3.1-pro-preview spent 1,152 tokens reasoning and returned 29 truncated words at $0.0145, the most expensive call in the jury and the one that delivered least.

Turning reasoning off (reasoning: {enabled: false}) fixed it for most, and made them 3–10× cheaper. Gemini refuses: Reasoning is mandatory for this endpoint and cannot be disabled — for that one, effort: minimal plus a larger budget is the only path.

Cost, since nobody mentions it

That 27-page PDF becomes 31,943 input tokens, and you pay it once per model:

An 8× spread for the same document. If you fan out to three models, you pay it three times.

Reproducing it

import asyncio, base64, httpx, os

KEY = os.environ["OPENROUTER_API_KEY"]
PDF = base64.b64encode(open("test.pdf", "rb").read()).decode()
MODELS = ["openai/gpt-4o", "google/gemini-3.1-pro-preview",
          "deepseek/deepseek-v4-pro", "z-ai/glm-4.7",
          "x-ai/grok-4.3", "minimax/minimax-m3"]

async def ask(http, model):
    body = {
        "model": model, "max_tokens": 500,
        "messages": [{"role": "user", "content": [
            {"type": "text", "text": "What is the subject of the contract? One sentence."},
            {"type": "file", "file": {"filename": "test.pdf",
                                      "file_data": f"data:application/pdf;base64,{PDF}"}},
        ]}],
        "plugins": [{"id": "file-parser", "pdf": {"engine": "native"}}],
        # Without this, reasoning models can spend the whole budget and return "".
        "reasoning": {"enabled": False},
    }
    r = await http.post("https://openrouter.ai/api/v1/chat/completions",
                        json=body, headers={"Authorization": f"Bearer {KEY}"}, timeout=120)
    d = r.json()
    if "choices" not in d:
        return f"{model}: {r.status_code} {d.get('error', {}).get('message', '')[:80]}"
    return f"{model}: {(d['choices'][0]['message']['content'] or '').strip()[:80]}"

async def main():
    async with httpx.AsyncClient() as http:
        for line in await asyncio.gather(*[ask(http, m) for m in MODELS]):
            print(line)

asyncio.run(main())

Put a fact on page one that no model could guess, and check whether the answer contains it. That’s the whole test.

What we changed because of this

• The list of models that can read a file is built by testing, not by reading capability flags. A model enters it only after answering a content question correctly.
• We count pages and don’t call a model that will reject the document. Better to leave it out than to bill a jury that came back one member short.
• Reasoning is off by default across the chain, with an automatic retry at effort: minimal for endpoints that refuse to disable it.
• The failure reason is logged and shown. Previously a juror could fail and the reason travelled to the client and vanished — which is why the 50-image cap took us days to find.

The practical takeaway

If you’re building anything that hands documents to an LLM: verify the read, don’t trust the status code. A 200 means the request was accepted, not that your file was opened.

The ten-second version, which works even from a chat window: ask about a detail that only exists inside the document. If it answers in generalities, it never saw your file.

We’re The Judge: several AIs answer the same question independently and one fixed judge rules on it with a confidence level. The numbers above come from making that work with real documents. The Spanish-language version of this, written for non-developers, is here.