What happens when you give an AI a PDF

You drag in a sixty-page PDF, type “summarise this”, and two seconds later you have a flawless answer. It feels like it read the thing.

Sometimes it did. Sometimes it never saw a single page.

We know because we measured it: we built a test tender document and handed it, one by one, to the models we say we use. What came back isn’t in anyone’s documentation.

A PDF doesn’t arrive as a PDF

The first thing worth understanding is that no model “opens” your document the way your computer does. There are two ways it can reach the model, and they don’t produce the same result:

As text. Something — the software sitting between you and the model — pulls the letters out of the PDF and sends only those. It’s fast and cheap. The problem is what gets left behind: an architecture diagram, a table that’s actually an image, a floor plan, an org chart. None of those are letters, so they vanish. The AI then answers confidently about a document it has only half seen.

As pages. The document is turned into images, one per page, and the model looks at them. It sees the diagrams, the tables, the layout. It’s faithful, and it’s considerably more expensive.

You are almost never told which one you’re getting. And it decides what the model is capable of answering.

The limit nobody warns you about

Here’s the finding that surprised us most. When the document travels as pages, each page counts as an image — and providers cap how many images one request may contain.

OpenAI’s cap is 50. With two tender documents adding up to 51 pages, the answer was literally this:

Too many images in request: 51, maximum allowed: 50.

No graceful warning, no “I read the first 50”: the whole request is rejected. One page over and there is no answer at all.

Other models have different limits and, more to the point, don’t publish them. One that handled a twenty-seven-page document without blinking returned a curt “failed to convert request” on the real documents. None of those limits appear on its spec sheet.

The worst case: the one that answers without reading

The failures above are annoying but honest: they tell you they couldn’t. There is something worse.

We gave the same PDF to several models with a question whose answer was on page one: what is the subject of the contract?

• Some read it and answered correctly.
• Some rejected the file with a clear error.
• And two accepted the request, ignored the document and answered anyway, with a polished paragraph about “the provision of services” that they had invented from scratch.

No error. No warning. An answer with the tone of something that had read every page.

If that answer reaches you on its own, you have no way of telling it apart from a good one. It sounds identical.

How to check, in ten seconds

Nothing technical required. Before trusting a summary, ask about a detail that only exists in the document and that you can verify at a glance:

• “What exactly does clause 4.2 say?”
• “What figure appears in the table on page 12?”
• “How many sections does chapter 3 have?”

If it gets it right, it read the document. If you get generalities back — “clause 4.2 sets out the technical requirements of the service” — it hasn’t seen it: it’s describing what documents like that usually contain.

And a second check if the document has images: ask about the diagram. “What’s in the chart on page 8?” If only the text made it through, that’s where it falls apart.

What we do about it

The Judge convenes several AIs for the same question, so this problem shows up multiplied by the number of jurors. These are the three rules we ended up with, and we’re sharing them because anyone can apply them on their own:

The document travels whole, images included. We tried extracting just the text: far cheaper, and it opened the jury up to more models. We reverted it. A tender document is full of diagrams and tables-as-images, and that is documentation that needs analysing — a jury that can’t see it is judging a different document.

We don’t convene anyone who’s going to reject it. If the document has more pages than a model accepts, that model doesn’t join the jury. Better not to call it than to call it so it can fail and leave you one juror short.

The list of who can read a file is built by testing, not by reading spec sheets. A model declaring that it accepts documents doesn’t mean it accepts them, and a call that returns no error doesn’t mean it read anything. The only valid test is asking about the content and checking the answer.

What to take away

When you hand an important document to an AI — a contract, a tender, a report — and it answers quickly and well, there are three possibilities: it read the whole thing, it read only the letters and missed the graphics, or it never opened it and is telling you what documents like that usually say.

All three sound exactly the same. A ten-second control question is what separates them.

And if there’s an expensive decision behind that document, the check isn’t optional.