AI for PDF analysis: what three Claude models got wrong

The test ran on 3 November 2026. It is a snapshot, and models change, so treat it as one day's evidence and not a ranking for good.

We made three fictional PDFs ourselves:

• a 1-page lease,
• a 1-page bank-style statement with a deliberate 100.00 error in the closing balance,
• a 1-page survey report with a table.

We wrote three questions per file, nine in total. Each question was asked in a fresh conversation, with the same system prompt the PDF tool uses and the text extracted from the PDF, the same text-extraction route the app uses.

One caveat on timing: we called the vendors' APIs directly, not inside the app, so reply times are indicative only. The raw outputs are kept in our repo.

We tested three models, all offered in the app: Claude Haiku 4.5 (5 Gems a message), Claude Sonnet 4.6 (5 Gems) and Claude Opus 4.7 (25 Gems, paid plans only).

We did not test GPT-4o mini, GPT-5, Gemini 2.5 Flash Lite or Gemini 2.5 Pro, because our test API accounts for OpenAI and Google were not usable that day. We also did not test DeepSeek V4 Flash, Grok 4.3 or GPT-5 Pro, because we had no test key.

So this page says nothing about how those models read PDFs. We do not rank them, and a comparison of Claude against GPT would need a second test.

We asked: "Does the closing balance match the opening balance plus the transactions?" The right answer is no. The transactions add up to 3,241.52, and the statement says 3,141.52.

Haiku got it wrong. It summed the transactions to 801.37 instead of 901.37 and answered "Yes, they match perfectly." A confident yes on a wrong sum is the worst kind of miss.

Sonnet got it right: "There is a discrepancy of €100.00".

Opus got it right too: "No, the closing balance does not match." It also noted that the gap equals the 100.00 ATM line.

This was the one clear failure of the test, and it was arithmetic over many lines, not finding a fact.

Renewal deadline. The lease ends on 28 February 2027 and needs notice 60 days before. Sonnet and Opus said 30 December 2026. Haiku said 29 December 2026, one day early, which cost it half a point. Opus added that the lease does not say whether notice counts when it is sent or when it is received. That is a useful catch, and the others did not make it.

Leaving early on 31 October. All three got two months' rent, 2,900.00. But Haiku and Sonnet wrote it with a pound sign, and the lease never uses one: it has no currency at all. Sonnet also cited "(p. 2.1)", a clause number written as if it were a page.

Can the landlord raise the rent during the term? The lease is silent. Haiku and Opus said the document does not address it. Sonnet said there is no provision allowing it, which is a slightly stronger reading than the text supports. Silence in a document is not the same as a ban.

All three were correct on the three report questions. The report said 468 people started the survey and 412 complete answers were kept. It found cyclists slept 0.7 hours more, and that this is an association, not a cause. Every model read the table and kept the cause-versus-association point straight.

Looking things up in a clean one-page file was easy for every model we tried.

Here is what the test supports, and no more.

• Lookups are easy. Every model found the right figures and clauses.
• Arithmetic over many lines is where the cheaper model slipped. For totals and reconciliations, use Sonnet, or recompute the sum yourself.
• Every model can add small invented details. A pound sign the lease never used, a clause number written as a page. Check each answer against the page.
• Opus costs five times Sonnet and was not more accurate here. It was more careful in wording, for example on the notice question.
• Quick questions. A fast model at 1 Gem a message is a reasonable place to start, but we did not test one this time, so we can't say how it does.

For more habits that make answers easier to check, see how to chat with a PDF. If you are weighing PDF tools and not models, see the ChatPDF alternatives comparison. If your PDF is a table, PDF to CSV with AI covers getting numbers out cleanly, and the totals warning above applies there too.

A few habits cause most bad answers, and none of them depend on the model.

• Asking a total question without the lines. If the model has to add many figures at once, ask it to list each line and show the sum. Then check the sum.
• Trusting a clean yes. A short "Yes, they match" gives you nothing to check. Ask for the numbers behind it.
• Reading silence as permission. If a document does not mention a price change, that does not mean one is banned or allowed. Ask the model to say when the file is silent.
• Uploading a scan. The tool reads text only. If you cannot select a sentence in your viewer, the model has nothing to read, and there is no OCR.
• Asking for everything in one message. Ask one narrow question at a time: one lookup, one sum, one quote. Narrow answers are easier to check.
• Adding files you do not need. You can send up to 4 files per message, each about 5.7 MB at most. More text is more for the model to mix up. Add only the files that matter to the question.

Run the same question on two models and compare. When they disagree, as Haiku and Sonnet did on the balance, you have found the line worth checking by hand.

Ask for the sum or the date arithmetic to be shown, so you can follow it. Ask for the quoted sentence, and look for it on the page. A quote you can find beats a confident paragraph.

One message costs the model's tier in Gems no matter how many files you attach, and you see the cost before you send. The PDF goes to the model provider you picked, and is sent again with each message. See all models for the full list.