A model that is right ninety-five percent of the time sounds impressive until you consider what the missing five percent might be. In a credit memo, the wrong number might be the leverage covenant, the change of control clause, or the borrower's actual EBITDA. Accuracy in finance is not a single number; it depends entirely on which five percent you get wrong and how visible the error is when it happens.
Why general benchmarks do not translate
Most published accuracy figures for language models come from general knowledge or reading comprehension benchmarks. Those are useful for judging a model's overall capability, but they say almost nothing about how it performs against a scanned credit agreement, a management-prepared trial balance, or an amendment referencing three prior documents by section number. Financial documents have their own failure modes: inconsistent formatting, defined terms that shift meaning between documents, and numbers that look correct but come from the wrong period.
The categories of error that matter
We evaluate along a small number of dimensions that map to how an underwriter actually reads a document.
- Extraction accuracy: did the system pull the right number, from the right table, for the right period.
- Attribution accuracy: does the citation point to the actual page and location the fact came from.
- Definitional accuracy: does the system apply the borrower's specific defined terms rather than a generic template definition.
- Consistency: does the same question asked two different ways produce the same answer.
How we build evaluation sets
A meaningful evaluation set for this domain has to be built from real documents, not synthetic examples, because the failure modes that matter show up in the messiness of actual filings and agreements.
- We assemble a held-out set of credit agreements, financial statements, and compliance certificates that were never used in any prompt design or fine-tuning work.
- A credit professional independently annotates the ground truth: the correct value, the correct source location, and the correct interpretation where judgment is required.
- The system's output is scored against that ground truth field by field, not as a single pass or fail for the whole document.
- Disagreements are reviewed manually, since some are genuine model errors and others reveal that the ground truth itself needed a second opinion.
Precision over recall, most of the time
In most consumer AI applications, a missed answer is a minor inconvenience. In a credit workflow, a missed covenant is a much larger problem than a system that says "not found" and asks a human to check. We tune extraction to prefer flagging uncertainty over guessing, because a confident wrong answer is more dangerous than a visible gap.
An underwriter should never have to wonder whether a number came from the system or was quietly guessed. The system should tell you which one it is.
What this means for how the product behaves
Every extracted fact carries a link back to its source page, and every generated statement in a memo draft is checked against the underlying extracted data before it reaches a reviewer. This does not eliminate the need for human review, and it should not. It changes what that review is for, moving the analyst from re-deriving every fact to verifying a smaller number of judgment calls, which is a faster and more defensible way to work.
Where the bar keeps moving
As the underlying models improve, our evaluation sets get harder rather than smaller, because the goal is not to pass a fixed test. It is to keep the system honest about where it is reliable and where a human still needs to look, which is the actual standard a credit committee holds any analyst to.
