Explainable AI in Finance: 5 Tests for a Glass Box

Explainable AI in finance is now a claim every vendor makes. In April 2026 BlackLine made "glass box" the center of its launch message. A label is not evidence, though. Here are five tests any controller can run in a demo or pilot to see whether a finance AI tool is actually auditable.

TL;DR: Explainable AI in finance means a reviewer can trace each AI output to source records, read the rule that produced it, replay it with the same result, see where the tool stopped short, and hand the evidence to an auditor who can reperform it without the vendor. Run all five tests before you sign. Fail one and you have a black box.

What does explainable AI mean in finance?

Explainable AI in finance is AI whose every output can be traced to its inputs, tied to a stated rule and reperformed by someone who did not build it.

Most content on this keyword is about lending and fraud models: why a credit application was declined, why a transaction was flagged. That work leans on model techniques like feature attribution. Accounting has a different bar. Your auditor does not need to know which neuron fired. They need evidence that supports the number in the ledger.

NIST's Four Principles of Explainable Artificial Intelligence (NISTIR 8312, September 2021) gives a useful vendor-neutral frame (NIST):

• Explanation: the system gives evidence or reasons for its outputs.
• Meaningful: the explanation is understandable to the person receiving it.
• Explanation accuracy: the explanation correctly reflects how the system actually produced the output.
• Knowledge limits: the system operates only within the conditions it was designed for, and says when it is outside them.

The five tests below turn those principles into checks a finance team can run in an afternoon.

Why does "glass box" matter to a controller now?

On April 15, 2026, BlackLine launched its agentic financial operations platform and described it as a "glass box" architecture. CEO Owen Ryan put the problem plainly: "CFOs need to leverage AI but remain personally liable for financial accuracy, so a 'black box' solution is not an option" (CPA Practice Advisor, Apr 15 2026).

That framing is right, and it is useful beyond one vendor. Once a major close vendor says black boxes are not acceptable, "glass box" becomes the bar every vendor will claim. The question for a buyer is how to check the claim.

COSO's February 23, 2026 guidance, Achieving Effective Internal Control Over Generative AI, names the risks a glass box has to answer: "opaque reasoning, model drift, and frequent configuration changes," and notes that GenAI "can be confidently wrong" (COSO via PR Newswire, Feb 23 2026). Each test below targets one of those risks.

What are the 5 tests for whether finance AI is auditable?

A glass box test is a check you run on the tool's output yourself, with your data, rather than a question you ask the vendor. Answers in a sales call are easy. Outputs are harder to fake.

• 1. Trace test: pick 10 AI outputs at random; open each one and follow it to the source rows. Pass: every output links to records that exist and agree. Maps to NIST "Explanation."
• 2. Rule test: for the same 10, ask the tool which rule or logic produced each result. Pass: a written, specific, versioned rule (a match condition, a threshold, a query), not "the model decided." Maps to "Explanation."
• 3. Replay test: rerun a closed period with the same inputs and rule versions. Pass: identical results, or every difference ties to a logged change. Maps to "Explanation accuracy."
• 4. Limits test: feed items the tool should not resolve, like a bank line with no ledger counterpart. Pass: it raises an exception instead of forcing a match. Maps to "Knowledge limits."
• 5. Outsider test: hand the evidence export to someone with no vendor login. Pass: they reperform three items from the export alone. Maps to "Meaningful."

Run them in order. If test 1 fails, tests 2 to 5 have nothing to stand on.

Test 1: Can you trace every output to source rows?

Pick outputs at random, not the examples the vendor chose for the demo. For a bank reconciliation, that means the matched bank line, the ledger entry, the amounts and the dates. For an invoice, the PDF, the extracted fields and the PO. A trace that ends at a summary screen or a confidence score is not a trace.

We covered what a match record should contain in AI bank reconciliation: what every match should explain.

Test 2: Is there a rule a reviewer can read?

A good answer reads like a control description: "matched on amount within $0.00 and date within 3 days, reference contains invoice number, rule v12." A weak answer is a narrative the model wrote after the fact. NIST's explanation accuracy principle is the reason this matters: a fluent explanation that does not reflect what actually ran is worse than none.

Want every match to come with the source rows, the rule and the SQL behind it? Join the Yoraito waitlist for early access.

Test 3: Does the same input give the same result?

This is the test that catches COSO's "model drift" and "configuration changes." Take a period you already closed, rerun it, and compare. If the tool cannot rerun a past period at all, note that as a fail: you will not be able to answer an auditor who asks why a September match exists.

If results are stored in tables, the comparison is one query:

SELECT COALESCE(o.bank_txn_id, r.bank_txn_id) AS bank_txn_id,
o.ledger_entry_id AS original_match,
r.ledger_entry_id AS replay_match,
o.rule_version AS original_rule,
r.rule_version AS replay_rule
FROM match_results_original o
FULL OUTER JOIN match_results_replay r
ON r.bank_txn_id = o.bank_txn_id
WHERE o.ledger_entry_id IS DISTINCT FROM r.ledger_entry_id
OR o.rule_version IS DISTINCT FROM r.rule_version;

Zero rows is a pass. Every row returned needs a logged reason: a new rule version, a corrected input, an approved override.

Test 4: Does the tool admit what it does not know?

Seed the test data with items that should not resolve: a bank deposit with no matching invoice, a vendor that is not in the master file, an invoice with a missing PO. A black box tends to produce a plausible answer anyway. A glass box routes the item to an exception queue and says why. This is NIST's knowledge limits principle applied to the close.

Test 5: Can an outsider reperform it without the vendor?

PCAOB AS 1105 lists reperformance, the independent execution of procedures or controls originally performed by company personnel, as one of the ways auditors gather evidence (PCAOB AS 1105). Your auditor will not have a seat in your AI tool. So export the evidence for three items and give it to someone who was not in the demo: a staff accountant, or your audit senior if they are willing. If they can reach the same answer from the export alone, the tool passes.

What does a pass look like next to a fail?

• Trace: glass box shows bank line ID, ledger entry ID, amounts and dates; black box shows "matched, high confidence."
• Rule: glass box shows rule ID, version and conditions; black box shows a model-written narrative.
• Replay: glass box gives the same output or a logged change; black box gives different results with no reason.
• Limits: glass box flags "no counterpart found, routed to exceptions"; black box forces a best-guess match.
• Outsider: glass box export is reperformable offline; black box evidence lives only in screenshots inside the app.

Score each test pass or fail, not on a 1 to 10 scale. A partial pass on traceability is a fail for audit purposes. If you are running a formal trial, these five slot into the audit trail line of our 30-day AI accounting pilot scorecard, and the matching vendor questions are in 12 audit trail questions for your AI accounting RFP.

Where do vendors usually fall short?

From the tests above, the common weak points are predictable:

• Confidence scores in place of evidence. A score tells you how sure the model is, not why it is right.
• Explanations generated after the fact. If the explanation is written by the same model, test 3 is the only way to check it reflects what ran.
• No rule versioning. Without versions, a replay cannot tell a drift from a deliberate change.
• Evidence that cannot leave the app. If the only way to show your auditor is a screen share, test 5 fails.

None of this means a tool that uses machine learning cannot pass. It means the learning has to produce something a reviewer can check: a rule, a record, a log. Your auditor will ask the same things; see 9 questions your auditor will ask about AI reconciliation.

FAQ

What is the difference between a glass box and a black box AI?

A black box AI gives an output without showing how it got there. A glass box AI exposes its inputs, the rule or logic applied and a log of each step, so a reviewer can check and reperform the result. For finance, the difference is whether the output can be audit evidence.

Is explainable AI required for financial reporting?

There is no US rule that names explainable AI for financial reporting. But management still owns its internal controls and auditors still need sufficient evidence under PCAOB AS 1105. In practice, AI output you cannot trace or reperform is hard to rely on in a control.

Can generative AI be explainable?

Partly. A language model's internal reasoning is not fully inspectable, and COSO flags "opaque reasoning" as a risk. A finance tool can still be auditable if the model's work is captured as checkable artifacts: source references, rules, queries and logs that pass the replay and outsider tests.

How do I test an AI tool before buying it?

Use your own data from a closed period, not the vendor's demo set. Run the five tests in this post, score each pass or fail, and have someone outside the evaluation try to reperform three results from the exported evidence.

Run the tests on your shortlist

Before your next vendor demo, send the five tests ahead and ask to run them on a closed month of your own data. A vendor confident in its glass box will agree. Yoraito is being built so finance teams and auditors can check AI output, not trust a black box. To hear when Yoraito opens up, join the waitlist.