Referral-Grade Evidence: Audit Trails That Survive CMS and an MFCU

In September 2020, because of fraud concerns, California's Employment Development Department (EDD) directed Bank of America to freeze 344,000 debit card accounts that paid unemployment benefits ^[1]. EDD had no plan in place to ensure it could unfreeze the accounts found to belong to legitimate claimants, and the California State Auditor found it had been slow to acknowledge its role in freezing them ^[1]. A fraud concern became a freeze, and legitimate claimants were left to prove they were real.

That failure is the reason this article exists. Referral-grade evidence means three things stacked together: confidence-scored detection signals, a documented human review step, and a preserved audit trail that reproduces exactly how the decision was made. A model score alone is a lead, not a verdict. If you cannot show a CMS auditor or a Medicaid Fraud Control Unit prosecutor who reviewed the case, what documents they examined, and which model version scored it, you do not have evidence. You have a liability.

The difference between a program that survives scrutiny and one that becomes the next headline is not detection accuracy. It is what happens between the score and the adverse action.

344,000 Frozen Accounts: The Failure That Should Haunt Every Program

The California EDD episode is the case study every program-integrity lead should keep on their desk. During the 2020 surge, EDD froze 344,000 debit card accounts over fraud concerns without a plan to unfreeze the ones that belonged to legitimate claimants ^[1]. The same audit found that EDD did not detect fraudulent claims filed under the names of incarcerated individuals, an estimated $810 million, until after it had paid them ^[1]. Detection came too late, and the enforcement that followed had no path back for legitimate claimants.

Here is the mechanical problem. An automated score answers one question: how anomalous does this application look relative to a model. It does not answer whether the anomaly is fraud, whether a human confirmed it, or whether the underlying documents support the flag. When a score triggers automatic suspension, you have skipped every step that turns suspicion into defensible evidence.

Contrast that with defensible enforcement. A score of 0.87 arrives in a case queue. An analyst opens a guided investigation, sees the specific signals that drove the score, reviews the source paystub and ID, records a determination, and either confirms or overturns the flag. Only after that human sign-off does an adverse action proceed. The score started the process. A person, with accountability and a paper trail, finished it.

That distinction is the whole discipline. Everything below is how you operationalize it.

What CMS, Congress, and an MFCU Actually Ask For

Three audiences will examine your program-integrity work, and they ask different questions. If you build for only one, you fail the other two. CMS oversight looks at security and privacy controls such as MARS-E and at program-integrity documentation. MFCUs investigate and prosecute Medicaid provider fraud and patient abuse or neglect ^[5], so they need evidence they can use in a criminal case. Oversight bodies such as GAO and legislative committees look at whether a program pays the wrong people and whether it wrongly cuts off eligible ones.

An unexplained model score fails all three at once. It has no reasoning a CMS reviewer can follow, no source documents an MFCU can enter into evidence, and no reviewer accountability that protects you from a wrongful-termination finding. The score is a black box, and every one of these audiences distrusts a black box that touches benefits.

The pattern across the table is consistent. Every audience wants to see the reasoning and the person behind the decision. Build the packet once, correctly, and it satisfies all three.

Anatomy of a Referral Packet That Holds Up

A referral packet that survives scrutiny has five fixed sections. Not four. Not "whatever the analyst had time for." Five, every time, so that a reviewer at CMS or an assistant attorney general at an MFCU can act without re-investigating the case.

The five sections are: subject identity facts, detection signals with confidence scores, source documents, reviewer determination and notes, and chain of custody. Fix the structure and quality becomes repeatable instead of dependent on which analyst caught the case.

Consider an illustrative broker-ring referral; the case and its numbers are hypothetical. Suppose broker-ring graph analytics surfaced 47 applications linked through a shared agent National Producer Number, three reused device fingerprints, and a common bank-routing pattern. The document forensics layer flagged 31 of those applications with paystubs sharing identical metadata and a cloned employer logo. Synthetic-identity intelligence found four SSNs belonging to deceased individuals and six reused across multiple applications. Each finding carries a confidence score. Each score links to the source artifact.

That packet does not say "these look fraudulent." It says: here are 47 applications, here is the graph edge that links them, here are the hashed source paystubs, here is the SSN death-master-file match, and here is the analyst who confirmed each. An MFCU can prosecute from that. A score cannot.

Use this checklist before any packet leaves the queue:

Confidence Scoring: Why a Number Is Not a Decision

A confidence score is only useful if it means something reproducible. A calibrated 0.82 should mean that, across your validation set, cases scored 0.82 turned out to be fraud roughly 82 percent of the time. Raw model output that has never been calibrated against outcomes is a vibe, not a probability, and you should never route enforcement on a vibe.

Calibration is what lets you set risk tiers that route cases to different review depths instead of terminating anyone automatically. High-confidence cases get expedited review. Mid-tier cases get a full guided investigation. Low-confidence cases get a lighter touch or a documentation request rather than an adverse action. No tier ends in automatic termination.

That pressure has not eased: federal agencies reported an estimated $186 billion in improper payments across 64 programs in FY2025 ^[3]. The lesson embedded in those numbers is that pressure to catch fraud fast is exactly what produces wrongful terminations. When the political cost of missed fraud is high, teams lower thresholds and skip review. Calibrated scoring plus mandatory human review is the counterweight. It lets you act on real fraud without punishing the innocent people who look statistically unusual because they are self-employed, recently moved, or newly enrolled.

This is the same discipline behavioral-baseline work applies elsewhere in identity security. A single anomaly is a signal, not proof. The mature move is progressive trust: escalate scrutiny as evidence accumulates rather than acting on the first flag.

The Human-in-the-Loop Step You Cannot Skip

The reviewer step is the difference between the California outcome and a defensible program. It is not a rubber stamp. A proper guided investigation presents the analyst with the specific signals, the linked entities, and the source documents, then requires structured output before the case can close.

The reviewer workflow needs four non-optional fields: a determination (confirm, overturn, or request documentation), mandatory free-text notes explaining the reasoning, the reviewer's identity, and a timestamp. The reviewer must also have real authority to overturn a signal. If a paystub the model flagged turns out to be legitimate on inspection, the analyst overturns it, and that overturn is recorded as data.

Override data is not overhead. It is the feedback loop that keeps your program fair. When analysts consistently overturn signals in a certain confidence band or from a certain detection layer, that is your calibration telling you the threshold drifted. Feed override rates back into threshold recalibration, and you close the loop between detection and fairness. A program that never overturns anything is not accurate. It is not looking.

Chain of Custody and Reproducibility for Digital Evidence

Digital evidence has to be reproducible or it is worthless in front of an MFCU. Reproducibility means immutable logging of who saw what, when, which model version scored the case, and what source documents were examined, each hashed so tampering is detectable.

Every case event should record the acting person or system, the timestamp, the action taken, and the artifact touched. That per-event record is what lets an outside investigator replay the case without access to your systems.

The model version matters more than teams expect. If you scored a case with model v3.1 and then retrained to v4.0, an appeal or audit six months later must reproduce the exact score the subject received. Log the model version with every determination or you cannot defend the decision later.

Set retention to match CMS records requirements and your state's appeals window, whichever is longer. Under-retaining is how a solid case falls apart on appeal two years later.

Metrics That Prove Your Program Is Not the Next Wrongful Freeze

You cannot manage defensibility without measuring it. Four metrics belong on a weekly dashboard: false-positive rate, human-override rate, referral acceptance rate by MFCU, and time-to-determination. Watch them weekly, not quarterly, because drift causes harm fast.

The two that predict a wrongful-freeze incident are false-positive rate and human-override rate. If your false-positive rate climbs, your thresholds are too aggressive and you are freezing legitimate people. If your override rate spikes, the model and reality have diverged and your scoring needs recalibration. Set trigger thresholds in advance: if either metric moves more than a set percentage week over week, recalibration is mandatory before the next batch of cases routes to enforcement.

Frequently asked questions

Can we terminate benefits based on the automated score? No. The score initiates review. A documented human determination must precede any adverse action, and the reviewer's identity and notes must be in the record.

What makes a referral packet acceptable to an MFCU? Five sections: subject identity facts, confidence-scored detection signals, hashed source documents, a named reviewer determination with notes, and a chain-of-custody log. The prosecutor should be able to act without re-investigating.

How do we stay audit-ready for CMS? Preserve model versions, reviewer actions, and source-document hashes for the longest retention period your CMS agreements, state law, and appeals window require, and produce weekly false-positive and override metrics. MARS-E is CMS's security and privacy guidance for the ACA program environment, and it applies to all ACA Administering Entities, including Exchanges and state Medicaid agencies ^[4].

What protects us in an appeal? Reproducibility. If you can reproduce the exact score, the documents reviewed, and the human who signed off, you can defend the decision. If any of those is missing, expect a reversal.

Start Here: A Quick Audit of Your Current Evidence Trail

Start with a short audit. Pull three recent adverse-action cases. For each one, check two things: is there a named human reviewer with a documented determination, and are the source documents preserved and hashed. If any of the three cases fails either check, you have wrongful-freeze exposure right now, and you should stop automated adverse actions until the human step is enforced.

Start tracking one metric this week: the human-override rate on high-confidence cases. If it is near zero, your reviewers are rubber-stamping and you have no real safety net. If it is very high, your thresholds are miscalibrated and you are flagging legitimate people. A healthy program sits in a measured middle and watches the number move.

Legitimate claimants were locked out of their benefits because a fraud concern became a freeze with no plan to undo it and nobody accountable in between. The fix is not better scoring. It is confidence-scored signals, a human who signs their name to the determination, and an audit trail that reproduces the whole decision. Build the packet with five fixed sections, log every action as a discrete event, and measure override and false-positive rates every week. That is what referral-grade evidence means, and it is what keeps your program off the front page.

References

[1] California State Auditor, "Employment Development Department: Significant Weaknesses in EDD's Approach to Fraud Prevention Have Led to Billions of Dollars in Improper Benefit Payments," Report 2020-628.2, January 28, 2021. https://information.auditor.ca.gov/reports/2020-628.2/index.html

[2] U.S. Government Accountability Office, "Unemployment Insurance: Estimated Amount of Fraud During Pandemic Likely Between $100 Billion and $135 Billion," GAO-23-106696, 2023 (most comprehensive federal estimate available). https://www.gao.gov/products/gao-23-106696

[3] U.S. Government Accountability Office, "Improper Payments," FY2025 government-wide estimate. https://www.gao.gov/improper-payments

[4] Centers for Medicare & Medicaid Services, "Minimum Acceptable Risk Safeguards for Exchanges (MARS-E) Document Suite, Version 2.2, Volume I: Harmonized Security and Privacy Framework," 2021. https://www.cms.gov/files/document/mars-e-v2-2-vol-1final-signed08032021-1.pdf

[5] U.S. Department of Health and Human Services, Office of Inspector General, "Medicaid Fraud Control Units." https://oig.hhs.gov/fraud/medicaid-fraud-control-units-mfcu/