To measure hallucinations in company context, define a hallucination as a material claim or action that lacks support from the approved evidence available for that task, entity, and date. The unit should be the claim, not the whole answer. One response may contain grounded facts, a reasonable inference, an unsupported assertion, and a correct refusal.
This definition prevents two common errors. A fluent answer is not automatically grounded, and a cautious answer is not automatically useful. The evaluation must test whether the system selects the right company source, represents uncertainty, and abstains when evidence is missing or authority is unclear.
Measurement begins with a scoped task inventory. List the decisions users ask the system to support, the sources allowed for each decision, and the consequences of a false claim. Separate entities and sensitivity levels so evidence from one company, client, or internal room cannot validate another.
Create a dated, claim-level evaluation set
Sample realistic questions across routine facts, changing operational state, ambiguous names, conflicts between documents, restricted material, and known gaps. Store the expected source and its effective date beside each case. Include questions that should receive a refusal or a request for clarification.
Label each answer at claim level: supported, contradicted, unsupported, stale, wrong entity, overconfident inference, or acceptable abstention. Reviewers also note whether a cited source actually contains the asserted meaning. Citation presence alone cannot prove grounding.
The explanation of how confident AI failures erode founder trust is useful when defining severity. A wrong office location and an invented payment approval are both false, but they do not carry the same operational consequence or release rule.
Keep a holdout set and preserve the exact corpus snapshot used by the test. Company knowledge changes. Without a date boundary, a later document can make an old answer appear correct even though the system could not have used that evidence when it responded.
Report rates with denominators and severity
Calculate separate rates for unsupported claims, contradictions, stale claims, entity leakage, and failed abstention. Show both the number of affected claims and the number of responses containing at least one material failure. These denominators answer different management questions.
Weighting by severity can support an internal release decision, but publish the weighting method beside the score. Do not present a custom risk score as a universal benchmark. A single critical authorization error may justify rejection even when most low-impact claims are correct.
For a hallucination dashboard, apply three checks for AI-invented metrics before accepting any reported rate. Definitions, event sources, exclusions, and missing observations must be fixed outside the evaluated agent so it cannot grade itself by changing the measurement.
Pair quantitative counts with reviewer notes. A declining rate can result from excessive refusal, a narrower task mix, or removed difficult cases. Track useful supported completion and appropriate abstention alongside hallucination classes to reveal that tradeoff.
Turn findings into a controlled correction loop
For each material failure, identify the responsible layer: missing source, wrong authority rule, retrieval miss, prompt interpretation, unsupported inference, tool defect, or reviewer ambiguity. Fix the narrowest layer and rerun the frozen cases plus a protected holdout.
An independent reviewer should challenge the labels and inspect high-severity examples. Disagreement is evidence about an unclear policy or expected answer. Resolve the contract before tuning the system, otherwise the test will reward whichever reviewer happened to label the latest run.
The verified-completion and retained-memory pattern turns a finished label into inspectable evidence for this measurement protocol. For hallucination measurement, retain inputs, corpus version, outputs, labels, reviewer decisions, and the correction that followed each critical failure.
Frequently Asked Questions
It is a material claim or action that is contradicted, stale, tied to the wrong entity, or unsupported by approved evidence for the task and date.
Claim-level labels preserve mixed outcomes, because one response can contain supported facts, an unsupported inference, and an appropriate statement of uncertainty.
Show affected claims, evaluated claims, responses with material failures, total responses, and the task mix so readers can interpret each rate.
Treat disagreement as evidence of an unclear authority rule or expected answer, resolve the contract, and then rerun the affected frozen cases.
A measurement protocol can reveal where company-context answers fail, but it cannot prove universal accuracy. When company-context accuracy is being measured, an AI governance and agent audit offers an outside review of sources, labels, and control rules; the organization still owns its thresholds and accepted risk.