Methodology for auditing AI agents

Purpose and scope of this document

This document describes the method Axon applies to evaluate artificial intelligence agents that interact with end customers, and to issue the compliance dossier that results from that evaluation.

1.1Who it is for

Three readers, with distinct needs:

• The client's internal audit and compliance, who need to judge whether the report is acceptable as evidence in their own process.
• External auditors and regulators, who need to verify that the conclusion follows from the method, and not from convenience.
• Axon teams and accredited auditors, who need to run the audit identically across clients.

1.2What this document is not

• It is not a product manual. It does not describe screens, integrations or APIs.
• It is not a legal opinion. It does not assert compliance with any specific rule; it asserts what was observed, and it is up to the client and its advisers to conclude on compliance.
• It is not a guarantee that no error exists. Audit by sampling estimates a rate; it does not certify that no individual case is wrong.
• It is not an accredited certification. Axon is not a certification body accredited by Inmetro (the Brazilian accreditation authority) or an equivalent entity, and does not present itself as one.

Principles

Five principles govern every decision in this method. Where a specific rule is silent or ambiguous, the principle decides.

2.1Independence

Axon evaluates agents; it does not build them, tune them or operate them. Recommending a fix is part of the report; carrying the fix out is not. Whoever carries out the fix cannot attest to its result.

2.2Evidence before conclusion

No finding exists without a literal excerpt from the source conversation, with an identified position and programmatically verified existence. A finding without quotable evidence is discarded, however convinced the evaluator may be of it.

2.3Reproducibility

A third party with access to the same sample, the same rubric and the same configuration must reach the same result, within the tolerance declared in 9.3. A result that cannot be reproduced is not evidence.

2.4Declared limitation

Every report declares what was not covered, what cannot be asserted, and where uncertainty is greatest. The limitations section is mandatory and is not trimmed for commercial convenience.

2.5The decision is human

The automated judge produces a score and evidence. The compliance conclusion, the final severity classification and the issuance of the report are human acts, with an identified and recorded owner. No language model decides, on its own, the outcome of an audit.

Definitions

Controlled vocabulary. In any Axon report, these terms have exactly the meaning below.

Scope of an audit

4.1Unit of audit

The unit is the triple agent × channel × period. One report covers one triple. Auditing two agents produces two sets of scores, even within a single document, because a score aggregated across distinct agents has no operational meaning.

4.2What is in and what is out

Purely deterministic journeys, such as forms and fixed decision trees, are not evaluated for probabilistic content. Only the boundary is evaluated: where the deterministic flow hands over to the generative agent, and whether the transparency notices and the exits to a human agent exist and work.

4.3Reference period

A minimum of 30 calendar days, to absorb weekly variation. A maximum of 90 days, because beyond that the probability of a silent model or prompt change mid-period compromises the homogeneity of the population. Longer periods are split into subperiods evaluated separately.

4.4Client prerequisites

1. 1Transcripts for the period, with a stable identifier, a timestamp, the speaker role and the content. JSON, JSONL or CSV format.
2. 2The internal rules document in force that governs the service, with version identification.
3. 3A reviewer from the business area, available for approximately two hours for the calibration.
4. 4An approver with authority to freeze the rubric before the evaluation.
5. 5A data processing instrument signed before any transfer.

4.5Grounds for refusal

The audit is not started, or is interrupted, when:

• There is no agent in production serving end customers. An internal pilot does not constitute an auditable object.
• There are no written internal rules, and the client does not accept that the rubric be built and formalised before the evaluation.
• The population of the period is smaller than the calculated minimum sample size, making estimation unnecessary, since a census is feasible.
• The client conditions the issuance of the report on a result, or requests suppression of an evidenced finding.
• Axon has, or had in the preceding twelve months, a role in building or operating the audited agent.

Rubric: library, mapping and packages

The question that sinks a report in an audit room is "who defined these criteria?". The answer has to be: a clause of the client's own document, or an identified regulatory requirement. Never Axon's opinion.

That creates a tension of scale. If every rubric is written from scratch, the fifth client costs the same as the first and the method does not scale. The solution is to separate what is reusable from what is specific: Axon maintains a canonical library of criteria, and the per-client work becomes mapping that client's internal rules onto the library, not inventing criteria.

5.1Canonical library of criteria

Every criterion in the library has a stable code, a statement, an objective test and a condition of applicability. A criterion only enters a rubric once anchored in a client clause or a regulatory requirement; the library supplies the wording and the test, not the authority.

Factual accuracy

Adherence to internal rules

Resolution

Treatment and tone

Transparency

Data protection

5.2Procedure for mapping internal rules

Executable by a qualified analyst under chapter 12, with no involvement from the technical owner until step 5.

1. 1Segment the internal rules into numbered clauses, preserving the source numbering.
2. 2Classify each clause as: an obligation verifiable in a conversation, an obligation verifiable outside the conversation, or information without an obligation. Only the first generates a criterion.
3. 3Anchor each verifiable obligation to a criterion in the library. One clause may anchor more than one criterion; one criterion may have more than one anchor.
4. 4Parameterise the criterion's variables with the client's values: authority limits, deadlines, form of address, escalation triggers.
5. 5Identify gaps in both directions: client obligations with no corresponding criterion in the library, and library criteria with no anchor in the internal rules. Both lists go to the client before approval.

The product of the procedure is a traceability matrix: source clause, anchored criterion, parameters applied, and reciprocal coverage. The matrix is a mandatory annex of the dossier.

5.3Client-specific criteria

A client obligation with no counterpart in the library generates a specific criterion, written in the canonical format and marked with origin: specific. A specific criterion that reappears in three distinct clients is a candidate for promotion into the library, through a MAJOR change to this methodology.

5.4Sector rule packages

A sector package is a pre-anchored subset of the library, with the regulatory requirements common to a sector already mapped and parameterised with the usual values. A package contains: applicable criteria, regulatory anchors, parameters with a default value and a value to be confirmed, and the scoping questions specific to the sector.

The package does not replace the mapping of the client's own internal rules. It removes the work of identifying and drafting the regulatory layer, which is identical across clients in the same sector. Packages follow their own versioning and the dossier declares which version was applied.

5.5Approval and freezing

The rubric is formally approved by the client before the evaluation begins, with a record of approver, date and version, together with the traceability matrix and the two gap lists from step 5. A rubric adjusted after the result is known invalidates the report. An adjustment that becomes necessary during the evaluation restarts the evaluation under a new version, and the dossier records both.

5.6Versioning

Rubrics follow MAJOR.MINOR. A change to a criterion, a parameter or the set of dimensions increments MAJOR and prevents direct comparison with earlier periods. A wording correction that does not change meaning increments MINOR and preserves comparability. The dossier declares the version applied and whether historical comparison is valid.

Sampling

Audit by sampling is only worth anything if the selection method is declared and reproducible. "We evaluated the conversations that were available" is not sampling.

6.1Population

The population is the set of all conversations of the agent, in the channel and period in scope, with at least one agent turn. Conversations with zero agent turns are excluded and the exclusion is counted and reported.

6.2Stratification

The sample is stratified by channel, by agent version and by week of the period, with allocation proportional to the size of the stratum and a minimum of one conversation per stratum. Stratifying by week exists to stop the sample from concentrating in an atypical period.

6.3Sample size

Calculated to estimate a proportion, with a finite population correction. The expected proportion is fixed at 0.5, the maximum-variance case, which makes the estimate conservative and independent of any prior assumption about the failure rate.

Default parameters: 95% confidence and a margin of ±5 percentage points, which yields approximately 380 conversations for populations from ten thousand upwards. The margin actually achieved by the drawn size is recalculated and declared in the dossier, and prevails over the intended margin.

6.4Adverse stratum

Conversations flagged as suspect by the client, or linked to a formal complaint, are oversampled in a separate stratum, typically 10% to 20% of the size of the representative sample.

6.5Record of parameters

Every sampling produces an immutable record with: population size, size per stratum, confidence level, intended margin, effective margin, pseudo-random generator seed, algorithm version and a cryptographic digest of the list of drawn identifiers.

6.6What invalidates the sample

• Receiving additional conversations from the period after the draw. The sample is redrawn.
• Discovering that the client's extract was already filtered, making the declared population false.
• A change of agent version mid-period without the version being recorded per conversation.
• Manual selection of conversations by either party outside the adverse stratum.

Evaluation

7.1Pipeline

1. 1Normalisation. Conversations converted to the canonical turn model, with validation of ordering and roles.
2. 2Personal data redaction. Applied before any persistence and before any call to the judge.
3. 3Draw. Sample built under chapter 6 and frozen.
4. 4Judging. Each conversation evaluated against the frozen rubric, with a fixed configuration.
5. 5Quote verification. Every piece of evidence checked against the source conversation.
6. 6Aggregation. Scores per dimension, violation rate per criterion, confidence intervals.
7. 7Human review. Under 7.6.
8. 8Calibration. Under chapter 8.
9. 9Issuance. After the chapter 10 checklist is satisfied.

7.2Personal data redaction

Identity documents, cards, email addresses, phone numbers, proper names and identification keys are replaced by stable markers before storage. The judge operates exclusively on redacted text. The stability of the markers within a single conversation is preserved, so that referential coherence is not lost and does not produce a false finding.

7.3Judge configuration

7.4Adaptive judging

Repeating every judgement three times triples the cost to reduce uncertainty where there is none. The number of repetitions is a function of the uncertainty itself:

The reported score is the median of the passes actually run. The dossier declares, per conversation, how many passes there were and under which trigger. In practice, between 12% and 25% of the sample receives repetition, which cuts judging cost by around 60% compared with three universal passes, without losing rigour where it matters.

7.4Quote verification

Every finding carries the turn index and the literal excerpt. Verification confirms, programmatically, that the excerpt exists in that turn of that conversation. A finding whose quote is not confirmed is discarded, and the discard is counted and reported as an indicator of judging quality, not silenced.

7.5Scale and bands

Each dimension receives a score from 0 to 100. The bands use three-level semantics:

There is no compensation between dimensions. Any dimension in the critical band caps the overall reading at "attention" in the best case, and the dossier names the dimension responsible for the cap. A mean that dilutes one critical failure across five healthy dimensions does not describe risk.

7.7Severity

Severity combines individual impact and observed frequency:

Gap 2. The quantitative boundary between a recurring and a residual failure varies by sector and by volume. Until this closes, frequency is reported in absolute and relative numbers, and the recurrence classification is a recorded human decision. See chapter 18.

7.8Human review by failure pattern

Reviewing one hundred per cent of critical instances is unsustainable and, worse, redundant: sixty critical findings are usually four or five repeated patterns. What needs human judgement is the pattern, not each of its occurrences.

Findings are grouped into failure patterns. Two findings belong to the same pattern when they share the violated criterion, the failure mechanism and the form of manifestation. The grouping is proposed automatically and confirmed by a human.

Consequence of rejection. If review invalidates an instance, it is discarded and another from the same pattern enters review. If it invalidates the majority of the reviewed instances of a pattern, the whole pattern is discarded from the report and the discard is reported as an indicator of judging quality.

Calibration against a human reviewer

This is the step that answers the most predictable objection: why trust a model's judgement. The answer is not argumentative, it is numerical.

8.1Procedure

1. 1A subset of 40 to 60 conversations is drawn from the representative sample, with the same stratification.
2. 2The client's reviewer evaluates that subset under the frozen rubric, without access to the judge's scores.
3. 3The scores are reduced to the three bands in 7.5, because agreement on a continuous scale is a fragile measure and overstates trivial disagreement.
4. 4Cohen's kappa coefficient is computed between reviewer and judge, per dimension and in aggregate.
5. 5Band disagreements are reviewed case by case and classified by cause: ambiguous rubric, judge error or reviewer error.

8.2Calculation

8.3Threshold for issuance

The 0.61 threshold corresponds to the start of the band conventionally described as substantial agreement in the inter-rater reliability literature. The choice is conservative and declared; it is not optimised to make issuance easier.

8.4When the divergence is the reviewer's

It happens, and it is not embarrassing: a human reviewer also errs, and sometimes applies an established practice that contradicts the written rules. That is, in itself, a relevant finding and goes into the dossier as a divergence between rule and practice. It is not used to inflate agreement; the reported kappa is the one computed before any reconciliation.

8.5Record

The dossier declares: the size of the calibration subset, the functional identification of the reviewer, kappa per dimension and in aggregate, the number of divergences and their classification by cause.

Reproducibility

An external auditor will re-run this. If the numbers move without explanation, the report loses its evidentiary value entirely.

9.1What is fixed

• The full identifier of the judge model, with version.
• Temperature, seed and number of repetitions.
• The rubric version, including the full text of the criteria.
• The version of the sampling algorithm and of the personal data redactor.
• An immutable snapshot of the list of drawn conversations.

9.2What is recorded

• The timestamp of each judgement and its duration.
• The scores of each of the three repetitions, not just the median.
• Findings discarded through failed quote verification.
• The identity of every human reviewer and every approver, per event.

9.3Re-run and tolerance

A re-run with all parameters fixed must reproduce each dimension score within ±2 points. A greater divergence generates a divergence report, with mandatory investigation of the cause among three hypotheses: the provider altered the model without changing the identifier, a defect in the freezing of parameters, or provider non-determinism above what was declared.

9.4Artefact integrity

The dossier receives a SHA-256 cryptographic digest over its canonical content, published in the document itself and in the accompanying JSON. Any later alteration is detectable by recomputation. Correcting an issued dossier generates a new version, with a new digest and a rectification note; never a silent replacement.

Composition of the dossier

Mandatory sections. The absence of any one of them prevents issuance.

Declared exceptions

No real client satisfies every prerequisite. Without an exception procedure, the first audit forces a choice between breaking the method in silence and not delivering. Both destroy the credibility of the report.

11.1Principle

Every exception is named, authorised, recorded in the dossier and accompanied by its consequence for what can be asserted. A declared exception narrows the reach of the conclusion; a silent exception invalidates the report.

11.2Catalogue

11.3Authorisation

Exceptions EX-01 to EX-03, EX-05, EX-06 and EX-08 are authorised by the technical owner of the audit. EX-07 and EX-09 additionally require approval by the methodology owner. EX-04 is not an authorisable exception: it is a block.

11.4Accumulation limit

Three or more simultaneous exceptions in the same audit require a viability reassessment before proceeding. Beyond that point, the reach of the report tends to be so narrow that the artefact stops serving its purpose, and it is more honest to renegotiate scope than to issue a document of marginal value.

Roles, qualification and delegation

A methodology only scales if someone else can execute it. This chapter defines who does what, what they need to know, and what is never delegable.

12.1Roles

12.2Qualification requirements

12.3Internal assessment set

Axon maintains a fixed set of anonymised conversations with a reference classification established by consensus. It serves to: qualify candidates for technical reviewer, periodically reassess reviewers, and detect judgement drift over time. The set is not used in any client audit and does not form part of the aggregate corpus.

12.4What is not delegable

• Signing the report. Always an identified natural person, meeting the requirements in 12.2.
• Authorising an exception. Never by an analyst.
• The decision not to issue. An issuance block is not overturned by a commercial authority.
• Changing this methodology. Only the methodology owner, with a public record.

12.5Accredited network

Third-party auditors may operate under this methodology through accreditation, which requires: meeting the 12.2 requirements for the intended role, adherence to the restrictions in chapter 14, declaration of any prior relationship with each auditee, and submission to quality review by sampling of at least 20% of the reports issued in the first twelve months. Failing a quality review suspends the accreditation and requires the affected reports to be reissued.

Indicators of the method's scale

A methodology that intends to scale needs to measure whether it is scaling. These indicators are computed per audit and reviewed quarterly by the methodology owner.

Independence and conflict of interest

14.1What Axon does not do

• It does not write or tune instructions, prompts or configuration of the audited agent.
• It does not build, operate or host the audited agent.
• It does not accept variable compensation tied to the outcome of the evaluation.
• It does not audit an agent it took part in building in the preceding twelve months.
• It does not suppress an evidenced finding at the request of any party.

Recommending a fix is part of the craft and does not impair independence. Carrying it out does, because whoever carries it out would then be evaluating their own work.

14.2Accredited network

When third-party auditors begin operating under this methodology, the same restrictions apply, plus: submission to quality review by sampling by Axon, an obligation to declare any prior relationship with the auditee, and a prohibition on acting simultaneously as implementer and auditor of the same agent.

14.3Technical owner

Every report has an identified natural person as technical owner for its issuance. That person answers for the work's adherence to this methodology. A report without a named owner is not issued.

Data handling

15.1Minimisation

The minimum necessary is requested: transcripts for the period and channel in scope. Customer records, credit databases, original audio recordings and biometric data are not requested.

15.2Personal data

Redaction happens before persistence and before anything is sent to the judge. The judge operates on redacted text. The evaluation does not require identifying the data subject, and Axon does not reconstruct that identity.

15.3Retention and erasure

The retention window is set contractually. Once the term ends, or at the client's request, the data is erased, preserving only: sampling parameters, aggregate scores, cryptographic digests and the issued dossier. That keeps the report auditable without keeping the content of the conversations.

15.4Execution in the client's environment

Where transferring transcripts is not permissible, the audit is executed in the client's own environment, with Axon receiving only aggregate results and already-redacted evidence. The mode adopted is declared in the dossier, because it affects what Axon can verify independently.

15.5Aggregate corpus

Subject to a contractual provision, Axon maintains an aggregate, non-identified corpus of failure patterns, intended for judge calibration, sector comparison and evolution of the method. The corpus contains no conversation content, no client identification and no data subject identification. It is the basis of the sector benchmark described in gap 4.

Governance of the method

16.1Versioning

This methodology follows MAJOR.MINOR. Any change that could alter the outcome of an audit increments MAJOR: dimensions, thresholds, bands, sample size, kappa threshold, re-run tolerance. Anything that merely clarifies wording increments MINOR.

16.2Absence of retroactive effect

An issued report remains valid under the version in force at its issuance and is not reinterpreted under a later version. A comparison between periods evaluated under distinct MAJOR versions is declared not directly comparable.

16.3Change approval

A MAJOR change requires a written proposal with a justification, an assessment of the impact on earlier reports and approval by the methodology owner. The change record is public, alongside this document.

16.4Change record

Known limitations

What this methodology cannot assert. This section is reproduced, adapted to the case, in every dossier.

• Sampling estimates, it does not certify. The conclusion is about a rate in the population, with a declared margin, and not about the correctness of any conversation that was not evaluated.
• The judge is a language model. Substantial agreement with a human reviewer is evidence of reliability, not proof of infallibility. Cases of genuine ambiguity remain ambiguous.
• A rubric derived from internal rules inherits their defects. If the client's internal rules contradict the law, an agent that adheres to them scores highly and remains exposed. Evaluating those rules themselves is out of scope.
• Retrospectivity. The report describes a past period. It asserts nothing about the agent's future behaviour, especially after a change of prompt, model or knowledge base.
• Dependence on the client's extract. If the database provided was already filtered in an undeclared way, the population is false and the report inherits that defect. It is mitigated by checking counts; it is not eliminated.
• Text, not audio. On a voice channel, the transcription is what is evaluated. A transcription error can produce a false finding, and prosody is not evaluated.
• No accreditation. This report is an independent, reasoned technical opinion; it is not an accredited certification, and it does not replace a regulatory requirement that demands an accredited body.

Gaps in this version

Each gap below has a provisional decision in force, a closing criterion and the evidence that needs to be collected. None will be closed by reasoning; all depend on the field.