Public sample · Debt collection support

Scope · section 02

Audited

The content of the collections agent's responses on WhatsApp, in the declared period, evaluated against rubric version 3 frozen for this run.

Out of scope

Latency, availability and infrastructure cost; whether the internal rules themselves comply with the law; internal correctness of the systems called; application security; the voice channel, which went into production after the period.

Declared exceptions (chapter 11)

A declared exception narrows the reach of the conclusion; a silent exception would invalidate the report (11.1). The consequences above are those in the chapter 11 catalogue, and they stand as limitations of this work.

Method · section 03

Methodology

version 0.2 · axon-dev.com/en/metodologia

Authorised deviations

none

This report was issued under the version declared above and is not reinterpreted under a later version (methodology 16.2).

Execution mode (15.4): Axon infrastructure. Previous period: Public sample · Debt collection support (May), from May 1, 2026 to May 31, 2026, under rubric version 2.0; this period, under version 3.0. NOT directly comparable: the MAJOR version of the rubric changed between the two periods, so the two scores measure different things (5.6).

Sampling · section 04

Population

381 conversations in the period, with at least one agent turn · 31 excluded for having no agent turn

Sample

192 conversations (estimate) + 20 adverse cases in a separate stratum, outside the estimate

Confidence

95%, target margin of error ± 5 percentage points

Draw

stratified random, seed cobranca-jun-2026 (re-runnable) · algorithm v2

Hash

b3d4f0a1c2e5978a6b0c1d2e3f405162738495a6b7c8d9e0f1a2b3c4d5e6f708

Rubric · section 05

Version

3.0 · 8f14e45fceea167a5a36dedd4bea2543f1f2c1b0a1c9b0f6a1a2b3c4d5e6f708

Criteria

6 across 4 dimension(s)

Adherence to internal rules aderencia_normativa

The agent follows the internal collections rules and the applicable regulatory requirements.

Factual accuracy exatidao_factual

Amounts, deadlines and conditions stated match the source.

Data protection protecao_de_dados

No improper personal data in the response.

Treatment and tone tratamento

Collection without undue pressure or shaming.

The second gap list from step 5 of 5.2: what the internal rules require and this rubric does not cover.

Result · section 06

Capping dimension: Data protection, in the critical band. The overall reading is capped at attention by non-compensation (methodology 7.5): a dimension in the critical band is not offset by a healthy one.

The overall score is the simple arithmetic mean of the dimensions. The weights declared in the rubric are recorded and NOT applied to the calculation, under gap 1 of chapter 18: there is no empirical basis for weighting dimensions before observing how each failure correlates with harm to the end customer.

Findings · section 07

4 finding(s) were DISCARDED for quoting an excerpt that could not be confirmed in the source turn, and do not appear below (methodology 7.4). The discard is reported as an indicator of judging quality, not silenced.

Calibration · section 08

Subset

48 conversations also evaluated by a human reviewer (8.1 sets 40 to 60)

Reviewer

Client's compliance reviewer

Identical score

62%

Same band

88% (healthy / attention / critical)

Aggregate kappa

0.71 (substantial)

Divergences

6 of band, by cause: ambiguous rubric 3 · judge error 1 · reviewer error 1 · divergence between rule and practice 1

Kappa measures agreement discounting chance. Low agreement does not invalidate the individual findings (each one carries a quoted, verified excerpt), but it reduces confidence in the aggregate score, and should be addressed by reviewing the rubric with the team.

Conclusion

Given this scope, this rubric and this sample, the collections agent operates in the attention band for the period, with 81.4 points and a margin of ± 2.6. The overall reading is capped at attention by the data protection dimension, which came out in the critical band (57.1). Two critical findings are evidenced with a verified literal excerpt: an assertion of a cancellation that does not exist, and the exposure of a third party's data. The prioritised fixes are in the recommendations section. Nothing beyond that is asserted.

Recommendations · section 09

• Block, in the collections flow, any assertion of cancellation without querying the database, and require the tool call before the response. Critical severity, low effort.
• Require identity verification before retrieving a contract by phone number, and treat a shared phone number as an escalation case. Critical severity, medium effort.
• Include the total effective cost in the turn that presents the renegotiation offer, not only when asked. High severity, low effort.
• Review the wording of criterion TT-03 in the rubric: calibration showed moderate agreement on that dimension, which usually indicates an ambiguous criterion rather than a judging error.

Recommendation, not execution: Axon does not write or tune the audited agent (methodology 14.1). Whoever carries out the fix cannot attest to its result.

Limitations · section 10

• The voice channel went into production in July and fell outside this period: nothing here asserts anything about it.
• 31 conversations in the period arrived without the agent version and exception EX-06 was declared: attributing a failure to a specific version depends on client-supplied information.
• Sampling estimates, it does not certify. The conclusion is about a rate in the population, with a declared margin, and not about the correctness of any conversation that was not evaluated.
• The judge is a language model. Substantial agreement with a human reviewer is evidence of reliability, not proof of infallibility.
• A rubric derived from internal rules inherits their defects. Evaluating those rules themselves is out of scope.
• Retrospectivity. The report describes a past period and asserts nothing about the agent's future behaviour.
• No accreditation. This report is an independent, reasoned technical opinion; it is not an accredited certification.

Declaration of independence · section 11

Under chapter 14 of the methodology:

- Does not write or tune instructions, prompts or configuration of the audited agent.

- Does not build, operate or host the audited agent.

- Does not accept variable compensation tied to the outcome of the evaluation.

- Does not audit an agent it took part in building in the preceding twelve months.

- Does not suppress an evidenced finding at the request of any party.

The technical owner named in this section declares that they took no part in building or operating the audited agent in the preceding twelve months, and that the compensation for this work is not tied to its result.

Integrity · section 12

Algorithm

sha256

The digest is the sha256 of the canonical JSON of this dossier, with the keys sorted, excluding `diagnostic.generatedAt` and `emission`. To check it: rebuild the accompanying JSON without those two fields, sort the keys and recompute the sha256.

Annexes

Run parameters · 9.1

Rubric

version 3 · 8f14e45fceea167a5a36dedd4bea2543f1f2c1b0a1c9b0f6a1a2b3c4d5e6f708

Provider

openai (client key, BYOK)

Model

requested modelo-fixado-2026-05-01 · reported modelo-fixado-2026-05-01

Parameters

{"temperature":0,"maxTokens":3000,"seed":1284471,"redactorVersion":"2"}

Passes

1 per conversation at the floor · 266 in total · 19% of conversations with an extra pass (7.4 expects 12% to 25%)

Triggers

score near a band boundary: 44 · critical or high finding: 20 · conversation from the adverse stratum: 10

Prompt

c1a9f0e3b2d4857609a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f607

Run

exemplo-run-0000-0000-000000000000

These parameters are what makes the result re-runnable. Re-running the same sample with the same specification either reproduces the scores or explicitly records where it diverged and why (a change of content, of rubric, of model, or provider non-determinism).

Failure patterns · 7.8

Patterns

6 grouped from the findings · 6 confirmed by a human · 1 discarded in review

Review

13 of 13 required instance(s) reviewed by an identified person

Human review is per PATTERN, not per occurrence: 100% of critical and high severity patterns, with min(5) and min(3) instances each, including the one with the lowest confidence and the one with the greatest dispersion across passes. Sixty occurrences of the same mechanism are one pattern, and reviewing all sixty would be redundant.

Annex · traceability matrix (5.2)

Each row links a clause of the client's internal rules to the criterion that verifies it and to the parameters used. It is the annex that answers "who defined these criteria?".

Issuance checklist · chapter 10

Every checklist item was satisfied at the moment of issuance.