Connect an agent, send transcripts, and read the A–F report — via SDK or REST.
Overview
Axon is the quality layer for AI agents. You send conversation transcripts; Axon evaluates each one with an LLM judge against a configurable rubric and returns an A–F report per dimension — with cited evidence, root cause and a concrete fix.
The flow: connect an agent → ingest transcripts → the judge evaluates (asynchronously, on a durable idempotent queue) → read the report → fix → lock it into a regression set → monitor and get degradation alerts.
Quickstart
From zero to your first report in three steps.
- Generate an API key in Settings → Integration. Store it as AXON_API_KEY.
- Connect your AI provider key (BYOK) in Settings → Integration → AI provider. Evaluations run on this key.
- Install the SDK (pnpm add @gabrielonrails/axon-sdk) or use plain cURL.
- Send a transcript and read the report. The agent is created on first ingest.
$ pnpm add @gabrielonrails/axon-sdk
import { Axon } from "@gabrielonrails/axon-sdk";
const axon = new Axon({ apiKey: process.env.AXON_API_KEY! });
await axon.ingest("cobranca", [{
id: "conv-001",
channel: "voz",
turns: [
{ role: "client", text: "Quero um desconto." },
{ role: "agent", text: "Posso aplicar 20% agora." },
],
}]);
const report = await axon.getReport("cobranca");
console.log(report.overall, report.dimensions);
BYOK — your AI key
Evaluation runs on the AI provider key you connect — Anthropic, OpenAI or any OpenAI-compatible endpoint. It is required: without it the judge does not run and the report stays empty.
Where to connect: in the app, Settings → Integration → AI provider. Paste the key, pick the provider and (optionally) the model. The key is encrypted at rest (AES-256-GCM).
Authentication
All API requests authenticate with an org-scoped API key in the Authorization header: Authorization: Bearer axon_sk_….
Keys are generated and revoked in Settings → Integration. Only a SHA-256 hash is stored — the full key is shown once. Revoking is immediate.
Authorization: Bearer axon_sk_xxxxxxxxxxxxxxxxxxxxxxxx
Node SDK
The official @gabrielonrails/axon-sdk is typed, zero-dependency (native fetch, Node 18+), idempotent and retries transient errors automatically.
Set id on each transcript: re-sending the same id updates the conversation instead of duplicating — safe for retries and replays.
Constructor options
new Axon({
apiKey: string, // axon_sk_... (obrigatório)
baseUrl?: string, // default: https://axon-dev.com
maxRetries?: number, // default: 3 (429/5xx/rede, backoff)
timeoutMs?: number, // default: 30000
fetch?: typeof fetch, // fetch custom (opcional)
})
baseUrl defaults to the current app URL and is overridable; point it at your own domain or a self-hosted deployment.
Methods
axon.ingest(agent, transcripts) // → AxonIngestResult
axon.getConversation(agent, externalId) // → AxonConversationStatus
axon.getReport(agent) // → AxonReport
axon.runRegression(agent, { version? }) // → AxonRegressionResult (gate de CI)
axon.waitForEvaluation(agent, externalId, { timeoutMs?, intervalMs? })
axon.agent(name) // → { ingest, getReport, getConversation, waitForEvaluation }
Types
interface AxonTranscript {
id?: string; // chave de idempotência (reenviar = atualiza, não duplica)
channel?: "voz" | "chat" | "whatsapp";
turns: {
role: "client" | "agent" | "tool";
text: string;
// tool calls deste turno → habilitam a avaliação de TRAJETÓRIA do agente
toolCalls?: { name: string; arguments?: unknown; result?: unknown }[];
}[];
// Trace de agente (opcional) — quanto mais rico, mais profunda a avaliação:
system?: string; // prompt de sistema do agente (aderência/trajetória)
context?: string[]; // contexto recuperado (RAG) → tríade faithfulness
metadata?: Record<string, unknown>; // metadados livres (persistidos)
// Metadados de filtro/agrupamento (não afetam a nota):
duration?: string; // duração da conversa, ex.: "4m12s"
version?: string; // rótulo da versão do agente (alimenta a comparação A/B)
result?: string; // desfecho, ex.: "resolvido" | "transferido" (outliers
// sempre avaliam, mesmo sob amostragem)
queue?: string; // fila/operação, ex.: "financeiro-cobranca"
}
interface AxonConversationStatus {
// "blocked" = sem assinatura/crédito (pague-antes); reason explica o porquê.
status: "pending" | "running" | "done" | "failed" | "blocked" | "unknown";
reason?: "no_entitlement" | "llm_not_configured" | "evaluation_failed" | null;
evaluation: { overall: Grade; confidencePct: number;
dimensions: { key; name; grade; score }[] } | null;
}
Real-time monitoring
Axon monitors your agents in streaming mode — you don't export a batch at the end of the day. The pattern is a conversation-end hook: as soon as a conversation finishes (the caller hangs up, the chat session closes, the ticket is resolved), send the transcript. Evaluation runs asynchronously and the dashboard updates on its own (the screen shows a "Live" indicator and refreshes periodically).
Plug the hook wherever conversations end: a call.completed webhook from your telephony, the chat/WhatsApp session timing out, your own AI agent right after the last message.
// Chame no momento em que a conversa encerra (fim de chamada,
// sessão de chat fechada, ticket resolvido). Não bloqueie o
// encerramento esperando a Axon — o SDK já faz retry com backoff.
async function onConversationEnded(conv) {
void axon.ingest(conv.agentName, [{
id: conv.id, // mesmo id em retries → não duplica
channel: conv.channel, // "voz" | "chat" | "whatsapp"
turns: conv.turns,
}]).catch((err) => console.error("[axon] ingest falhou:", err));
}
Because each conversation is evaluated the moment it arrives, Axon detects degradation in near real-time — it fires alerts (email/webhook) when a conversation scores low (D/F) or the agent's recent average drops versus the prior window, without waiting for a weekly report.
Without the SDK
The hook is just an HTTP POST — no SDK required. Call the ingestion endpoint straight from your backend or even from your telephony/chat provider's webhook. The id keeps retries idempotent.
# Dispare isto no fim da conversa, do seu backend ou direto do
# webhook da telefonia/chat. Sem SDK — só HTTP.
curl -X POST https://axon-dev.com/api/v1/agents/cobranca/transcripts \
-H "Authorization: Bearer $AXON_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "transcripts": [
{ "id": "conv-001", "channel": "voz",
"turns": [ { "role": "client", "text": "..." },
{ "role": "agent", "text": "..." } ] } ] }'
# → 202 { "accepted": 1, "status": "evaluating" }
# Reenviar o mesmo "id" é idempotente: atualiza, não duplica.
REST API
Three endpoints, all authenticated by API key. Ingestion returns 202 and evaluates asynchronously; poll the status endpoint or read the report.
Base URL: https://axon-dev.com
POST /api/v1/agents/{agent}/transcripts
Authorization: Bearer axon_sk_...
Content-Type: application/json
{ "transcripts": [
{ "id": "conv-001", "channel": "voz",
"turns": [ { "role": "client", "text": "..." },
{ "role": "agent", "text": "..." } ] }
] }
→ 202 { "accepted": 1, "queued": 1, // queued < accepted sob amostragem
"agent": { "id", "name" },
"conversation_ids": ["..."], "status": "evaluating" }
GET /api/v1/agents/{agent}/transcripts/{externalId}
Authorization: Bearer axon_sk_...
→ 200 { "status": "done",
"evaluation": { "overall": "C+", "confidencePct": 92,
"dimensions": [ { "key", "name", "grade", "score" } ] } }
GET /api/v1/agents/{agent}/report
Authorization: Bearer axon_sk_...
→ 200 {
"agent": { "id", "name", "channel" },
"overall": "C+", "degrading": true, "sample": 2430,
"dimensions": [...],
"problems": [
{ "severity", "title", "dimension", "impact", "rootCause",
"evidence": { "turnIndex", "quote", "verified" } } // verified = citação conferida
],
"trajectory": { // quando há tool calls: qualidade do uso de ferramentas
"assessed": true, "score": 78, "summary": "...",
"findings": [ { "type": "tool-correctness", "severity", "detail" } ]
},
"rag": { // quando há contexto recuperado: tríade RAG
"assessed": true, "faithfulness": 64, "answerRelevance": 90,
"contextRelevance": 71, "summary": "...", "findings": [...]
},
"correction": { // a correção recomendada (o diferencial)
"tags": ["política de desconto", "concessão não autorizada"],
"impactFrom": "C+", "impactTo": "A-",
"fixes": [
{ "type": "PROMPT", "title": "...", "description": "...", "code": "..." },
{ "type": "GUARDRAIL", "title": "...", "description": "...", "code": "..." },
{ "type": "EVAL", "title": "...", "description": "...", "code": "..." }
]
} // null quando ainda não há correção
}
The report also returns `correction` — the recommended fix (root cause + concrete PROMPT/GUARDRAIL/EVAL patches), or null when none has been generated yet.
Regression & golden sets
Lock quality in: turn an evaluated conversation into a golden case (the expected outcome). The regression suite re-checks those cases on every deploy.
The cycle: (1) flag a case from a report/correction; (2) run the suite; (3) turn on the CI gate — a deploy that regresses a case is blocked, with an audited bypass approval.
Today the regression cycle is operated in the app (Team plan) and cases are created from flagged conversations. A dedicated REST endpoint for regression is on the roadmap.
CI gate (block bad deploys)
Run your regression suite from CI and fail the build if the agent regressed. The endpoint returns HTTP 200 when it passes and 422 when it regressed, so curl --fail gates automatically.
GitHub Actions example:
name: Axon quality gate
on: [pull_request]
jobs:
gate:
runs-on: ubuntu-latest
steps:
- run: |
curl --fail -sS -X POST \
-H "Authorization: Bearer ${{ secrets.AXON_API_KEY }}" \
-H "content-type: application/json" \
-d '{"version":"${{ github.sha }}"}' \
https://axon-dev.com/api/v1/agents/YOUR_AGENT/regression
Or with the SDK:
const r = await axon.runRegression("your-agent", { version: sha });
if (!r.gate) process.exit(1); // bloqueia o deploy se regrediu
Requires the Pro plan or above. Each case is re-evaluated with your BYOK key.
FinOps & cost
Every evaluation records the real tokens consumed on your key (BYOK). The FinOps dashboard shows estimated cost and tokens by agent, model and day, with a monthly budget and an alert when it's exceeded.
Cost visibility lives in the app at /finops. A REST endpoint to read cost/tokens is on the roadmap.
Rubric & grades
Each conversation is graded A–F overall and per dimension (0–100 score). A/B = healthy, C = attention, D/F = critical. The rubric is configurable per agent; the default has six dimensions:
A/B healthy · C attention · D/F critical
Alerts & webhooks
Axon computes a weekly baseline per agent and fires a degradation alert when the grade drops vs. the previous week.
Alerts are delivered in-app and to the channels you configure in Settings → Integration: webhook (POST JSON) and Slack (incoming webhook).
POST (seu endpoint)
{ "source": "axon", "severity": "alto",
"title": "Queda de qualidade em Cobrança",
"body": "A nota geral caiu 9 pontos vs. a semana anterior." }
Limits & contract
What enterprise engineering asks on the first call:
- Evaluation throughput is rate-limited per org: 3 requests/min to the judge, ~8k input and ~2k output tokens per minute. Overflow is queued (FIFO). Ingestion returns 202 immediately; evaluation runs asynchronously.
- Transient errors (429/5xx/network) retry automatically in the SDK with exponential backoff.
- Recommended ≤ 50 conversations per request and ≤ ~100 turns per conversation. Smaller, frequent batches beat one giant batch.
- An id per conversation makes re-sends idempotent (updates, never duplicates).
- The report returns the 3 most relevant problems per agent (no pagination).
Errors
Errors return a JSON body { error, message } with the appropriate HTTP status.