8 · The audit trail

What this step adds: an append-only record of every step, tool call, result, score and decision. Seventy events for three alerts.

The new idea

The docstring says it better than a definition would:

In agent terminology this is the “trace” or “trajectory”; in bank terminology it is the audit trail your Head of Fraud shows a regulator. Same artefact.

That equivalence is worth sitting with. The observability feature an AI team builds to debug their agent and the compliance artefact a bank is required to produce are the same file. One team calls it tracing, the other calls it evidence. If you build it once, deliberately, it serves both.

The code

class AuditTrail:
    def __init__(self, path, echo=True):
        self.path = path
        self.echo = echo          # also print to the console as we go
        self.entries = []

    def record(self, alert_id, stage, event, detail):
        """Append one immutable audit event."""
        entry = {
            "timestamp": datetime.now().isoformat(timespec="seconds"),
            "alert_id": alert_id,
            "stage": stage,
            "event": event,
            "detail": detail,
        }
        self.entries.append(entry)
        with open(self.path, "a") as f:
            f.write(json.dumps(entry) + "\n")
        if self.echo:
            self._print(entry)

Five fields, "a" for append, one JSON object per line. Same shape as Part 1, with two additions that matter at this size.

stage makes failure locatable. Every event is tagged with which of the five stages produced it. grep '"stage": "ENRICHMENT"' gives you every tool call the enrichment worker made across every alert. Without it you have seventy undifferentiated lines.

echo makes it watchable. The same events print as aligned columns while the run happens:

ENRICHMENT  | tool_call      | get_customer_profile({'customer_id': 'CUST-1001'})
ENRICHMENT  | tool_result    | {"home_country": "AU", "avg_txn_amount": 2100.0, ...}
RISK        | signals        | new_device, cross_border, high_value, new_payee
RISK        | score          | 100/100 (HIGH, confidence high)
HUMAN_GATE  | human_decision | APPROVE_RECOMMENDATION (agent recommended ESCALATE)

Being able to watch an agent work — rather than see its output — changes how quickly you find problems. Every failure documented in this course was found by reading that stream, not by inspecting a result.

What gets logged, and why each

Event	Why it’s there
stage_start	Which stages ran, and in what order
tool_call	What the model asked for, including its arguments
tool_result	What it actually got
signals	The facts the score was computed from
score	The number, band and confidence
recommendation	What the system proposed
human_decision	What the person decided, and why if they overrode

Logging both tool_call and tool_result is the one people skip, and it is how you catch two things nothing else will: a tool the model asked for and never got, and the gap between the arguments it produced and the arguments _bind_args substituted.

The gap this still has

The record does not say who approved.

"by": "analyst" is a placeholder. In production that must be an authenticated user identity, and it is the single most important missing field — “approved by” is the entire point of an approval, and under an accountability regime an unattributed sign-off is not a control at all.

It stays as a placeholder here because faking authentication in a teaching prototype teaches nothing. But do not ship it.

Run it

cd part2
uv run --with ollama run.py

TRIAGE RUN COMPLETE
  ALERT-001  ->  ESCALATE  (risk 100/100, confidence high)
  ALERT-002  ->  CLEAR     (risk   0/100, confidence high)
  ALERT-003  ->  MONITOR   (risk  35/100, confidence low)

  Full audit trail written to: audit_log.jsonl
  (70 events logged)

Then read the file:

python3 -c "
import json
for line in open('audit_log.jsonl'):
    e = json.loads(line)
    if e['alert_id'] == 'ALERT-001':
        print(f\"{e['stage']:<11} {e['event']:<16} {str(e['detail'])[:70]}\")
"

That output is the deliverable. Not the recommendation — the reconstruction. It is the difference between “the AI flagged it” and a record showing the alert, every tool call and what it returned, the signals extracted, each weight that contributed, the resulting score, the recommendation, and who decided what at which second.

One of those survives a regulator.

Where this leaves you

Three alerts, three correct outcomes, and — more usefully — four documented failures in a system that produced correct outcomes anyway. That combination is the thing to take away.

Next: the four failures collected, then replaying any step from git.