6 · Risk scoring in code

What this step adds: the risk score. It is computed by explicit weights in plain Python, and no model touches it.

The new idea

This is the step Part 1 was arguing toward.

The obvious design is to hand the model the enrichment results and ask for a score out of 100. It would work, in the sense that a number would come back. And it would be unusable in any setting where the number has consequences, for four reasons:

• It moves. Same alert, same facts, different number — you saw a 1-in-5 category result in Part 1.
• It can’t be explained. “Why 72?” has no answer beyond “the model said so”.
• It can’t be tested. You cannot unit-test a weight that doesn’t exist.
• It can’t be changed deliberately. Ask Fraud Ops to make new devices count for more, and your only lever is rewording a prompt and hoping.

The code

WEIGHTS = {
    "new_device": 35,
    "cross_border": 25,
    "high_value": 20,
    "above_average_amount": 10,
    "new_payee": 15,
    "contact_changed": 20,
    "elevated_mcc": 20,
    "known_device": -10,
    "home_location": -10,
}

HIGH_BAND = 65    # >= this  -> HIGH   -> ESCALATE
MEDIUM_BAND = 35  # >= this  -> MEDIUM -> MONITOR   (below -> LOW -> CLEAR)

That dictionary is the bank’s risk appetite, written down. It is nine lines, anyone can read it, and changing the policy means editing a number in a diff someone reviews.

Scoring is two functions with no cleverness in them:

def score_risk(signals, similar):
    """Compute a 0-100 risk score with a reason for every contribution."""
    score = 0
    reasons = []
    for sig in signals:
        pts = WEIGHTS.get(sig, 0)
        if pts:
            score += pts
            reasons.append(f"{'+' if pts > 0 else ''}{pts}  {sig}")
    ...

Every contribution produces its own sentence. Not a total — a list:

+35  new_device
+25  cross_border
+20  high_value
+15  new_payee
+20  contact_changed

Six months from now, someone asking why ALERT-001 escalated gets that list. They do not need the source, the model, or the person who built it.

The graceful-degradation detail

One line is worth pointing at:

avg = max(profile.get("avg_txn_amount") or history.get("avg_amount") or 1.0, 1.0)

Two .get() calls, a fallback chain, and a floor.

On one run while this was being built, the model skipped get_transaction_history entirely — the instruction says to call all three tools, and it called two. In a system where the model produced the score, that run silently loses a signal.

Here, extract_signals falls back to the profile’s average, still computes above_average_amount, and ALERT-001 still scores 100. The comment above it says so out loud:

# .get defaults so a model that skips a tool degrades gracefully rather
# than crashing — missing context simply yields fewer signals.

That is a deterministic control absorbing a model failure, live, on the case that matters most. It is the best single argument in the codebase for the architecture.

Be honest about what it is, though: degradation, not a fix. A run with two tools has less evidence than a run with three. It happened to reach the same answer here. The audit trail records which tools actually ran, so a reviewer can see the difference — which is the point of logging tool calls rather than just results.

Run it

cd part2
uv run --with ollama run.py

RISK        | signals        | new_device, cross_border, high_value, new_payee, contact_changed
RISK        | score          | 100/100 (HIGH, confidence high)
RISK        | recommendation | ESCALATE

Run it ten times. The number does not move. The wording around it will; 100 will not.

What’s still wrong

You have a defensible number and nothing that explains it to a human. The last model in the system writes that explanation — and gets it backwards.