Four failures, observed

These four are not hypotheticals or cautionary tales collected from elsewhere. They happened during the build, in the order the steps were written, and each one is reproducible on your own machine.

They are the most valuable output of Part 2. The system producing three correct recommendations is unremarkable. The system producing three correct recommendations while the model was wrong four times is the entire argument.

1 · It invented a fact

Where: Step 2, before any tools existed. ALERT-003.

Asked to assess the alert with no tools, the model described COINFLARE as

a newly registered merchant

There is no registration date in the alert, in the fixtures, or anywhere else in the system. It was fabricated, in the exact register a real analyst would use, inside otherwise sensible analysis.

Why it matters: a fabricated fact is more dangerous than a refusal. “I don’t know when this merchant was registered” sends you to look it up. A confident invention flows into a case summary and gets read as established.

What fixed it: tools. Not a better prompt — the instruction in the finished system explicitly says “Do not invent any fact that is not in the case”, and that alone would not have prevented it.

2 · It skipped a tool

Where: Step 6. ALERT-001.

The enrichment instruction names three tools and says to call all of them. On one run the model called two, silently omitting get_transaction_history.

The score was still 100. extract_signals fell back to the customer profile’s avg_txn_amount, still derived above_average_amount, and the recommendation was unchanged.

Why it matters: this is a deterministic control absorbing a model failure, on the case where being wrong would cost most. If the model had been producing the score, that run quietly loses a signal and you never find out.

Be precise about it, though: the run had less evidence than the others. It degraded rather than failed. The audit trail records which tools actually ran, so the difference is visible to a reviewer — which is why you log tool calls and not just results.

3 · It narrated the case backwards

Where: Step 7. ALERT-002, then again on ALERT-003.

The Case worker described risk-reducing signals as evidence of high risk. On ALERT-003 it produced this:

…driven by positive signals from new_payee and elevated_mcc, offset slightly by negative signals from above_average_amount and home_location.

Three of the four signs are wrong, and above_average_amount is grouped with the wrong set entirely.

It then gave the correct recommendation, because the recommendation was never its to give.

Why it matters: this is the sharpest version of the whole architecture argument. The confusion is real and it is in the output — but it is confusion in a paragraph a human reads, not in a number a system acts on. Move the score into the model and this failure becomes the score.

4 · Identical runs called different tools

Where: Step 4 onward. Every alert, intermittently.

Same code, same alert, same instruction: two tools on one run, three on the next. Nothing changed but the sampling.

Why it matters: it is the clearest possible demonstration that “I ran it and it worked” is not evidence about an agent. It is one sample from a distribution you have not measured.

This is also the failure that makes the other three inevitable rather than unlucky. A system built on a non-deterministic component will exhibit every one of its failure modes eventually. You are not designing to prevent them. You are designing so that when they happen, nothing important depends on them.

The pattern across all four

Each failure lands somewhere the system had already decided the model’s output was advisory rather than authoritative:

Failure	Landed in	Consequence
Invented merchant fact	Free-text analysis, pre-tools	Removed by grounding
Skipped a tool	Missing enrichment	Absorbed by a fallback
Backwards narration	The written summary	A wording defect
Non-determinism	Which tools ran	Logged, visible

None of them reached the score, the recommendation, or the action — because none of those are the model’s to produce.

That is the takeaway from both parts of this course, stated as plainly as it goes:

You will not make the model reliable. You decide, in advance, what its unreliability is allowed to touch.

Try to reproduce them

Failures 3 and 4 show up within a handful of runs:

cd part2
for i in 1 2 3 4 5; do uv run --with ollama run.py | grep -c tool_call; done

Different counts across runs is failure 4. Read any case summary closely for failure 3.

Failure 1 needs step 2’s code — git switch --detach to that commit, per replaying steps.