When it breaks
Everything on this page actually happened while this course was being built. None of it is hypothetical, and none of it was edited out — because each failure teaches something the working code can’t.
1. The model that ignored its own tool results
Section titled “1. The model that ignored its own tool results”The first version of this course used llama3.2 (Meta’s 3B model). It emitted tool calls perfectly — correct function, correct arguments. Then the tool result came back, and the model replied:
“I don’t have real-time access to account information, and I couldn’t retrieve the data. The previous response was an error on my part.”
It was handed the answer it asked for, and disclaimed it.
The diagnosis matters more than the bug. The method:
- Reproduce — run it three times. Failing every time means a cause, not noise.
- Isolate one variable at a time. Message format? Changed it — no effect. Subject domain (banking vs neutral warehouse data)? No effect. Feeding the identical text back as a
usermessage instead of atoolmessage? Worked instantly. The model mishandles thetoolrole specifically. - Swap the model, measure again. Same code, same task, five runs each:
| Model | Size | Correct |
|---|---|---|
| llama3.2 | 3B | 0/3 |
| qwen2.5:3b | 3B | 3/3 |
Three lessons that generalise:
- “Supports tool calling” is not a yes/no property. It’s at least two skills — deciding to call and using what comes back — and a model can have the first without the second. No vendor capability table will tell you this.
- Size is not capability. Two models, identical parameter count, opposite results.
- Never theorise before printing. The first hypothesis (“the message format is wrong”) was wrong, and only the data said so.
2. The eval that turned 100% into 20%
Section titled “2. The eval that turned 100% into 20%”An earlier categorising agent looked at a deliberately suspicious transaction — a cryptic merchant, ZK*TRDNG LTD — and labelled it correctly. Demo complete, success declared.
Then the same test was run five times:
| Case | Correct |
|---|---|
| Clean grocery transaction | 5/5 |
| Rideshare transaction | 4/5 |
| The suspicious one | 1/5 |
On four of five runs the suspicious transaction got a different, wrong category — even though the lookup tool returned the correct answer in every single run. The model paraphrased away the one fact that mattered.
Three lessons:
- One passing run proves nothing. Models are non-deterministic; a demo is a coin that happened to land heads.
- Never accept aggregate accuracy where cases matter unequally. 10/15 overall sounds fine. 1/5 on the case the system exists for is unfit for purpose.
- The fix was not a better prompt. The category was already in the tool’s return value — the fix was to stop asking the model to restate a fact the code already held, and parse it deterministically. After that: 5/5, 5/5, 5/5. Don’t ask a model to relay a fact you already have.
3. “Your card has been blocked”
Section titled “3. “Your card has been blocked””A support-ticket router built during the same course handled a fraud report. Its generated reply to the customer:
“We have temporarily blocked your card for security reasons. A fraud specialist will contact you shortly.”
Nothing was blocked. No tool ran. No system was touched. The model wrote a sentence describing an action that never happened — and the pipeline would have sent it to a customer.
This isn’t a model error. The model was asked to write a reassuring reply and did. The failure is architectural: the system let generated text assert facts about the world. Text is cheap to generate and looks identical whether or not the world agrees with it.
The fix is the shape of steps 6 and 7: generated text never describes actions; actions happen in code, behind a human gate, and the record of the action — not the model’s description of it — is what gets communicated.
The meta-lesson
Section titled “The meta-lesson”All three failures share a shape: the system looked like it worked. Correct-looking tool calls, a passing demo, a fluent reply. Each was one unmeasured step from production.
The countermeasures are boring, and that’s the point: run it more than once, score the case that matters, never let prose stand in for state. If you take one habit from this course, take that one.