Skip to content
Dustin's AI Lab
Go back

Instead of Begging the Model Not to Lie, I Wrote a Hook That Stops It

The sequel to the Opus 4.8 confabulation post: I moved "don't make up numbers" from a plea in CLAUDE.md to a pending-guard hook that blocks git commit at PreToolUse.

Last time I wrote about Opus 4.8 fabricating tool output four days in a row. That post was about the symptom. This one is about what I did afterward.

I spent some time researching how to defend against Opus 4.8 faking tool output and poisoning context with hallucinations, and I confirmed one thing empirically: a rule written in CLAUDE.md is only a probabilistic constraint on the model. You write the prohibition, and it can still ignore it. The thing that actually holds is a hook—because a hook is code, not something the model chooses whether to obey.

First I added a line of principle to the “factual claims” section of task-execution.md: numbers and conclusions must point to a source tool_result; when parallel results aren’t all in yet, mark a ⏳ pending and don’t fill in a guessed value. That line is for the model to read. It lives in the probabilistic layer.

The deterministic layer is this hook. I added ~/.claude/hooks/pending-guard.sh, wired to the Bash matcher at PreToolUse. The logic is simple: if the staged diff still contains an unresolved ”⏳ pending” marker, block git commit. If that pending is meant to land in the repo as-is, the commit message has to contain [pending-ok] to pass.

Before turning it on I spun up a temporary test repo and ran four cases: pending with no pass marker (should block), pending with a pass marker (should pass), no pending at all (should pass), pending already resolved (should pass). All four checked out, and only then did I register the hook on the Bash matcher in settings.json.

Begging the model not to lie is a probability no matter how many times you say it. Write it as a hook, and it can’t get past that commit.


Share this post on:
Previous Post
Two Traps Running Claude Code on Local Ollama: Truncated Context, and A3B Buckling Under Heavy Verification
Next Post
Second Harness Diet: Global Skills From 58 Down to 40