Comparing Haiku, Sonnet, and Fable on a Real-World Task

I gave the same real task to three Claude models, Haiku, Sonnet, and Fable, then had Fable grade all three. Not a benchmark; an actual chore off my own machine. The scoreboard was interesting, but the thing I'll remember is which failure mode each model encountered.

The system being audited

I am a HUGE fan and evangelist for AutoHotkey! My Autohotkey scripts have grown organically over more than 20 years and on the one hand, they are a mess, but on the other hand, they are beautiful works of personal art. I use all the things in Autohotkey - text expansion, GUI manipulation, queries to REST APIs, etc. My whole system is based on custom defined actions that can be triggered in one or more of these three ways: a standard hotkey (Ctrl+Shift+F10 toggles a window's Always-on-Top state), an arbitrary three-character code (aot does the same thing, with no menu displayed), and a named entry in an Actions.txt file surfaced through a launcher that works like a local Spotlight.

The three tiers grew up in that order. Hotkeys came first, and worked fine until I had too many to remember. So I built the launcher for discoverability - it lists everything from opening a URL to running a git sequence to driving a full chain of GUI actions inside some app. Then I wanted to fire certain actions without even opening the launcher, which is where the three-character codes came in. Why three characters? Because two characters (1,296 combinations) is not quite enough, and three (36³ = 46,656) is more than I'll ever need.

The rule I am getting the AI to help me with is creating "parity": that is, every hotkey gets a matching three-character code and a documented launcher entry, so if I forget one path there's always another. The launcher ties them together. Type a couple of keywords and it shows me every way to fire the thing and reminds me of what crazy naming (that seemed so obvious to me when I created it) is actually in use.

That's the grand and glorious theory. But with hundreds of actions, things get confused easily: codes break, hotkeys get orphaned, launcher entries go bad. I have been "meaning to" clean all this up for years. Now, AI to the rescue: what was an annoyingly overwhelming task becomes doable (by it, not me!)

The Ask

Auditing cross-referenced consistency across hundreds of entries is tedious, mechanical, and exactly the kind of work AI should handle well. With discounted Fable access at the moment, it was also a chance to run three models against the identical prompt:

Each model ran independently. Then I had Fable evaluate all three outputs against each other - including its own.

Cost and speed

The "before" and "after" rows are running session totals; the task cost is the difference.

Haiku was fast and cost pennies - $0.46, against Fable's $7.31 for the same job. On paper that's insanely appealing. In reality, Haiku's output was confidently wrong (mansplaining?) in ways that would have quietly corrupted the audit, and in a code audit an invisible hallucination is the worst possible failure mode. Speed didn't track price either: Haiku was the fastest, and Sonnet - the middle option on cost - was the slowest on the clock.

Where each model broke

The scoreboard mattered less to me than how each one failed.

Sonnet was the best pure parser of the three. Its hotkey inventory was actually more complete than Fable's - it caught two real hotkeys Fable had dropped, which Fable later traced to a quoting bug in its own extraction regex. But Sonnet missed about a third of the cross-references, and it flagged none of the broken links: two codes that call actions which no longer exist sat in its table with blank action columns, indistinguishable from codes that legitimately have none. Good mechanical extraction, weak cross-linking, and honest about its own gaps.

Haiku didn't just miss things; it papered over them. It quietly redefined my evaluation rules so that three-character codes with no action counted as "OK," collapsing roughly 120 rule violations into a reported 3. It hallucinated a link to a timer action that doesn't exist. Sheesh. And its summary statistics contradicted its own table: it claimed 25 full-parity entries when the table showed about 5. Everything read clean and authoritative, which is scary.

Fable produced the only complete cross-reference, caught all four broken references, and included the 519-row action-only tail the other two skipped. It also missed two hotkeys on the first pass, which, surprisingly, Sonnet caught. But, it flagged and fixed that once it audited the group. Fable's own one-line summary is the sharpest thing to come out of the exercise: "Haiku's errors are confident and invisible, which for an audit is the worst failure mode."

The bug it caught

The most valuable outcome wasn't in any table. One hotkey's code comment, its documentation, and an on-screen reminder I'd refactored that same morning all cited the wrong key combination, one that actually belongs to a different tool. I only noticed because the three outputs disagreed on that row. Using one model to audit another surfaced a live bug that had survived weeks of daily use.

What I'd actually do now

This wasn't about crowning a best model. It's about matching failure mode to risk:

1. Choose Sonnet on mechanical extraction I'm going to review by hand anyway. It's accurate and it tells you where it's unsure.
2. Pay the big bucks (or usage, since I am on the Max plan) for Fable, or a multi-model grading pass, when an invisible error costs more than the API bill.
3. Keep Haiku away from structural audits unless there are deterministic checks sitting behind it to catch what it glosses over.

Since I never really hit my limits, I am going to continue to use the "adversarial" approach: point a second model at the first and make it argue. What fun. It cost more, but it found something weeks of daily use hadn't. If you're building LLM pipelines and trying to work out where human review actually belongs, that's the part worth stealing.