Ask before you build

How Ponderly, an open source Claude Code plugin, keeps coding agents asking instead of guessing. Covers the hooks, routes, sub-agents and reviewers that make it work, and what its evals and a public code review benchmark say about whether it does.

article · updated 8 Nov 2026
~/dansabin.com — chat Ask about this project

Why does your coding agent ask so many questions?

Because a question costs one answer, and a wrong guess costs a review round or a bad deploy. It asks less than you'd think. On this site, 43 of 2,529 agent runs stopped to ask, about one in sixty, and of 173 pull requests merged, half went from opened to merged in 18 to 34 minutes, with a median of 20. Ponderly is what makes it ask.

What happens when you and the agent agree on something that isn't true?

One day I was building the part of this site's chat window that tells a visitor whether the AI is switched on. The session's report on a batch of work said that when I switch the AI off, the API answers with "the kill switch's deliberate 503". I took it and ran: if the 503 shows up first, I said, the page could tell visitors chat is down before they ever hit send. It sounded right to both of us, and the session wrote it into the next two tasks and started six agents.

About two minutes later, the first of them was back without having written a line: "The task says GET /status answers a 503 while the kill switch is on. The API never does that." It had read the code. That route answers 200 either way, with the switch as a field in the reply; its 503s mean the site is down.

The session wrote back, "thank you for catching my wrong premise," fixed both tasks and corrected the second agent while it was still working. Forty-seven seconds after the stop the batch was back on course, and the change merged that same day.

Where the wrong 503 is caught Two paths from the same wrong premise. Caught before the code, an agent reads the code, stops before writing anything, and the two tasks are fixed to read the switch from the reply. Caught later, the tests pass because they were built on the same 503, and the first time the kill switch is on, no "AI is paused" line shows. CAUGHT BEFORE THE CODE A report says "a deliberate 503" Two tasks six agents start An agent reads code stops before writing Tasks fixed reads the switch CAUGHT LATER A report says "a deliberate 503" Two tasks six agents start Tests pass built on the 503 Kill switch on no paused line Where the wrong 503 is caught Two paths from the same wrong premise. Caught before the code, an agent reads the code, stops before writing anything, and the two tasks are fixed to read the switch from the reply. Caught later, the tests pass because they were built on the same 503, and the first time the kill switch is on, no "AI is paused" line shows. CAUGHT BEFORE THE CODE A report says "a deliberate 503" Two tasks six agents start An agent reads code stops before writing Tasks fixed reads the switch CAUGHT LATER A report says "a deliberate 503" Two tasks six agents start Tests pass built on the 503 Kill switch on no paused line
Two paths from the same wrong premise. Caught before the code, an agent reads the code, stops before writing anything, and the two tasks are fixed to read the switch from the reply. Caught later, the tests pass because they were built on the same 503, and the first time the kill switch is on, no "AI is paused" line shows.
Where it's caughtWhat it costs
Before the code, when the agent reads itOne question and two edited tasks. Under a minute.
In the pull requestA reviewer who knows every answer the route can give. The tests pass, because they were written from the same premise.
The first time I switch the AI offNo "AI is paused" line: visitors learn chat is off only when a send is refused. And a real outage reads as a pause.

Why it stopped

Each task goes to a fresh agent that starts from the task and the code, and is told to work from what the code says. The conversation the premise grew up in isn't in its context, but the code is. When the two disagree, it's told to stop and quote the code rather than work around it.

Does it work?

The number I watch is how long a change takes to reach a pull request that merges cleanly, and how wide that range is. On this site, the middle half of 173 merged pull requests took 18 to 34 minutes, and none has been reverted. The 503 wasn't a one-off: the reviewer that reads each batch of work has found 235 problems it graded as blockers, across 410 batches, each while the code was still being written. And on a public code review benchmark, against someone else's answer key, the review finds about 59% of the known problems.

I don't have a before figure from this site to set against those. A team can get one: count time to merge, how widely it ranges, and reverts from its own history, then count again after.

What it costs

A reviewer agent for every batch of work, six more agents for a pull request review, and my attention: if I answer yes to everything, the questions catch nothing. And an agent only catches a premise the code contradicts; the rest goes to the reviewer, and then to me.

For reference

How the rules arrive. Hooks, scripts Claude Code runs at set moments, load the six rules into every session and sub-agent, bring them back after a compaction, which is when Claude Code summarizes a long conversation to free up room, and bring the rule for a file back when an agent reads or writes one. Claude Code cuts each hook's output off at 10,000 characters, so a check in CI measures the rules on every pull request, and adding a sentence to one rule means cutting one from another.

The front door. start-task reads the code a request touches and picks a route: a bug goes to bugfix, which reproduces it first; a one-sentence change with no design decision goes to one executor; everything else gets requirements, a design and a task list, each approved before the next. A one-shot prompt is golf played between planets: a nanometer off at the tee and you miss Jupiter. The three documents get you to orbit first.

A separate reviewer. implement-spec builds a spec in waves, batches of tasks that can run at once, with a fresh task-executor per task. A read-only wave-reviewer grades each wave's problems blocker, should-fix or nit while the full suite runs. Questions come in the customer's language, what a visitor will see or what an operator will deal with at 2 a.m., not which library to call.

Tests and review. One test per code path, one input per test, boundaries over tables, and contract values written as literals. The review skill reads a pull request for correctness, reliability, security, maintainability, documentation and testing at once, and the wave reviewer checks the merged list.

On this site. Between October 19 and November 2, this site merged 54 pull requests built this way and Ponderly merged 17. In that time I haven't had to tear out and rewrite a section of the codebase.

A public code review benchmark. 25 real pull requests with someone else's answer key, run in batches of five. The first run found a third of the known problems. The latest runs find about 59%, 88% of the findings that match the answer key are real, and every high-severity finding it raised was real. It found failures too: seven false findings on formatting-only edits, and five sessions that finished and then hung on a wakeup timer they never canceled. Both became specs and were fixed.

The rule evals. These ask a narrower question: does a rule change behavior at all? Each case gives Claude a written task, and graders, checks that each pass or fail one thing it should have done, score the result. A score is the share of graders passed, from 0 to 1, averaged over three runs with the plugin and three without, on November 3, 2026. Difference is the plugin's score minus plain Claude's, and Cases is how many tasks each rule was tested on. Plain Claude was never told to work this way and the graders encode the rules' own preferences, so this isn't a benchmark. The evidence page has every case.

RuleCasesWith pluginWithoutDifference
Collaboration20.930.66+0.27
Communication70.970.81+0.15
Code changes50.970.68+0.28
Tests21.000.54+0.47

On leading with what the user sees, the plugin scored what plain Claude did, 0.78, and plain Claude now asks from a short list of options almost as well.

What it costs in dollars. A review of one large pull request ran through its $8 budget and was rerun at $12; another took seven minutes and about $3. I don't measure tokens per pull request yet.

Not yet proven. The fair test of ask instead of guess, with requirements left out and hidden tests deciding who built the right thing, hasn't been run. Neither has a long-session replay of the reminders, so they're still a bet.

Bringing AI to a team. Start with a bug, and watch where the agent stops to ask. Then hand it a problem you don't know how to solve, as a rubber duck that has read everything. And keep a person at the wheel, fresh enough to catch the mistake on the fourth check instead of the fortieth.

Want the details?

Ask the window anything this page skips.