How Ponderly was built

Ponderly was built out of dogfood. Why communication, coding and testing are the hardest rules to keep alive, what using the plugin on this site changed, and what I can't prove about the weight of a word.

case study · updated 8 Nov 2026
~/dansabin.com — chat Ask about this project

When an agent tells you what it did, how do you know?

You check what it touched, not only what it says. In Ponderly, every executor now lists every path it deleted, and the session that launched it checks that list before it uses anything else. That rule, like a lot of the plugin's tuning, came from using it on this site, where I've merged more than 100 pull requests built this way.

What does a rule look like when it comes from a real run?

On one tighten-tests run on this site, the review sent a small fix back to a task executor: back up one source file, break it to prove a test catches the break, put it back, and delete the copy. The executor did all of it, and the test did catch the break. Then it cleaned up the folder the copy sat in, which was also the launcher's own run folder, and reported, "Then I deleted the copy and its directory."

From where it stood, that was true. No brief had said which folder was its own. The launcher found out at its next step, when the records it meant to compare against were gone: "the recorded starting state", the audit's records and the mutation results.

It spent the next few minutes rebuilding the proof another way, and the original 1,031 tests and the slimmed suite turned out to cover the same 1,913 lines.

I filed the issue and worked on the fix. Now every launch names one folder that is the executor's own, and its summary lists every path it deleted. The launcher reads that list the moment the executor hands back, before it trusts its own records.

When it's noticedWhat it costs
As the executor hands back, from the paths it lists as deletedOne question, before anything else runs.
At the launcher's next stepA few minutes rebuilding the coverage and source checks another way, and gap notes written from the conversation instead of the audit's records.

The number I watch here is how many steps a wrong report survives before something checks it. That time it survived one. Now it survives none. An eval replays the incident to check the rule holds, and its one run so far passed.

A stranger's fix

About four hours after I filed it, someone outside the project opened a pull request with a fix, about 27 minutes before my own commit. I saw it only when I went to merge mine. They had found the same bug, and part of their change closed a gap mine left. I didn't want to close their work, so I thanked them and asked them to narrow it to that part. It's still waiting.

Why rules fade

The rules hardest to keep alive are communication, coding and testing, and the reason is how rules load.

Rules you maintain on your own work get help from Claude Code: its docs describe files that load at launch, load when a matching file is touched, and come back after a compaction. A plugin gets none of that. Its rules arrive only through its hooks, and what a hook added earlier is summarized away with the rest of the conversation. Sub-agents generally do well, because they start with the rules and run for short sessions. Long sessions are where rules fade.

Where the hooks bring the rules back Ponderly's rules arrive only through its hooks. Across one session, the hooks bring them back at four moments: the full rules at the start, the rule for that kind of file when a file is read or written, a short pointer with each message I send, and the rules again after a compaction. Each sub-agent starts with the full rules too. Each hook's output stays under 10,000 characters. Ponderly plugin six rules, through its hooks each output under 10,000 characters EACH SUB-AGENT At the start full rules, short session At the start full rules At the file rule for that kind of file At each message short pointer After a compaction rules load again A SESSION, AS IT GETS LONG Where the hooks bring the rules back Ponderly's rules arrive only through its hooks. Across one session, the hooks bring them back at four moments: the full rules at the start, the rule for that kind of file when a file is read or written, a short pointer with each message I send, and the rules again after a compaction. Each sub-agent starts with the full rules too. Each hook's output stays under 10,000 characters. Ponderly plugin six rules, through its hooks output under 10,000 characters A SESSION, AS IT GETS LONG At the start full rules At the file rule for that kind of file At each message short pointer After a compaction rules load again EACH SUB-AGENT At the start full rules, short session
Ponderly's rules arrive only through its hooks. Across one session, the hooks bring them back at four moments: the full rules at the start, the rule for that kind of file when a file is read or written, a short pointer with each message I send, and the rules again after a compaction. Each sub-agent starts with the full rules too. Each hook's output stays under 10,000 characters.

So the hooks bring the rules back at the start, at each file, after a compaction, and at each message I send, which carries a short pointer to the line agents miss most: "Lay out a problem, an explanation or a decision as the person it changes will live it."

The evals don't cover the reminders yet. They start fresh sessions, where reminders have nothing to fix. Replaying long sessions with and without them is an open issue, so some of this is still a bet.

What using it changed

Some of this I brought from Amazon, where we were using AI for coding. On this site, three things changed.

Talking back. Early on, my own note on it was that it seems to communicate in a sub-optimal way that hides important details and mentions unimportant points. That note became the report skill. Then I asked for explanations from three points of view: the owner, the user and the operator. I'd asked for that view eight times across two repositories, and every reply written after I asked was accepted first time. It became part of the communication rule.

It still isn't solved. The longer a conversation goes, the more I have to harp on it. On the one eval that tests leading with what the user sees, the plugin ties plain Claude at 0.78.

Testing and coding. Agents tend to write too many unit tests. I had preferences for this in the site's memories, so I had a site session dispatch the work to the plugin's own workspace. Those preferences became the tests rule and the readable-code and tighten-tests skills.

Grouping work. One run spawned an executor per file, and the answers came out slightly inconsistent. Now the skill groups like files first, and each group gets one executor.

Measuring the rules

The repository has its own evals. Its rule evals answer a narrower question: does a rule change behavior at all? Each score is the share of graders passed, over three runs per arm, on plugin 1.6.1:

RuleWith pluginWithoutDifference
Collaboration0.930.66+0.27
Communication0.970.81+0.15
Code changes0.970.68+0.28
Tests1.000.54+0.47

Those aren't an outside benchmark. Plain Claude was never told to behave this way, and the graders encode the rules' own preferences, so the plugin wins by construction. On one case, the rule to lead with what the user sees made no measurable difference. The skills, agents and hooks haven't been measured against plain Claude.

The weight of a word

It's hard to understand the weight of a word in a skill, task, or requirement. I don't honestly know when changing one word changed how agents behaved. A sentence, though, can move a score. The review had a rule to put a finding at each place a defect is copied. On a pull request with formatting-only edits, one arm wrote seven findings under it, one per file, all false. That was seven of the run's nine false findings, and that pull request's precision was 57% when the others ran 80% to 100%. We narrowed the rule to verified defects.

Who it suits

The tuning suits a developer who is diligent about their output. Someone who'd rather move faster may find it stops to ask more often than they'd like. In my experience over the last several years with AI, the chance of a mistake is low but not zero.

Want the details?

Ask the window anything this page skips.