Developer Builds Self-Verifying, Self-Healing AI Rule System Using Transcript Markers

I Built a Closed-Loop Self-Healing System for My AI Config — By Accident

I didn't set out to build a self-healing system. I just wanted to stop forgetting things. What emerged was a feedback loop that fixes itself without me. After 40+ Claude Code sessions, I noticed a pattern: identify a problem → write a rule → forget about it → rediscover the same problem 5 sessions later → write the same rule again. The rules folder was growing. But behavior wasn't changing. The Alertmanager Problem Prometheus Alertmanager has a concept that changed how I think about systems: alert → acknowledge → fix → auto-resolve. Without the "auto-resolve" step, alerts pile up. You get a graveyard of acknowledged-but-never-fixed issues. They're not "open" — they're dead. Nobody looks at them. Nobody fixes them. They just sit there, eroding trust in the entire monitoring system. My AI config had the same problem. Problems were identified. Rules were written. But nothing tracked whether the rules were actually followed — or whether the problem was actually fixed. The missing piece wasn't more rules. It was a closed loop. The Accidental Architecture Three files. That's all it took to close the loop: config-health.py (Stop hook) ↓ counts markers, detects gaps pending-verifications.md (persistent memory) ↓ surfaces unresolved items BODY.md (startup rules, read every session) ↓ tells AI to focus on pending items ↓ AI follows the rule, produces markers config-health.py (next Stop hook) ↓ detects markers → auto-deletes from pending The circuit: Measure: config-health.py greps the transcript for rule markers like [✓THINK] and [✓CONTEXT] Remember: rules with low execution rates get written to pending-verifications.md — they persist across sessions Surface: BODY.md reads pending-verifications.md on startup — the AI sees "these rules need attention" Re-check: next Stop hook, config-health.py counts again — if the rule is now being followed, auto-delete from pending I didn't design this loop. It emerged from three independent decisions: Rules should be self-verifiable (embed markers in rule text) Monitoring should be silent on success (zero tokens on normal path) Unresolved issues should persist across sessions (write to file, read on startup) Each decision solved an immediate problem. Together, they formed a complete circuit. The Self-Healing Part Here's the part I didn't expect: the loop runs without me. I don't check dashboards. I don't review reports. I don't maintain a tracking spreadsheet. The system surfaces issues when they exist and silences them when they're fixed. My only job is to follow the rules when BODY.md tells me to. Last week's session: BODY.md surfaced "THINK rule execution: 40% (target: >70%)." I paid extra attention. Hit [✓THINK] 12 times. Next session, pending-verifications.md was clean — config-health had auto-deleted the entry. I didn't manually "close" anything. The loop closed itself. What Makes This Work Three properties that any self-healing loop needs: Measurement is mechanical, not judgment. Regex counting doesn't get tired, doesn't rationalize, doesn't say "eh, probably fine." It counts or it doesn't. Memory survives sessions. pending-verifications.md is a file on disk. It doesn't disappear when the AI's context window resets. Problems can't hide by waiting you out. Re-check is automatic. You don't have to remember to verify. The hook runs every session. Fixed problems auto-resolve. Only real issues stay visible. The General Principle Closed loops don't require complex architecture. They require a measurement, a memory, and a re-check. Three files, if they form a complete circuit, are enough. This is the same pattern that makes TDD work (write test → run test → fix → re-run). The same pattern that makes CI work (push → build → test → report). The same pattern that makes any feedback system work. The insight isn't new. What's new is applying it to personal AI configuration — the rules and habits that govern how you work with AI assistants. Try It Yourself Take one rule in your AI config. Add these three pieces: A measurement: grep -c '\[✓YOUR_RULE\]' transcript.jsonl A memory: write the count to a file that gets read on startup A re-check: run the same grep next session, compare That's your first closed loop. If you get the circuit right, it might start self-healing without you noticing. The loop lives in delivery-gate (the mechanical layer) and checkgrow (the methodology layer). Both MIT licensed, both open to contributions. This article is part of the *Engineering Trustworthy AI Output** series:* Part 1: I Made My AI Rules Self-Verifiable — Here's How Part 2: Your AI Agent's Best Work Produces Zero Output — And That's the Point Part 3: I Built a Closed-Loop Self-Healing System for My AI Config — By Accident ← you are here

2 hours ago

I Made My AI Rules Self-Verifiable — Here's How

The problem wasn't that my rules were bad. The problem was I had no idea if they were being followed. I had 15 rules for my AI assistant. "Think before acting." "Check context before asking." "Capture learning after complex tasks." Good rules. Important rules. After 30 sessions, I finally checked: my "think before acting" rule had been followed exactly zero times. Not because the AI was ignoring it. Because the rule was a wish — not a verifiable constraint. It had no teeth. The TDD Insight Nobody Applied to AI Config Test-Driven Development taught us a hard lesson: requirements without assertions aren't requirements. A test that always passes is useless. But a requirement without a test is just a wish. Every software team learned this. CI pipelines enforce it. Linters check it. But AI configuration rules? We write them like sticky notes on a monitor — good intentions with no verification mechanism. I realized my rules were all wishes. Every single one. The Fix: Embed the Assertion in the Rule Here's the before and after: # Before (wish): Think before acting on any request. # After (verifiable): Think before acting → mark [✓THINK] when done. That [✓THINK] isn't decoration. It does two things simultaneously: For the AI: the marker creates a binary action — "did I think or didn't I?" No ambiguity. For the tooling: grep -c '\[✓THINK\]' transcript.jsonl becomes a health metric. No AI required. The marker is the assertion. The grep is the test runner. And the rule text itself is the specification — self-contained, self-verifying. The Three-Line Architecture The whole system boils down to three lines of logic: Rule text → contains its own success criteria ([✓MARKER]) Transcript → machine-readable evidence (regex-countable) Health script → mechanical verification (no AI inference) I wrote config-health.py — 350 lines of Python, zero dependencies, stdlib only. It runs as a Claude Code Stop hook. Every session, it: Greps the transcript for rule markers ([✓THINK], [✓CONTEXT], [✓DELIVERY]) Writes execution rates to a JSONL log Manages a pending-verifications.md file — rules that haven't been followed show up as pending items The script never blocks. It only reports. Exit code 0, always. But the data it produces changes behavior — because I read pending-verifications.md on next startup. What Changed After deploying this: my rule execution rate went from "I have no idea" to tracked visibility across every session. I still miss rules sometimes. But now I know when I miss them. The dashboard surfaces exactly which rules need attention. The surprising part: the markers themselves changed how the AI behaves. [✓THINK] is a concrete action — not a vague "consider the problem," but a specific thing to output. The AI hits it more often because it's clearer what "done" looks like. The General Principle Any rule without an embedded success criterion is a wish. Wishes don't execute. This applies beyond AI config. Your team's code review checklist? If "check for security issues" doesn't have a verifiable step, it's not happening. Your personal productivity system? If "review weekly goals" isn't measurable, you're not reviewing them. Verifiability isn't about distrust. It's about closing the loop between intention and evidence. Try It Yourself Pick one rule in your AI config. Add a verification marker. Write 5 lines of grep to count it: grep -c '\[✓YOUR_MARKER\]' ~/.claude/transcripts/*.jsonl You'll learn more from that grep output than from 10 new rules you never verify. The config-health.py script is part of delivery-gate — a dual-layer mechanical gate for Claude Code. MIT licensed. The methodology behind it lives at checkgrow. This article is part of the *Engineering Trustworthy AI Output** series:* Part 1: I Made My AI Rules Self-Verifiable — Here's How ← you are here Part 2: Your AI Agent's Best Work Produces Zero Output — And That's the Point Part 3: I Built a Closed-Loop Self-Healing System for My AI Config — By Accident

2 hours ago