Scarab Diagnostic Suite field tests target configuration, retrieval, framework, and artifact truth boundaries

When I started building what became Scarab Diagnostic Suite, I was not trying to create a theory of AI-assisted software development. I was trying to survive my own repo. I was building an intricate frontend/backend system with Codex. Not a toy app. Not a landing page. A real system with a public site, internal content architecture, member-facing surfaces, product logic, backend APIs, frontend routes, editorial workflows, operational concerns, and a lot of moving parts that all needed to keep agreeing with each other. Codex could move fast. Very fast. It could generate files, wire components together, write migrations, patch errors, build routes, add tests, explain itself, and keep pushing the project forward. In the beginning, that felt like the whole promise of AI coding agents. Then the drift started. Not failure, exactly. Drift. That distinction matters. The agent could still produce code. The build could still pass. A test could still go green. The UI could still render something. But underneath that apparent progress, the repo would start losing its shape. A file would grow too large because the agent kept adding “just one more thing.” A helper would appear where a real module boundary should have existed. A frontend route would start owning behavior that belonged in the backend. A generated artifact would quietly become treated as source truth. A test would prove the patch instead of proving the intended behavior. A workaround would survive long enough to become architecture. The code worked. The repo was less coherent. That is where Scarab began. Not as another coding agent. Not as a replacement for tests. Not as a magic repair tool. Scarab began as a response to one very specific pressure: how do you keep an AI coding agent from slowly pulling a real software project out of alignment with itself? The problem was not that AI could not code AI can code. That is no longer the interesting question. The more important question is whether AI can keep coding inside the shape of the system you are actually trying to build. That is much harder. A repo is not just a pile of files. It has ownership. It has boundaries. It has rules. It has assumptions. It has places where truth is supposed to live. It has framework conventions. It has tests that are supposed to prove something meaningful. It has public behavior, internal behavior, generated behavior, runtime behavior, and maintainer expectations. When those layers agree, the system feels coherent. When they stop agreeing, the system starts drifting. The uncomfortable part is that we often ask AI to preserve context that humans have not actually made stable. We want the agent to know what matters. We want it to patch narrowly. We want it to respect architecture. We want it to understand which file owns which responsibility. We want it to know which tests are authoritative and which ones are incidental. We want it to know when a workaround is acceptable and when it is a quiet violation of the system. But very often, the repo itself does not clearly know those things. The docs may be stale. The tests may only prove part of the behavior. The runtime may contradict the intended architecture. The issue may describe the symptom but not the boundary that failed. The human may know the rule intuitively, but the repo may not preserve that rule anywhere the agent can reliably obey. So we put an AI coding agent into an unstable field and act surprised when it drifts. But if the system has not defined its own boundaries, why would we expect the AI to enforce them? More context is not the same as stable context A lot of the AI coding conversation is still orbiting the same set of answers: bigger context windows, smarter models, more autonomous agents, better prompts, stronger tools, more memory. Those things can help. But they do not solve the underlying problem by themselves. A larger context window can hold more text. It does not automatically decide which text is authoritative. A smarter model can reason better. It can still reason toward the wrong baseline. A more autonomous agent can move faster. It can also spread a bad assumption across more surfaces before anyone notices. The problem is not simply that the AI needs more context. The problem is that the context needs ownership. Some context should not live as a fragile conversation with the model. Some context should not depend on whether the agent remembered a sentence from three prompts ago. Some context should not have to be re-explained every time the AI touches the repo. The repo should be able to say: This surface owns this behavior. This file is canonical. This pattern is deprecated. This test proves this boundary. This generated output is not source truth. This workaround is temporary. This framework convention must be preserved. This is what “done” means here. That is a very different posture from “let the AI figure it out.” It gives the AI a better job. Instead of asking the agent to be the architect, the historian, the maintainer, the tester, the reviewer, and the builder all at once, the repo can hold the system rules outside the model. Then the AI can do what it is actually good at: move quickly inside a bounded structure. That is when AI becomes more powerful, not less. The real breakthrough is not dumping more responsibility onto AI. It is taking the weight of unstable context off the AI. Diagnostics are not optional in serious systems There is a reason every imagined advanced system, from science fiction ships to humanoid machines, is always running diagnostics. Before the system acts, it checks itself. Are the subsystems online? Are the readings coherent? Is the signal real? Is the failure isolated? Which boundary is compromised? That pattern exists because it reflects something fundamental: serious systems need ways to verify their own operating state. Software is no different. And AI-assisted software needs this even more, not less. If an AI coding agent can touch dozens of files quickly, then the repo needs a way to determine whether those changes preserved the system’s truth. Not just whether the code compiles. Not just whether the check passed. Not just whether the agent says the task is complete. The repo needs to ask: Did this change respect ownership? Did it preserve the boundary? Did it move behavior into the right layer? Did it make the test more truthful or more theatrical? Did it reduce drift or simply silence the visible failure? Did the system become more coherent after the repair? That is the diagnostic layer Scarab is aimed at. Scarab is not trying to be the coding agent. It is not trying to make subjective product decisions. It is not trying to replace maintainers or developers. It is trying to hold repo truth outside the AI. That distinction is the whole product. The first use case was active AI-assisted development The original Scarab use case was not public open-source repair. The original use case was active development. I needed a way to keep Codex working inside the boundaries of a repo that was becoming too large and too intricate to keep entirely in my head at all times. The loop looked something like this: The human defines the project direction. The repo preserves the rules, boundaries, and accepted structure. The AI coding agent writes or changes code. Scarab checks whether the change still aligns with the repo’s accepted truth. If a contradiction appears, Scarab surfaces the diagnostic boundary. The human or coding agent repairs under direction. Scarab runs again to verify that the contradiction has actually been resolved. That is very different from “AI writes code and we hope the tests catch it.” It is also different from “AI writes code and later someone cleans it up.” The point is to keep the repo’s truth under watch while the repo is being changed. That is the forward-facing use case: prevent drift from becoming normal. Then the field tests changed the shape of the theory Once Scarab started working as an active guardrail, a larger question appeared. If Scarab can detect drift while it is forming, can it also detect drift after it has accumulated? That question pushed Scarab into public field testing. I started running diagnostic passes against real open-source issues. Not toy repos. Not staged demos. Real issues in real software projects, across different languages, frameworks, runtimes, and maintainer cultures. The point was not to prove that every bug was caused by AI. That would be too narrow, and honestly not true. The point was to test whether the underlying diagnostic lens held up outside my own repo. Could the same boundary theory apply to systems that were already drifting before Scarab arrived? The answer is starting to look like yes. That is the part that changed the product for me. Scarab began as a way to supervise AI-assisted development. But the field tests started showing that the same principle can work backward. It can enter an existing failure surface, identify the boundary that stopped preserving truth, and help drive a narrow repair candidate. That matters because drift does not care whether it was introduced by a human, a team, an AI coding agent, a migration, a stale test, a runtime change, or five years of “just patch it for now.” Drift is drift. The system has fallen out of alignment with itself. The field tests are proving the theory in public The most important thing about the field tests is not that they produce patches. The important thing is that the same diagnostic question keeps working across very different systems: What truth was this system supposed to preserve, and which boundary stopped preserving it? In a pnpm field test, the issue was not really “self-upgrade is broken” in some broad sense. The boundary was narrower: a command meant to upgrade the pnpm binary was falling through into project dependency behavior when no project manifest existed. The repair was accepted and merged because the boundary was specific: skip the dependency-status auto-install path when the no-manifest condition is intentional, while preserving the existing fallback behavior where a project manifest exists. That is not a vibes-based repair. That is a boundary repair. In a Rust compiler field test, the visible issue was that infinitely recursive nested structs had started compiling in release mode. That sounds huge from the outside. But the repair surface was narrower: raw pointer metadata needed to remain conservative, while the optimized/codegen path still needed to force pointee layout validation so impossible recursive layouts did not slip through. Again, the useful diagnostic was not “Rust is broken.” It was: this specific validation ownership boundary was lost in this specific optimized path. In a Playwright field test, the visible symptom was a hang: response.allHeaders() could fail to settle for a worker script loaded inside an iframe. But the diagnostic boundary was smaller than “Playwright hangs.” The response metadata completion path lost ownership in a worker-frame session. Normal Chromium response metadata behavior should remain intact, but the worker-main-script case needed a bounded fallback so the public API could complete instead of waiting forever for metadata that would not arrive. Different stacks. Different languages. Different symptoms. Same pattern. Something stopped preserving the truth another part of the system depended on. That is the Scarab lane. Narrow repair is not small thinking One thing I keep emphasizing in the field reports is the phrase “narrow repair.” That can sound minor. It is not. A narrow repair is not small because the bug is unimportant. It is small because the boundary is clear. That is exactly what you want. A broad patch often means the failure boundary is still blurry. The repair starts touching nearby code. It adds guards. It rewrites tests. It changes behavior around the symptom. It makes the red light go away, but it may leave the deeper contradiction intact. A narrow repair says: The break is here. This handoff lost ownership. This test proves the boundary. This behavior must stay unchanged. This is out of scope. This is the smallest lane that can restore coherence. That is not just useful for open-source contribution. It is useful for AI-assisted development in general. AI coding agents are very good at generating broad plausible changes. That is both the gift and the danger. If the repair lane is not bounded before the agent begins, the agent may repair the visible failure by creating three new invisible ones. Diagnostics make the repair smaller before the repair begins. That is where AI becomes safer and more useful. Scarab is not anti-AI I want to be very clear about this because the conversation around AI coding often collapses into two extreme positions. One side acts like AI should be trusted to build everything autonomously if the model is strong enough. The other side acts like AI-generated code is inherently suspect and the safest answer is to avoid it entirely. I do not think either position is serious enough. AI coding agents are powerful. They are also easy to misuse. The answer is not to pretend they cannot help. The answer is also not to pretend they can hold every boundary of a complex system by themselves. The better posture is to give AI a structure it can work inside. That is what good human teams already do. A serious engineering team does not rely on one developer’s memory to preserve the whole system. It uses architecture, tests, review, conventions, ownership, documentation, CI, runtime checks, and operational discipline. AI-assisted development needs the same kind of seriousness. Actually, it needs more. Because the agent moves quickly, sounds confident, and can produce changes faster than a human can fully inspect by intuition. That does not mean the agent should be discarded. It means the repo needs a diagnostic layer around the agent. Rules and boundaries are not there to limit intelligence This is one of the deeper shifts for me. At first, it is easy to think of guardrails as restrictions. Do not let the AI touch this. Do not let the AI do that. Do not let the AI go too far. But I think the more interesting version is almost the opposite. Rules and boundaries make AI more powerful because they stop the agent from wasting intelligence guessing the shape of the system. If the AI has to infer everything every time, it spends its power reconstructing context. It has to guess what matters. It has to guess which layer owns which behavior. It has to guess whether the current implementation is intentional or accidental. It has to guess whether a test is authoritative or just historical residue. That is a terrible use of AI. The repo should hold that. The human should not have to keep every implementation rule in their head. The AI should not be trusted to reinvent the architecture from scratch. The repo itself needs to preserve enough structure that the AI can build creatively without constantly distorting the system. That is the deeper theory behind Scarab. Human vision can be messy. Creative direction can be emergent. You do not always know the final shape of the thing before you start building it. That is true in software, writing, design, research, and product work. AI can be extremely useful in that process because it can turn vague direction into working material quickly. But if the project cannot preserve its own boundaries, AI may turn early uncertainty into permanent architecture too fast. It may solidify the wrong thing. It may preserve an accidental implementation as if it were intentional design. It may make the compromised version feel like the only version you can ship. That is the tragedy. AI should help builders reach the real version of the project. It should not pressure them into accepting the version that survived the drift. The backward-facing use case proves the forward-facing one The field tests are useful because they show Scarab working in recovery mode. A public issue already exists. A failure has already surfaced. The repo has already hit a contradiction. The diagnostic pass works backward from symptom to boundary. That is valuable by itself. But to me, the deeper value is that recovery mode proves the forward-facing use case. If the same diagnostic lens can identify accumulated drift after the fact, then it can also be used earlier, while the repo is being changed, to keep the contradiction from becoming normal in the first place. Those two modes reinforce each other. Backward-facing mode says: here is where the system drifted. Forward-facing mode says: do not let this kind of drift enter silently again. That is the product direction I care about most. Not “AI writes code faster.” Not “AI fixes everything.” Not “humans are obsolete.” The more interesting future is: Humans define intent. Repos preserve rules and boundaries. AI agents build inside those constraints. Diagnostics prove whether the system stayed coherent. That is the workflow I think serious AI-assisted development is moving toward. The core principle is simple All systems have rules and boundaries. If those rules are not explicit, they do not disappear. They become implicit. If they are implicit, they become easy to violate. If they are easy to violate, AI will eventually violate them faster than humans can notice. That is not because AI is malicious. It is because AI is operating inside whatever structure we give it. If we give it a repo with unclear ownership, stale docs, partial tests, accidental patterns, and no stable source of truth, we should not be surprised when it produces locally plausible changes that deepen the incoherence. The fix is not just better prompts. The fix is better system discipline. A repo should know what it is. It should know which rules matter. It should know where truth lives. It should know what must be preserved. It should know what has changed. It should know when a boundary has been crossed. And when an AI agent enters that repo, the agent should not be carrying all of that alone. That context should be held outside the model. That is where I think the real AI coding breakthrough begins. Not with an agent that remembers everything. With a repo that can prove what is true. Where this is going Scarab is still evolving. The field tests are part of that evolution. They are not polished marketing demos. They are pressure tests. They show where the diagnostic theory holds, where it needs sharpening, and how it behaves across real projects with real maintainers and real failure surfaces. So far, the pattern is holding. AI did not invent code drift. It made it impossible to ignore. But the answer is not to fear the AI. The answer is to stop asking AI to carry rules that the system itself has not stabilized. If we want AI coding agents to become serious development partners, then we need more than generation. We need diagnostics. We need repo-local truth. We need boundaries that can be checked. We need evidence before repair. We need a way to tell the difference between a patch that silences a symptom and a repair that restores coherence. That is what Scarab is becoming: a deterministic diagnostic layer for AI-assisted development and accumulated software drift. The public field tests are the proof trail. The theory is simple: the bug begins where the boundary stops holding. The repair begins when the system can prove where that happened.

This field test was against Next.js. The issue was Next.js #94450: https://github.com/vercel/next.js/issues/94450 The reported problem involved production browser source maps when React Compiler and Turbopack were involved. The visible symptom was that the final browser source map could expose transformed compiler output instead of preserving the original client source content. That matters because source maps are not just debugging extras. They are provenance artifacts. They tell the developer what source the browser output came from. If a source map claims to represent a source file but its sourcesContent contains compiler-transformed output instead of the original file content, then the debugging artifact has drifted from the source truth it is supposed to preserve. The useful diagnostic boundary was: original client source → transform source map → Turbopack source-map composition → final browser chunk map The important proof was that the Babel/React Compiler transform itself could produce a source map whose sourcesContent still represented the original client file. So the loss was not simply: React Compiler changed the code The sharper issue was: the browser source-map composition path was not preserving original source authority all the way into the final artifact That made the repair lane much narrower. The local repair candidate has two parts: Preserve the original loader input source in the Babel loader transform map. Fill missing source-map file provenance from the origin path when an incoming transform map omits it, so Turbopack has enough identity information to match the transform map back to the generated intermediate file during composition. The goal is not to rewrite source-map behavior broadly. It is not to patch the final browser map after the fact. It is to preserve source authority at the point where the transform map is composed into the browser artifact. A regression fixture was added around a React Compiler client component with an original source marker. The test asserts that the production browser source map keeps the original client source content and does not expose compiler-transformed markers such as React Compiler runtime output. Local validation passed for the focused browser source-map proof, the relevant Turbopack source-map utility test, Rust formatting on the touched file, and diff hygiene. Nothing has been opened publicly yet. Field Test #012 Project: Next.js Issue type: production browser source-map provenance Boundary: transform source-map authority vs final browser artifact composition Result: narrow local repair candidate and regression fixture Status: local proof prepared; no public PR or comment opened yet The important part of this field test is that the bug was not treated as a vague “source maps are wrong” problem. That would be too broad. The useful question was: where did original source truth stop being preserved? The answer was not at the first transform. The transform could still produce a map with original source content. The break appeared later, when the browser artifact was composed. That is the kind of distinction Scarab is built to surface. A broad patch might try to rewrite source-map handling, disable fallback content, or adjust the final output after the fact. A better repair looks for the exact boundary where provenance is lost. In this case, the source map needed enough identity information to carry original-file authority through composition. That is why this field test matters. It is another example of software truth drift in a major framework system. Not AI drift specifically. Not a simple runtime crash. Not a failing UI. This was artifact truth. The source file had one truth. The transform map preserved it. The final browser source map drifted from it. Scarab’s role is to identify that kind of boundary before repair begins. Because once the boundary is clear, the repair can be much smaller than the symptom suggests.

This is the fourth field test in my Scarab Diagnostic Suite series. The first field test was a guided provider/config repair in Open WebUI. The second found a retrieval truth boundary in Open WebUI’s Memory/RAG path. The third moved into Vite and surfaced a dev/build hook-filter contract divergence. This fourth field test stayed in Vite, but it tested a different kind of boundary: build artifact truth. The target was Vite issue #21456: https://github.com/vitejs/vite/issues/21456 The reported problem was specific: A CSS file is explicitly declared as an entry through array-based rollupOptions.input. That CSS file contains only @import statements. Vite emits a manifest entry for the CSS output. But the manifest entry does not preserve: text isEntry: true The same shape works when the input is declared as an object. So the bug was not simply: “CSS manifest is broken.” The deeper contract was: A file declared as an entry should still be represented as an entry in the emitted manifest. That is the diagnostic surface SDS needed to understand. The SDS surface For this field test, I named the internal surface: text build_artifact_truth_boundary That name matters. The goal is not to teach SDS one Vite-specific bug. The goal is to teach SDS a reusable boundary: text declared input → build pipeline → emitted artifact → manifest metadata The core question is: Does the artifact metadata still tell the truth about what the build input declared? In this case: The user declared a CSS file as an entry. The build produced an artifact. The manifest included the artifact. But in the reported failure shape, the manifest forgot the entry identity. That is build artifact truth loss. What SDS found SDS surfaced the generic boundary without being fed the issue text. The diagnostic surface was: text framework_build_artifact_truth_boundary That was the important product result. SDS did not need a detector named: text vite_css_import_only_array_input_missing_isEntry That would be too specific. Instead, SDS identified the broader class: build output metadata may not be preserving source/build declaration truth. That is the reusable detector class I want inside Scarab Diagnostic Suite. The issue-specific failure was only one instance of that class. The field-test repair In the affected field-test target, Codex created a focused reproduction and repair. The focused repro failed before repair. The repair touched Vite’s CSS/asset/manifest path and added a regression test for the exact issue shape: array-based input, explicit CSS entry, CSS contains only imports, manifest should preserve isEntry: true. After the repair, the focused test passed. SDS reran and the targeted build_artifact_truth_boundary finding cleared. That proved the internal diagnostic loop: SDS found the boundary. The affected target reproduced the issue. Codex repaired the boundary. The functional test passed. SDS confirmed the selected boundary cleared. That was the SDS field-test result. The current-upstream check After the field-test repair, I ran a clean current-upstream check against Vite main. That changed the public outcome. Current Vite main already preserved the needed behavior, likely because another merged upstream change had already addressed the relevant CSS entry map behavior. So the issue-shaped regression test passed on current upstream without needing my field-test code patch. That meant the local repair remained valuable as field evidence, but it was not appropriate to submit as a public runtime fix. This is an important distinction: SDS did not decide whether to open a PR. SDS is not a PR tool. SDS is not a bug-fix submission system. SDS diagnosed the boundary, helped verify the field repair, and produced evidence. The upstream check was part of my external field-testing workflow, not native SDS behavior. What this field test proved This field test proved that SDS can detect a real build-system truth boundary: text declared input identity → emitted manifest metadata It also proved that field-test repair evidence and public contribution shape are not the same thing. A local target can reproduce and repair a bug. The current upstream branch can already contain the relevant behavior. Both can be true. In that case, the correct result is not to force a public patch. The correct result is to log the field-test evidence and keep SDS clean. Why this was still valuable This was not a wasted run. It gave SDS a reusable detector class: text build_artifact_truth_boundary That applies beyond this one Vite issue. It applies to any build system where declared source/config input becomes emitted output metadata: asset manifests, route manifests, chunk maps, generated indexes, schema registries, plugin output maps, sitemaps, bundle metadata. If an output artifact claims to represent source truth, SDS should be able to inspect whether that truth survived the pipeline. That is the product value. Not the patch. The patch was field-test evidence. The detector class is the reusable SDS refinement. Internal classification Field Test #004 Project: Vite Issue: #21456 Surface: build artifact truth boundary Contract class: declared input identity → emitted manifest metadata Mode: blind diagnostic surface detection followed by local red/green repair and current-upstream verification Outcome: local field repair successful, SDS finding cleared, current upstream already passed the issue-shaped regression, no public repair PR needed Primary lesson: SDS should store reusable diagnostic patterns, not repo-specific bug fixes The product boundary This field test helped clarify an important boundary for Scarab Diagnostic Suite: SDS should store patterns, not patches. Patterns: text Declared input identity should survive into emitted manifest metadata. Not patches: text Fix Vite #21456. That distinction matters. If SDS becomes a collection of bug-specific repair memories, it becomes noisy. If SDS learns reusable truth boundaries, it becomes more powerful. This field test belongs in the second category. The larger pattern By Field Test #004, the series had covered four distinct surfaces: provider/config contract truth, retrieval/source-context truth, dev/build runtime-mode truth, build artifact metadata truth. That is the emerging map. SDS is not just finding syntax problems. It is finding places where software says one thing in one part of the system and then contradicts, drops, or mutates that truth somewhere else. That is where AI coding agents need diagnostic supervision. Because once the truth boundary is named, the repair can stay smaller. And once the evidence is captured, the field-test result can be used correctly: sometimes as an upstream PR, sometimes as a test-only proof, sometimes as an internal ledger entry, sometimes as a detector refinement. The product is not the PR. The product is the diagnostic boundary. This field test proved that distinction.

This is the third field test in my Scarab Diagnostic Suite series. The first field test showed SDS converting a known Open WebUI provider/config issue into a bounded repair lane. The second found a retrieval truth boundary in Open WebUI before the issue text was used as functional proof. The third field test moved into a different kind of repo entirely. Not an AI application. Not a RAG pipeline. Not a provider adapter. This time the target was Vite. Specifically: Vite issue #21979, where a plugin transform hook filter behaved differently in dev and build. That shift matters. This tested whether Scarab Diagnostic Suite could generalize beyond AI-app runtime bugs into framework/build-system contracts. The issue The reported bug was small on the surface. A Vite plugin defines a transform hook with an id filter: ts filter: { id: { include: '*/.(js)', }, } In dev mode, the filter is applied and the transform reaches JavaScript files. In build mode, the same filter shape does not apply. The reporter noted that: text */.js works in both dev and build, but: text */.(js) does not. The expectation was simple: Even if a particular glob syntax is unusual, Vite’s hook filter should behave consistently between dev and build. That is the important part. This was not just: “a glob pattern is weird.” The real contract was: The same plugin hook filter should mean the same thing in dev and build. That is exactly the kind of consistency boundary SDS should be able to see. What SDS found SDS surfaced the failure as a framework/build lifecycle hotspot. The key diagnostic shape was: text framework_runtime_mode_filter_contract_consistency Plain English: A filter contract that should behave consistently across runtime modes was producing different results depending on whether Vite was running in dev or build. That is a different surface from the Open WebUI tests. In Open WebUI #25038, the unstable boundary was retrieval truth moving through a Memory/RAG source-context path. In Vite #21979, the unstable boundary was framework behavior moving through two execution modes: dev mode, build mode. Same plugin contract. Different outcome. That is runtime-mode contract drift. Why this matters Framework users do not usually think in terms of internal execution-mode contract divergence. They think: “I wrote a plugin hook.” “It works in dev.” “Why does build behave differently?” That is why these bugs matter. Dev/build divergence creates a trust problem. A developer can test something locally, see it work during dev, and only discover during build that the contract was not actually stable. The outward symptom is small: “my plugin did not run during build.” The deeper issue is: “the system is interpreting the same declared filter through different semantic paths.” Scarab’s job is to find that deeper boundary. The repair lane The repair was bounded to the build transform path. The local change made the build wrapper apply Vite’s transform hook filter before invoking the plugin handler, rather than allowing lower-level build filtering behavior to decide that path first. The goal was not to redesign hook filters. The goal was not to change all glob semantics. The goal was not to patch Rolldown broadly. The goal was one narrow contract repair: For the reported filter shape, build should apply the transform consistently with dev. That was the repair slice. Red/green proof The focused regression test proved the reported behavior. Before repair, the test failed because the transform hook did not reach: text /src/main.js during build. The failing shape was: text expected false to be true After repair, the same focused test passed. The validation package was broader than a single unit test: focused regression passed, eslint passed on touched files, oxfmt passed, Vite build-bundle passed, Vite build-types passed, Vite typecheck passed. That mattered because framework/build-system patches need more than “one test went green.” They need confidence that the change does not disturb the surrounding build pipeline. SDS after repair After the repair, SDS reran against the target. The specific runtime-mode filter contract finding cleared. The broader repo was not magically clean, and that is fine. SDS is not supposed to pretend the entire repo becomes perfect after one bug repair. The important part is that the selected diagnostic surface changed in the expected direction: the focused contract finding cleared, the hotspot quieted enough for Stepwise to move onward, and the repair remained bounded to the issue-shaped failure. That is the SDS loop working again: diagnose the surface, repair only the selected lane, prove the behavior, rerun diagnostics, confirm the targeted finding changed. What this field test proved Field Test #003 proved something new: SDS is not only useful for AI-app bugs. It can detect framework/build-system contract drift. This is a different class of failure than the Open WebUI tests. The Open WebUI examples were about runtime data and response-shape boundaries. The Vite example was about execution-mode consistency inside a build tool. That matters because it suggests the underlying SDS model is not just memorizing one kind of bug. It is beginning to reason about repo truth across different software surfaces. In this case, the truth was: A plugin hook filter should mean the same thing in dev and build. When that truth broke, SDS found the surface. Internal classification Field Test #003 Project: Vite Issue: #21979 Surface: framework runtime-mode filter contract Contract class: dev/build hook-filter consistency Mode: blind diagnostic surface detection followed by focused red/green repair Result: local repair successful; targeted SDS finding cleared; focused regression passed Primary lesson: SDS can detect and guide repair of framework/build lifecycle contract drift Why this matters to Scarab This field test broadened the product story. Scarab Diagnostic Suite is not a bug-fix bot. It is not tied to one repo. It is not limited to AI application middleware. It is a diagnostic supervision layer for AI-assisted development. That means it needs to understand different kinds of repo truth: provider config truth, retrieval context truth, external API metadata truth, runtime-mode behavior truth, build artifact truth. Vite #21979 added a new one: execution-mode contract truth. Same input. Same plugin contract. Different runtime mode. Different behavior. That is a real diagnostic surface. And SDS found it. The larger pattern Across the first three field tests, the pattern is becoming clearer. The value is not that an AI agent can write code quickly. The value is that SDS makes the repair smaller before the AI agent starts writing. That is the difference between unmanaged AI coding and diagnostic-led repair. Unmanaged AI often starts with a symptom and wanders. SDS starts with a boundary. In this field test, the boundary was not: “Vite is broken.” It was: “Vite’s transform hook filter contract diverges between dev and build.” That is smaller. That is testable. That is repairable. That is the point of Scarab Diagnostic Suite.

This is the second field test in my Scarab Diagnostic Suite series. The first test was guided. A known Open WebUI provider/config issue was turned into a bounded repair lane. This second test was more important. This was the first field test where SDS moved closer to the thing I actually wanted to prove: Can Scarab Diagnostic Suite identify the right failure surface before being told the exact bug? The target was Open WebUI issue #25038. The user-facing symptom looked like a web search problem. Open WebUI successfully queried SearXNG. The UI showed valid search results. The retrieved snippets already contained useful information. Then the app entered the “Querying” phase. Then it showed “No sources found.” Then the model answered as if live web search was unavailable. That is a deceptive class of failure. Search worked. Retrieval worked. The UI knew results existed. But the retrieved information did not survive into usable model context. The system appeared to succeed and fail at the same time. The field-test setup I ran SDS against a fresh Open WebUI v0.9.5 target. The important constraint: SDS did not receive the GitHub issue text. No custom diagnostic script was aimed at the known failure. No hand-written detector was inserted for this issue. The goal was to see whether SDS could identify a suspicious subsystem without being handed the answer. That matters because it is easy to make an AI system look impressive if it is fed the bug. I wanted to know whether SDS could surface the correct class of instability before the repair agent started changing code. What SDS found SDS selected the Memory/RAG surface and identified a retrieval truth boundary problem. The findings were under: text retrieval_truth_boundary with the contract class: text source_context_contract_consistency The flagged paths involved variables like: text res results queries_response This was the first important signal. SDS did not say: “This is Open WebUI issue #25038.” That would have been suspicious. Instead, it said something more useful: “This retrieval/source-context boundary is noisy. Response and result shapes crossing this boundary are not visibly stable.” That matched the real failure class. The visible issue was “No sources found.” The deeper issue was that retrieved truth was not safely surviving the trip into model context. That is the kind of boundary Scarab Diagnostic Suite is meant to detect. Why I call it a retrieval truth boundary In AI applications, retrieval is not enough. A search provider can return good results. The UI can show those results. A source-processing path can begin. A document loader or vector pipeline can start. But if the final model context does not receive the retrieved truth in a usable shape, the system still fails. That is what made this bug interesting. The user saw: text Searching the web Searched 5 sites valid URLs valid snippets Querying No sources found Then the model responded as if it never had web access. That is not simply “web search is broken.” That is: retrieved truth existed, but the boundary that should carry it into model context collapsed. That is a retrieval truth boundary failure. The bounded repair After SDS selected the hotspot, Codex performed one bounded stabilization pass. SDS did not become the repair tool. SDS identified the diagnostic boundary. Codex performed the repair. Then SDS reran to verify whether the targeted surface quieted. The initial SDS finding count for the Memory/RAG retrieval boundary was 3. After the bounded repair, the targeted findings dropped to 0. The selected surface moved from noisy/contradictory to quiet/organized. That proved one internal thing: The repair addressed the exact surface SDS had selected. But I still needed functional proof. A diagnostic finding clearing is useful, but the actual bug still has to go red to green. Functional proof Only after SDS selected the hotspot did I use the issue’s concrete failure shape as functional proof. The repro reproduced: text TypeError: 'JSONResponse' object is not subscriptable The failure happened because chat_web_search_handler could receive a JSONResponse object but then consume it as if it were an OpenAI-style completion dictionary with a choices payload. That is a response-shape contract failure. The repair added a response-shape boundary. Instead of directly subscripting the response, the code checks whether the generated-query response has the expected completion shape. If not, the web-search path falls back safely to the original user query instead of crashing during source/context processing. The focused regression passed after repair. SDS reran afterward and the retrieval truth boundary remained quiet. That gave the full loop: SDS detected the surface. Stepwise selected the hotspot. Codex made a bounded repair. The real failure went red to green. SDS verified the selected boundary cleared. What made this different from Field Test #001 Field Test #001 was useful, but guided. The issue was known, and SDS helped turn it into a constrained repair lane. Field Test #002 was stronger because SDS found the relevant subsystem before being given the issue text. It did not know the GitHub issue. It did not magically know the whole bug. But it identified the right class of instability: retrieval/source-context response-shape inconsistency. That is the difference between a tool that merely follows instructions and a diagnostic suite that can surface a real failure boundary. Internal classification Field Test #002 Project: Open WebUI Issue: #25038 Surface: Memory/RAG retrieval truth boundary Contract class: source/context response-shape consistency Mode: blind-ish diagnostic surface detection followed by functional proof Result: local repair successful; targeted SDS findings cleared; focused regression passed Primary lesson: SDS can identify the correct failure surface before issue text is used as functional proof Why this matters A lot of AI application bugs live between systems. Search works, but context injection fails. A tool returns data, but the model never receives it. A provider response exists, but the handler expects a different shape. The UI says one thing, the runtime does another. These are truth-boundary failures. Scarab Diagnostic Suite is being built to find those boundaries and keep AI coding agents from making broad, noisy repairs. This field test showed that pattern working. Not perfectly. Not magically. But usefully. The suite found the unstable surface. The agent repaired one slice. The test proved the real failure. The diagnostic rerun confirmed the boundary quieted. That is the repair loop I want: not bigger AI, not more autonomous wandering, but bounded diagnostic repair. A system that can look at a noisy repo and say: The problem is probably here. This is the boundary. Repair only this lane. Now prove it.

I have started running Scarab Diagnostic Suite against real open-source issues as field tests. Not synthetic examples. Not toy repos. Not controlled demo fixtures. Real repos. Real failures. Real diagnostic pressure. The first field test was against Open WebUI issue #25078, where Azure OpenAI connections broke after connection settings were edited. The reported failure had a clear shape: A working Azure config originally included: text azure: true After editing the connection, the saved config could become: text provider: "azure" without preserving the older azure flag. The backend runtime path then failed to recognize the connection as Azure-shaped and fell through to a generic OpenAI /models path. The result was an Azure request going to the wrong endpoint shape and failing. That is the kind of bug that looks obvious once named, but can become noisy during repair because it touches multiple surfaces: configuration persistence, provider metadata shape, runtime provider branching, Azure endpoint behavior, model listing behavior. The purpose of this first field test was not to claim blind discovery. This was a guided diagnostic repair pass. The issue was known. The failure was described. The test was whether Scarab Diagnostic Suite could take a known live issue and convert it into a bounded diagnostic repair lane. What SDS found SDS identified the issue as a provider/config contract failure: text provider_config_contract.runtime_branch_missing_provider_shape Plain English: Open WebUI had more than one persisted way to describe an Azure connection, but the runtime branch only recognized one of them. The broken contract was: text provider: "azure" could exist in persisted config, while the runtime path only checked: text azure: true That meant Azure-shaped config could be treated as generic OpenAI config. This is the kind of thing Scarab Diagnostic Suite is designed to name. Not: “Something is wrong in this file.” But: “A provider identity contract exists in one layer and is not being honored by another layer.” That distinction matters. A bug report gives a symptom. A diagnostic suite should identify the contract that broke. The repair lane The local repair was intentionally small. The runtime path needed to treat both of these as Azure-shaped config: text azure: true provider: "azure" The repair did not need to redesign provider configuration. It did not need to rewrite connection persistence. It did not need to touch unrelated OpenAI provider behavior. The repair lane was: text Persisted provider shape → runtime provider branch That was the contract boundary. The regression test proved the issue-shaped failure before repair and passed after repair. Before repair, the failing path attempted a generic /models request against an Azure base URL. After repair, the Azure provider-shaped config was routed through the Azure-aware branch. SDS after-check After the repair, SDS reran the same focused diagnostic lane. The original provider/config finding cleared. That is the loop I wanted to test: diagnose the contract boundary, repair only that boundary, prove the behavior with a focused regression, rerun diagnostics, confirm the finding cleared. This is the first important Scarab pattern: The point is not just that the code changed. The point is that the diagnostic finding changed in response to a bounded repair. That gives the repair a shape. It is not just “the agent patched something.” It is: SDS identified a contract failure. Codex repaired that contract failure. A regression proved the behavior. SDS confirmed the targeted finding cleared. What this field test proved Field Test #001 proved a specific thing: Scarab Diagnostic Suite can take a known live bug report, identify the underlying contract boundary, constrain the AI repair lane, and verify that the targeted finding clears after repair. It did not prove blind bug discovery. It did not prove full repo repair. It did not prove that SDS should own the fix. It proved something more practical: A diagnostic suite can keep an AI coding agent from turning a known bug into an overbroad patch. That is the product thesis beginning to show. AI coding agents are fast, but speed without boundaries creates drift. Scarab’s role is not to be the coder. Scarab’s role is to maintain repo truth while the coding agent operates. In this field test, that meant turning a provider/config bug into a narrow, testable repair lane. Internal classification Field Test #001 Project: Open WebUI Issue: #25078 Surface: provider/config contract Mode: guided diagnostic repair pass Primary finding: provider_config_contract.runtime_branch_missing_provider_shape Repair lane: persisted provider metadata shape → backend runtime branch Result: local repair successful; focused regression passed; SDS finding cleared Primary lesson: SDS can translate a known issue into a bounded contract repair lane Why this matters A lot of AI-assisted coding fails because the agent starts with a symptom and then wanders. It fixes nearby code. It edits unrelated files. It changes tests to match the patch. It follows the noise instead of the boundary. It makes the repo harder to reason about. This field test showed the opposite pattern: one issue, one contract boundary, one focused regression, one bounded repair, one diagnostic after-check. That is the repair style I want Scarab Diagnostic Suite to enforce. Not louder AI. Better boundaries.