Same task. Same prompt, byte for byte. Same model, claude-sonnet-5. Fourteen scored builds. The only variable is what stands between the model and merge-ready.

What actually happened

The task is a small booking app: SQLite, Express, a form. Real enough to have real failure modes - ownership checks, date validation, a booking that must be unique, user text rendered back to the page.

Bare Claude Code is good at this. Its best run scored 110 of 120 and took two minutes. That is the honest part of this page: code quality is not the difference. The difference is that it was run four times on the identical prompt and the four runs did not agree - two shipped stored XSS, three accepted 2026-13-45 as a date - and with no gate, every one of those builds was handed over as done.

Two early Vibestrate rounds wrote the same XSS. Neither reached merge-ready: the review named it, the gate held, both runs ended blocked. That is the claim this page is actually making - not that the model stops making mistakes inside Vibestrate, but that something stands between the mistake and your main branch, and on the runs where it mattered, it held.

One control matters here and it is disclosed rather than buried: bare Claude, given four engineering standards pasted into its prompt, also shipped no XSS. Telling the model helps, in either arm. What the standards did not fix - in any bare run, ever - is the sharpest result in the benchmark: every bare build, and the hand-written reference implementation used to validate the judge, served the whole company’s booking history and per-person charges to a request with no identity at all. Across all fourteen scored builds, exactly two refused that request. Both were Vibestrate runs, on a flow that puts an independent security panel between the code and the verdict.

The scoreboard, all of it

Fourteen scored builds; every score file is in the benchmark repo, including the iteration rounds. secure-build is not a built-in: it was written for this benchmark in the product’s own flow format - a security panel, a fix pass, a re-arbitration, a verify gate. That it can be written in an afternoon is the point; that it does not ship in the box is the honest caveat.

Where Vibestrate fell short, and what changed

The default flow’s 94 is on this page on purpose, and so is the fact that two early Vibestrate builds wrote the same XSS the bare runs wrote. The cause was a real product defect this benchmark found: project policies - rules like “never render untrusted data as HTML” - only reached the reviewer, never the code-writing seat. A rule the writer never sees can only catch the defect after it exists, at the cost of a full review-and-fix round trip.

It was fixed, and then it was re-measured, because “fixed” is a claim. Same prompt bytes, same model, same four one-line policies as the round that scored 97 - the only change was the product, and the prediction was committed to the repo before the run. Result: 110 of 120, approved first pass, in 7 minutes for $1.84. The fix loop never ran, because the rules reached the writer and the XSS, the date bug and the missing database constraint were never written at all. Prevention turned out cheaper than catching: the rematch cost less than the round it replaced.

The billing endpoint is the honest remainder. No one-line policy names it - the brief never defines who “admin” is - so the default flow ships it, the way every bare run and the reference author did. Catching it took the security panel, and even the panel caught it in two of its five runs, not all. That is what the numbers actually support: rules you write are now enforced before the code is written; the defects nobody named still need a reviewer standing in the right place, and sometimes a human.

Round two: SWE-bench Verified, judged by the official harness

The booking app above is our task and our rubric, so round two removes us from the judging entirely. SWE-bench Verified is the industry-standard benchmark for coding agents: 500 real GitHub issues from twelve open-source projects (our six drawn instances landed in django, sympy, pytest and sphinx), where the ground truth is the maintainers’ own fix and the judge is the repository’s own test suite, run by the official swebench harness in Docker. A patch resolves an instance only if the tests the maintainers wrote for the real fix flip from failing to passing while the rest of the suite holds. Nothing of ours grades anything.

Six instances were drawn by a rule committed before the draw: sort all 500 by the SHA-256 of their id, take the first six. Both arms got the same repository at the same commit and the same issue text, byte for byte. Bare Claude Code got the issue piped straight in; Vibestrate ran it as a task with its stock configuration, disclosed in full below. Every patch was submitted to the harness, including the ones Vibestrate itself declined to certify - the gate does not get to hide from the judge.

Six instances, reported instance by instance - six is what one machine and one day honestly buys, and we will not print a percentage from it. Failed means the maintainers’ test for the real fix still fails (13798: test_latex_basic; 18211: test_issue_18188). Neither arm broke a single previously-passing test on any instance. Bare run cost was not instrumented; Vibestrate’s costs are as measured by the product.

Read the two columns first: identical, instance for instance. Same four resolved, same two failed, down to the same failing test. Claude writes the code in both arms - on resolution, the harness found no difference, and we report that plainly.

Now read the third column. Bare Claude’s final word was no signal: some runs ended asking to run tests, some warned tests never ran - on correct and broken patches alike - and its one flat “it’s been verified working” claim sits on a patch that fails the maintainers’ tests. Vibestrate’s verify gate passed exactly one patch - which is correct - and its reviewer and verifier refused to certify the other five, because no run managed to execute the repos’ test suites, and Vibestrate refuses to certify code whose tests never ran. Both actually-broken patches sit inside that refused set. No patch that fails the maintainers’ tests ever passed Vibestrate’s gate. The cost is stated in the same breath: three of the five refusals were over-caution on patches that turned out correct. The gate never claimed to know which patches were wrong; it declined to claim what it could not prove - and on exactly the two instances where that proof was missing and mattered, the strongest “done” claim in either arm was bare Claude’s, on a broken patch.

The exact configuration, in full

The Vibestrate arm is stock: vibe init --yes plus three setters, the built-in default flow, the scaffolded default crew. No project policies, no validation commands, no custom rules beyond what init writes for everyone.

vibe init --yes
vibe config set providers.claude '{"type":"claude-code","command":"claude","args":["-p"],"input":"stdin","settings":{"safeMode":true}}'
vibe config set profiles.claude-balanced.model sonnet
vibe config set profiles.claude-balanced.power medium
vibe run "$(cat ISSUE.md)" --flow default --crew default --unattended

The default flow: Plan → Architecture → Implement → Validate → Review, looping back through Fix and Re-validate up to twice (three review passes in all), then an independent Verify gate decides merge-readiness. The default crew seats six roles - Planner and Architect read-only, Implementer and Fixer with code-write, Reviewer and Verifier read-only - every seat on the same profile: Claude Code, sonnet, medium effort. The bare arm runs the identical model and effort: claude -p --safe-mode --model sonnet --permission-mode acceptEdits --effort medium.

What this round disclosed about us

A correction, added after this page first went up. We originally described those five refusals as environmental - a vetoed network install here, a missing test command there. That was not the whole story, and the larger part was our fault. Vibestrate’s permission model let a seat be marked as allowed to run commands while never actually granting it: the headless CLI auto-approves file edits but not shell commands, so a command outside the host’s own allow rules waited for an approval nobody could give, and the call simply hung. One reviewer seat made 39 attempts and collected 21 “this command requires approval” results while its own profile said it had shell access. So the gate was not being cautious by design on those runs; it was refusing to certify work it had been prevented from checking.

That defect is fixed. We have not re-run SWE-bench Verified since, so the table above stands as measured and we are not restating it. On a separate leg that we did re-run - a chess move generator scored by exact node counts - the same fix moved certification from one run in three to three in three, with zero approval denials, while the code itself scored a perfect 53 of 53 on every run before and after. The gate stopped refusing correct work; it did not make the code better.

• The benchmark found a real Vibestrate defect: the task field was capped at 2,000 characters, so two of the six issues were refused before any agent ran. We did not truncate the issues or work around it - we fixed the product (the cap is now GitHub’s own 65,536 issue-body limit, behind one shared schema) and re-ran only those two instances on the fixed build. That is the third first-contact defect this benchmark program has found and fixed.
• Unattended runs cannot answer approval requests, so “refused to certify” is the expected terminal state when tests cannot run; a human in the loop could have approved the dependency install and gotten a certified-or-rejected verdict instead. sympy‑17318’s run passed its own verify and was held only by the no-auto-merge rule - Vibestrate never merges on its own.
• Evaluation ran the official x86_64 harness images under emulation on an arm64 machine - identically for both arms. The first instance (django‑16100) ran on both arms before the submit-everything rule was written down; the rule was committed before the other five ran.
• The draw rule, per-run predictions, amendments, patches, harness reports and per-instance logs are all committed in the benchmark repo.

Why you can trust the numbers

• The 100-point core rubric and the judge were committed before either path ran. Neither path is told a test suite exists. Two checks were added later - stored XSS after round 1, the billing read after round 2 - and applied retroactively and identically to every build of both paths, all re-scored on the final judge.
• Nothing is graded by a model. Every point is an HTTP status code, a row count, or the schema.
• The judge is itself tested: a deliberately correct reference scores 110 of 120, and a four-bug mutant scored 85 of the 110 points that existed when it ran, failing exactly the four checks the bugs break.
• Both paths get the same prompt bytes (checksummed), the same model, the same permission mode for the code-writing turn, the same host isolation. One asymmetry, disclosed: Vibestrate injects the 48-line rules.md its init scaffolds; the bare path never saw it.
• Two harness bugs and one discarded run (the host slept mid-response) are disclosed in the write-up. This page itself was rewritten after a hostile fact-check of its first draft.

The claim is not that Vibestrate writes better code than Claude. Claude writes the code either way, and its best bare run nearly tops the rubric. The claim is that across four identical runs it had two bad days - and the difference between the two columns is whether anything was standing there when it did.