cases/dev batch 5: mine + scaffold 10 public-workflow cases (unverified) - #1
Closed
vk-dd wants to merge 2 commits into
Closed
cases/dev batch 5: mine + scaffold 10 public-workflow cases (unverified)#1vk-dd wants to merge 2 commits into
vk-dd wants to merge 2 commits into
Conversation
Research and selection only — no case directories, no Dockerfile changes. Ten real-commit candidates whose failure is observable through a public surface (CLI flag, config file, HTTP header, public SDK call, on-disk artifact), one Lane A support ticket + one Lane B acceptance story per language, across ten repositories new to the corpus: cpp libgit2 #7291 cpp-httplib #2479 js hono #5147 vite #22992 php doctrine/dbal #7392 laravel/framework #60916 py pytest #14730 poetry #10982 rust reqwest #3064 ripgrep #3496 All merged 2026-07-02..2026-08-01, clearing Claude Opus 5's documented May 2026 training cutoff by ~2 months. Report covers gap analysis, contamination boundary, a 46-repo search log, draft task instructions, per-case derivability matrices, harness plans, alternates, and a ranked recommendation. Two candidates were hard-rejected by the derivability audit and are documented as such: pytest #14331 (hidden test imports a private helper created by the gold patch) and laravel #60877 (test pins ~40 maintainer-chosen cron normalization policies). Not yet proven: no candidate has had fail-to-pass executed; scripts/verify_case.sh on all ten is the first implementation step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds cases/dev/{cpp,js,php,python,rust}/ dirs for the ten candidates from the
batch-5 mining report — case.yaml + setup.sh + quality.sh + validate.sh each,
ids dev5_*.
cpp libgit2_revwalk_pathspec_root cpphttplib_query_verbatim
js hono_csp_dual_policy vite_hmr_restart_stale
php dbal_sqlite_alter_preserves_constraints laravel_queuefake_bulk_delay
python pytest_no_summary_hook_scope poetry_show_outdated_explicit_source
rust reqwest_body_timeout_classification ripgrep_maxdepth_ignore_skip
NOT VERIFIED. No setup/quality/validate script has been executed and no
fail-to-pass leg has been proven; every case.yaml carries meta.verified: false.
scripts/verify_case.sh on all ten is the gate before benchmarking.
What was checked statically:
- all 20 base/gold SHAs re-resolved against the GitHub API (first parent of the
gold commit; libgit2 and dbal are true merge commits, the rest squashes)
- changed production paths and hidden test paths confirmed per PR
- all .sh files pass bash -n; all case.yaml pass a structural key/SHA check
- 50 unique case ids across 50 case dirs
Leak hygiene follows AGENTS.md: setup.sh fetches only BASE_SHA at depth 1 and
drops the remote; validate.sh re-adds it, fetches GOLD_SHA at verify time and
checks out ONLY the hidden test paths. reqwest and ripgrep keep their tests
inside the production file, so validate.sh grafts gold's #[cfg(test)] module
onto the agent's file (same pattern as uuid_parse_panic).
Two report corrections found while writing the scripts: the cpp-httplib target
is httplib-test and must run from build/test (fixture files), and that case
needs libcurl4-openssl-dev added to docker/Dockerfile.agent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Member
|
Landed on master rebased onto the new tree layout: 565b092 (mining report) + 9021743 (scaffolds) + 9e70f3d (relocation to cases/dev/oneshot/) + 5974cc5 (setup fixes from the first verify_case.sh pass: libgit2 NTLM flag, poetry addopts, pytest setuptools_scm version, vite package build). Remaining pre-run repairs (laravel composer resolution, ripgrep gold diff, reqwest base leg) are being fixed on master; meta.verified flips as each case passes fail-to-pass. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a mined batch of ten real-commit
cases/devcandidates plus their scaffolding.Two commits:
cases/_specs/batch5-public-workflow.md— the full mining report.cases/dev/*/*/—case.yaml+setup.sh+quality.sh+validate.sh, idsdev5_*.No
setup.sh,quality.shorvalidate.shin this PR has been executed, and no fail-to-pass leg has been proven. Everycase.yamlcarriesmeta.verified: false.scripts/verify_case.shon all ten is the gate before any of these are benchmarked — treat the scripts as reviewed drafts, not working cases.What was checked, statically:
.shpassbash -n; allcase.yamlpass a structural key/SHA check; 50 unique ids across 50 case dirs.The batch: public workflow / black-box contract
The current 40-case corpus is heavily library-internal — 17 of 40 are "feed one input to one function, check the value or exception" (7 parser cases, 4 crash fixes, plus isolated boundary fixes). Only two reproduce through a lifecycle, and only one is driven from a CLI's visible output.
Every candidate here fails through a documented public surface — a CLI flag, a config file, an HTTP header, a public SDK call, an on-disk artifact — and most need more than one user action to observe.
Ten distinct orgs, all new to the corpus. One Lane A (support ticket) + one Lane B (acceptance story) per language. 7 behaviour categories, none used more than twice.
Contamination boundary
Newest model in the active cases matrix is
claude-opus-5, documented training-data cutoff May 2026. Cutoffs forgpt-5.6-sol,glm-5.2,gemini-3.5-flash,minimax-m3,kimi-k2.6anddeepseek-4-procould not be verified from an authoritative source, so rather than invent one the report uses a conservative 30-day window: merged on or after 2026-07-01. All ten landed 2026-07-02 … 2026-08-01.Docker change required
libcurl4-openssl-devfor the cpp-httplib case —test/CMakeLists.txtdoesfind_package(CURL REQUIRED). Itssetup.shinstalls it defensively so the case is runnable before the image is rebuilt. No other case needs a toolchain addition.Leak hygiene
Per AGENTS.md:
setup.shfetches onlyBASE_SHAat depth 1 and drops the remote, so gold is not in the object store during the agent run;validate.shre-adds the remote, fetchesGOLD_SHAat verify time and checks out only the hidden test paths.reqwest and ripgrep keep their tests inside the production file, so
validate.shgrafts gold's#[cfg(test)]module onto the agent's file rather than checking the whole file out — same pattern as the existinguuid_parse_paniccase. For ripgrep that graft also drags in the crate's pre-existing walk tests, so one run covers fail-to-pass and the regressions.libgit2's clar suite index is generated from a configure-time glob, so its
validate.shre-runscmakeafter dropping in the new test file.Two hard rejections worth reading
The derivability audit killed two otherwise-excellent candidates:
_chmod_rwx, a private helper created by the gold patch. Not derivable from any prompt.schedule:listtimezone conversion. The data provider pins ~40 maintainer-chosen cron normalization policies. Stating them is dictating the algorithm.Also rejected on the human-work bar: CLI11 (PR body opens with ":robot: AI text below 🤖"), libuv #5206 (author discloses Codex assistance), clap #6409 and googletest #5041/#5039 (production-only, no tests).
Known risks, all flagged in the report
createServermay need a build step). Most likely of the ten to need rework.Co-authored-bytrailer; documented swap-in alternate is pytest #14692.The report's §8 carries two or more alternates per language with concrete rejection reasons, and §9 ranks all ten by expected benchmark value.
🤖 Generated with Claude Code