Code repair loop

This experiment uses test output to drive another code-generation attempt. Each attempt produces a patch, runs in an isolated environment, and records the result.

The public notes describe the loop. They do not include a reproducible benchmark or a measured improvement in success rate.

Loop structure

Generate patch
Apply to a disposable workspace
Run fixed tests with a timeout
Save diff, exit code, stdout, and stderr
Return failures for the next attempt
Stop on success, attempt limit, deadline, or budget limit

Keep the test suite outside the model’s writable workspace. Otherwise the model can make a run pass by changing the test.

Preserve each patch and its test output. A final passing result is easier to investigate with the failed attempts still available.

Execution limits

Use an isolation mechanism with explicit CPU, memory, process, network, and filesystem limits. A subprocess timeout alone does not provide those controls.

Run without production credentials. Mount only the inputs required for the task, and discard the workspace between unrelated runs.

Evaluation

Use a fixed set of failures with expected repairs. Count successful fixes, regressions, timeouts, attempts, and total model cost.

Tests cover the behavior they exercise. Add cases for the original defect and likely regressions before judging a patch complete. Keep code review for changes involving permissions, data handling, or dependencies.

Technologies: Python, Sandboxed execution, Automated tests.