Fix flaky tests · Kestrel Docs

A flaky test is the perfect task for an agent: easy to state, slow to chase, and simple to check. This guide takes one from a failing CI job to a merged fix.

Find the test

Pick the test and how often it fails. Your CI history has both. Here, checkout.test.ts fails about one run in ten.
Ask for proof, not a guess

A fix for a flaky test is only proven by running the test many times. Tell Kestrel how many with --repeat-tests. Twenty passes in a row makes a 1-in-10 flake very unlikely to have survived.
```
kestrel run "fix the flaky checkout test" --repeat-tests 20
```
```
await kestrel.runs.create({
  repo: "acme/web-app",
  task: "Fix the flaky checkout test",
  evals: { repeatTests: 20 },
});
```
Let the agent reproduce it first

The agent runs the test in a loop until it fails, then reads the failure. In the trace you'll see a reproduce step before any edit. A fix that comes without a reproduction is a guess, and the evals reject it.
Read the fix

When all twenty runs pass, Kestrel opens the pull request. The description names the cause, for example a timer that wasn't awaited, and the trace shows every repeat.
Terminal
```
✓ sandbox ready              0.8 s
✓ reproduced on try 7
✓ 20 of 20 repeats passed
✓ pull request #418 opened
```
Teach it the pattern

If the cause is common in your codebase, add it to memory so the next run checks for it first.
Terminal
```
kestrel memory add --repo acme/web-app --topic tests "Await clock.runAllAsync() after advancing fake timers"
```

Tip

Start a run for each flaky test at once. They queue and run side by side, and each opens its own pull request.

Find the test

Ask for proof, not a guess

Let the agent reproduce it first

Read the fix

Teach it the pattern