# Retries

Network errors and busy models are retried with growing waits between tries. A run is never started twice, because each request carries an idempotency key.

![A call fails and is tried again after a short wait. Each failure doubles the wait, with a little jitter, until a try succeeds.](/illos/retry-backoff.svg)

## What is retried

The SDKs retry a request when it fails for a reason that may pass: a dropped connection, a timeout, a `429`, or a `5xx`. They don't retry a `4xx` that will fail the same way again, like a bad key.

Waits double each time, with up to 25% jitter, so a thousand clients that fail together don't all come back together.

| Try | Wait before it |
|---|---|
| 1 | none |
| 2 | about 1 s |
| 3 | about 2 s |
| 4 | about 4 s |
| 5 | about 8 s |

A `Retry-After` header always wins over the table.

## Change it

```ts TypeScript
const kestrel = new Kestrel({ maxRetries: 2 });
// or for one call
await kestrel.runs.create({ repo: "acme/web-app", task: "Bump lodash" }, { maxRetries: 0 });
```

```python Python
kestrel = Kestrel(max_retries=2)
# or for one call
kestrel.runs.create(repo="acme/web-app", task="Bump lodash", max_retries=0)
```

## Idempotency keys

A retried `POST /runs` could start the same task twice. The SDKs send an `Idempotency-Key` header with each create, and reuse it on every retry, so Kestrel starts the run once and returns the same run to every try.

```bash Terminal
curl https://api.kestrel.dev/v1/runs \
  -H "Authorization: Bearer $KESTREL_API_KEY" \
  -H "Idempotency-Key: 4f1c2a8e-ci-build-8812" \
  -d '{ "repo": "acme/web-app", "task": "Bump lodash" }'
```

> [!NOTE]
> Keys are kept for 24 hours. Use something tied to the job, like a CI build number, so a re-run of the same build doesn't start a second Kestrel run.

## Retries inside a run

Inside a sandbox, Kestrel also retries the steps that fail for passing reasons: a model that is busy, or a package registry that times out. Those retries show in the [trace](/runs/traces) as extra tries on one step, not as new steps.