# Sandbox bench

> Sandbox bench: cloud sandbox providers for AI agents, timed through one SDK (Sandbox SDK, built by us; 0.2.0 is public on npm and the 1.0 used here will be published soon) from one server in Florida, US, on two tracks: each provider's default machine, and the same 4 vCPU, 8 GB machine. Medians, p95s and 95% confidence intervals per provider; every provider-side failure kept, failures caused by our own code set aside.

Providers (id → name): e2b → E2B, daytona → Daytona, upstash → Upstash Box, boxd → boxd, freestyle → Freestyle, tenki → Tenki, superserve → Superserve.

## Tracks

- `default`: Default machines. Each provider's default sandbox, so machine sizes differ. Boat is coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.
- `same-spec`: Same machine: 4 vCPU, 8 GB. Every provider asked for Boat's machine, 4 vCPU and 8 GB, or its nearest size. Boat is coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that. Superserve is not in this track: our Superserve plan caps templates at 2 vCPU, so there was no 4 vCPU machine to test. A paid plan may lift this; we will add it if it does.

The page shows one track at a time: `/?track=same-spec` opens the same-machine track.

## Quick picks

### Default machines

- Fastest cold start: Superserve 0.51 s (runner-up E2B 0.56 s)
- Fastest agent work (tool replay): Daytona 11 s (runner-up Superserve 12 s)
- Fastest 25-sandbox fan-out: Superserve 0.79 s (runner-up E2B 1.0 s)
- Fastest reconnect: Daytona 0.33 s (runner-up Superserve 0.34 s)
- Cheapest task: Upstash Box $0.00023 per task, $0.23 per 1,000 (billed) (runner-up Daytona $0.00024 per task, $0.24 per 1,000 (billed))
- Most reliable: E2B, Daytona, Freestyle, Superserve 100.00% (tied) (runner-up Tenki 99.91%)

### Same machine: 4 vCPU, 8 GB

- Fastest cold start: Freestyle 0.66 s (runner-up E2B 0.95 s)
- Fastest agent work (tool replay): Daytona 11 s (runner-up E2B 15 s)
- Fastest 25-sandbox fan-out: Freestyle 1.4 s (runner-up E2B 1.5 s)
- Fastest reconnect: Daytona 0.31 s (runner-up Upstash Box 0.52 s)
- Cheapest task: Upstash Box $0.00047 per task, $0.47 per 1,000 (billed) (runner-up Daytona $0.0012 per task, $1.19 per 1,000 (billed))
- Most reliable: E2B, Daytona, Tenki 100.00% (tied) (runner-up Freestyle 99.91%)

## Coming in the next update

Boat, and any snapshot or burst a chart marks as coming. Default-machine timings come from one run on one harness commit. Snapshots, bursts, the same-machine track and cost per task come from runs on a later harness commit whose changes only touched cleanup. Tenki's same-machine run stopped early at 78 of 100 cold starts; each chart shows its attempt count. A burst size is published for a provider only when all three of its bursts finished, and a chart names the reason for any provider it leaves out. Failures caused by our own code are set aside, not counted. Boat will be added in the next update.

## Data

- [results.json](/results.json): both tracks. Per provider: what we asked for and measured (allocation, equalSpec), cost per task with its kind (billed, metered, estimate), source and confidence, SDK-ready time, prices, regions and failures. Per metric: median, p95, attempts, failed and status. Start here.
- [results.md](/results.md): the same numbers as Markdown tables, one section per track.

## How to read it

Pick a track first: `tracks.default` compares each provider's default sandbox, so sizes differ; `tracks["same-spec"]` asks every provider for 4 vCPU and 8 GB. In same-spec, `allocation.equalSpec: false` means the machine measured inside was not 4 vCPU, 8 GB, so compare that provider with care. median and p95 are in `unit`; `better` says which direction wins; results are sorted best first. attempts counts every try, failed included. A null median means the provider does not support the test, every attempt failed, or the result is excluded (see `status`). Cold start is the provider's own time, create() until the first command; `sdkReady` adds our SDK's setup and is ours, not the provider's. `costPerTask.kind` is billed (provider charge), metered (provider usage at published rates) or estimate (list price, where billing gives no amount); a task is one sandbox that is created, runs the 100-call tool replay and is deleted, from a separate cost run: never compare them as if they were the same kind without saying so.

Prices cover compute only at list rates; disk, transfer, subscriptions and credits are excluded. Every call starts in Florida, so providers hosted in Europe (Boat in France, boxd in Germany) pay a transatlantic round trip per call. In the same-machine track, usable memory measured inside sits a little under 8 GB; equalSpec allows 7.6 to 8.4 GiB. Amounts are US dollars unless marked EUR.

## Metrics

- Start-up: Cold start (`cold`)
- Agent work: Agent tool replay (`tool-replay`), npm install (`npm-install`), pip install (`pip-install`), git clone (`git-clone`), Dev server ready (`dev-server`), Preview URL first byte (`preview-first-byte`), Edit, rebuild, reload (`edit-loop`)
- Everyday calls: Run a command (`command`), Write + read 1 KB (`files-1kb`), Write + read 10 MB (`files-10mb`), Reconnect (`reconnect`), Destroy (`destroy`)
- Pause and snapshots: Pause + resume (`pause-resume`), Take snapshot (`snapshot`), Start from snapshot (`create-from-snapshot`)
- Bursts: 10 at once, all ready (`burst-10`), 25 at once, all ready (`burst-25`)
- Machine speed: CPU, one core (`cpu-single`), CPU, all cores (`cpu-all`), Memory bandwidth (`memory`), Disk write (`disk-write`), Disk read (`disk-read`), 4K random write (`iops-write`), 4K random read (`iops-read`)