Sandbox providers we benchmark
Sandbox bench times 9 cloud sandbox products for AI agents through one SDK: cold start, real agent work, everyday calls, pause and snapshots, bursts, machine speed, reliability and cost per task. Every timed result is a median with its p95 and 95% interval, on two tracks: every provider asked for the same 4 vCPU, 8 GB machine (or its nearest size), and each provider's default machine. A row marked “machine differs” measured something else inside. Upstash appears twice because Box and Ephemeral Box are separate products.
Sandbox bench runs on Sandbox SDK, which the same team (opencoredev) builds. We may work with some of these providers. No provider paid for these results or decided how they were tested.
Every provider at a glance
Cold start is the provider's own time from create() to the first command. A task is: create a sandbox, run the 100-call tool replay, delete it. Cost per task comes from a separate cost run, billed per sandbox. Cost kinds differ, so compare them with care. Linked names have their own page with every test.
| Provider | Cold start, 4 vCPU, 8 GB | Cold start, default | Cost per 1,000 tasks, 4 vCPU, 8 GB |
|---|---|---|---|
| E2B | 0.79 s 4th of 9 | 0.50 s 1st of 9 | $1.56 (metered) |
| Freestyle | 0.69 s 3rd of 9 | 0.72 s 4th of 9 | $1.97 (estimate) |
| Tenki | 1.1 s 5th of 9, tied | 0.86 s 5th of 9 | $1.79 (estimate) |
| boxd | 1.2 s 7th of 9machine differs | 1.1 s 6th of 9 | $2.00 (metered) |
| Superserve | 0.52 s 1st of 9 | 0.53 s 2nd of 9 | $1.32 (billed) |
| Upstash Box | 2.7 s 9th of 9 | 2.6 s 9th of 9 | $0.34 (billed) |
| Daytona | 1.3 s 8th of 9 | 1.5 s 8th of 9 | $1.36 (billed) |
| Upstash Ephemeral Box | 0.68 s 2nd of 9 | 0.68 s 3rd of 9 | $0.26 (billed) |
| Boat | 1.1 s 6th of 9, tied | 1.2 s 7th of 9 | $0.52 (estimate) |
Not tested yet
Modal, Vercel Sandbox, Cloudflare Sandbox, Runloop, Northflank, Blaxel, Fly Sprites, CodeSandbox and Morph have not been run through the bench, so there are no numbers for them here.
How to read these numbers
- Every call starts from one server in Florida, US. Providers hosted farther away pay that round trip on every call; Boat (France) and boxd (Germany) pay a transatlantic one.
- There are two tracks. Same machine: 4 vCPU, 8 GB: Every provider asked for Boat's machine, 4 vCPU and 8 GB, or its nearest size. A provider marked “machine differs” measured something else inside. Default machines: Each provider's default sandbox, so machine sizes differ.
- Cost per task has a kind. Billed comes from the provider's bill, metered from its usage records at published rates, and estimate from our measured time at list price. Kinds are not directly comparable. Prices cover compute only at list rates: disk, transfer, subscriptions and credits are excluded.
- Pause and resume times come from the previous SDK run, because the newest runs skipped them. A burst size is published only when all three bursts finished.
- Failures caused by our own code are set aside, not counted. Every provider-side failure is kept.
- Ranks compare medians. Medians inside each other's 95% intervals count as tied, so a close rank can be noise.
More results
- E2B sandbox benchmark
- Daytona sandbox benchmark
- Superserve sandbox benchmark
- Upstash Box sandbox benchmark
- Upstash Ephemeral Box sandbox benchmark
- Boat sandbox benchmark
- Freestyle sandbox benchmark
- Tenki sandbox benchmark
- E2B vs Daytona, side by side
- Upstash Box vs E2B, side by side
- Upstash Box vs Daytona, side by side
- E2B alternatives, measured against E2B
- Every provider on the same 4 vCPU, 8 GB machine
- Every provider on its default machine
- Raw data: results.json · results.md · llms.txt