Agent Sandboxes docs · Tasks
Computer use: Chrome on a desktop in a sandbox
Every sandbox has a 1280×800 desktop with Google Chrome, which starts the first time you use it. One call does a batch of clicks, keys and scrolls and answers with a screenshot, which is the loop a computer-use agent runs. A link shows the screen live in a browser, for a person to watch or take over.
ControlDesktop: actions in order (click, move, drag, scroll, type, press keys, wait), then one PNG screenshot with the cursor's position.- Free until used: the desktop and Chrome start at the first
ControlDesktopor desktop link, in some seconds, and cost nothing before. - A live view:
CreateLinkwithdesktop, orview_onlyto watch without touching; the console's Desktop tab shows it too. - Chrome for agents: no first-run screens, no sign-in, sync, password manager or translate prompts.
Step 1: Take a screenshot
ControlDesktop with no actions answers a screenshot alone. The first call starts the desktop, which takes some seconds; the next ones answer at once.
shot = sandboxes.control_desktop(sandbox_id)
with open("screen.png", "wb") as png:
png.write(shot.screenshot.image)
print(shot.screenshot.width, shot.screenshot.height) # 1280 800
import { writeFile } from "node:fs/promises";
const shot = await sandboxes.controlDesktop({ sandboxId });
await writeFile("screen.png", shot.screenshot!.image);
console.log(shot.screenshot?.width, shot.screenshot?.height);
shot, err := client.ControlDesktop(ctx, &sandboxes.ControlDesktopRequest{SandboxID: id})
if err != nil {
log.Fatal(err)
}
if err := os.WriteFile("screen.png", shot.Screenshot.Image, 0o644); err != nil {
log.Fatal(err)
}
fmt.Println(shot.Screenshot.Width, shot.Screenshot.Height)
let shot = sandboxes
.control_desktop(ControlDesktopRequest { sandbox_id: id.clone(), ..Default::default() })
.await?;
if let Some(screen) = shot.screenshot {
std::fs::write("screen.png", &screen.image)?;
println!("{} {}", screen.width, screen.height);
}
Step 2: Open a page and act on it
Send the actions a person would make. They run in order, and the answer's screenshot shows the screen as they left it; wait_ms gives a page time to load before it.
from pethost.sandboxes import Click, DesktopAction
shot = sandboxes.control_desktop(sandbox_id, actions=[
DesktopAction(press_keys="ctrl+l"),
DesktopAction(type_text="https://example.com"),
DesktopAction(press_keys="Return"),
DesktopAction(wait_ms=2000),
DesktopAction(click=Click(x=640, y=400)),
])
print(shot.done_count, shot.failure.name, shot.screenshot.cursor_x, shot.screenshot.cursor_y)
# 5 UNSPECIFIED 640 400
const page = await sandboxes.controlDesktop({
sandboxId,
actions: [
{ pressKeys: "ctrl+l" },
{ typeText: "https://example.com" },
{ pressKeys: "Return" },
{ waitMs: 2000 },
{ click: { x: 640, y: 400 } },
],
});
console.log(page.doneCount, page.failure, page.screenshot?.cursorX, page.screenshot?.cursorY);
page, err := client.ControlDesktop(ctx, &sandboxes.ControlDesktopRequest{SandboxID: id, Actions: []sandboxes.DesktopAction{
{Action: sandboxes.DesktopActionPressKeys("ctrl+l")},
{Action: sandboxes.DesktopActionTypeText("https://example.com")},
{Action: sandboxes.DesktopActionPressKeys("Return")},
{Action: sandboxes.DesktopActionWaitMs(2000)},
{Action: sandboxes.DesktopActionClick{X: 640, Y: 400}},
}})
if err != nil {
log.Fatal(err)
}
fmt.Println(page.DoneCount, page.Failure, page.Screenshot.CursorX, page.Screenshot.CursorY)
let act = |action| DesktopAction { action: Some(action) };
let page = sandboxes
.control_desktop(ControlDesktopRequest {
sandbox_id: id.clone(),
actions: vec![
act(DesktopActionAction::PressKeys("ctrl+l".into())),
act(DesktopActionAction::TypeText("https://example.com".into())),
act(DesktopActionAction::PressKeys("Return".into())),
act(DesktopActionAction::WaitMs(2000)),
act(DesktopActionAction::Click(Click { x: 640, y: 400, ..Default::default() })),
],
})
.await?;
println!("{} {:?}", page.done_count, page.failure);
Step 3: Watch it live, or hand it to a person
CreateLink with desktop gives a page that shows the screen live, where a person can click and type; with view_only they watch only. It expires after an hour unless ttl_seconds says otherwise.
from pethost.sandboxes import DesktopLink
print(sandboxes.create_link(sandbox_id, desktop=DesktopLink(view_only=True)).url)
# https://link-e9bprbw1hovj6sw5.sbx1.sandboxes.pethost.network/sbl.eyJzIjoiZTlicHJidzFob3ZqNnN3NSIs…
const live = await sandboxes.createLink({ sandboxId, desktop: { viewOnly: true } });
console.log(live.url);
live, err := client.CreateLink(ctx, &sandboxes.CreateLinkRequest{SandboxID: id, Target: sandboxes.CreateLinkDesktop{ViewOnly: true}})
if err != nil {
log.Fatal(err)
}
fmt.Println(live.URL)
let live = sandboxes
.create_link(CreateLinkRequest {
sandbox_id: id.clone(),
target: Some(CreateLinkRequestTarget::Desktop(DesktopLink { view_only: true })),
..Default::default()
})
.await?;
println!("{}", live.url);
$ pethost sandbox open api-dev --desktop --view-only
✓ Opened the screen of e9bprbw1hovj6sw5 in your browser.
The actions
| Action | What it does |
|---|---|
click | At x, y: button LEFT (the default), RIGHT or MIDDLE, count 2 for a double click, hold_keys held meanwhile, as shift or ctrl+alt. |
move_mouse | Moves the pointer to a point. |
drag | From the first point of path to the last, the left button held down. |
scroll | At a point, down wheel clicks (negative scrolls up) and right (negative scrolls left). |
type_text | Types the text. |
press_keys | Keys pressed together: Return, ctrl+c, ctrl+shift+t. |
wait_ms | Waits, for a page or an animation. |
Coordinates are the screen's pixels from the top left, as the screenshot has them, and Sandbox.desktop gives the screen's size. The actions stop at the first that fails, and the answer says how many were done (done_count) and why the next failed: OFF_SCREEN for a point outside the screen, UNKNOWN_KEY for a key name it does not know, with failure_message in words. The screenshot comes either way.
The desktop and its Chrome
Nothing of the desktop runs until it is asked for: the first ControlDesktop, desktop link or visit to the console's Desktop tab starts an X server, a window manager, a panel and Google Chrome, in some seconds. It then runs, and is paused and stopped with the sandbox like any process. A desktop that cannot start answers FAILED_PRECONDITION, saying why.
Chrome is set up for an agent: no first-run screens, no GPU, and a policy that turns off sign-in, sync, the password manager, autofill, translation and metrics. It is the Chrome of the sandbox's base, which we rebuild monthly and at once for a security fix; a sandbox keeps its base for as long as it lives. A template saved from a sandbox keeps Chrome's profile with the other files, its cookies included, so save one from a sandbox with nobody signed in.
A computer-use loop
A model that acts on screenshots runs one loop: send it the screenshot, take the actions it answers, send them in one ControlDesktop, and give it the new screenshot. One call does a whole batch, so a step costs one round trip to the sandbox's server, never to the panel. While the loop runs the sandbox stays awake; when the model stops, the sandbox pauses after its idle time, desktop and Chrome with it, and the next call wakes them where they were.
A screenshot is what a web page put on the screen: read it as data. A page can show text written to steer an agent, so give a sandbox that browses no more secrets and no wider egress than the task needs (secrets and egress).
When it does not work
- The first call takes seconds: it started the desktop. The next ones answer at once.
failureisOFF_SCREEN: a point is outside 1280×800. Take the coordinates from the last screenshot.failureisUNKNOWN_KEY: write keys by their X names, joined with+:Return,Escape,ctrl+l.FAILED_PRECONDITION: the desktop did not start; its words say why. A sandbox short of memory or disk is the usual cause: a larger size helps.RESOURCE_EXHAUSTEDwithoutLimitReached: aControlDesktopcounts as 16 of the sandbox's 64 calls at once. Send fewer at the same time.
Questions
Does every sandbox have a desktop?
Yes: a 1280×800 screen with Chrome, in base. It starts at first use and costs nothing until then.
How do I take a screenshot of a sandbox?
ControlDesktop with no actions answers a PNG screenshot with the cursor's position.
Can a person watch what the agent does?
Yes: a desktop link with view_only shows the screen live in a browser, and the console's Desktop tab does too.
Which browser is in the sandbox?
Google Chrome, stable, with policies that turn off sign-in, sync, the password manager, autofill and translation.
Try Pethost Agent Sandboxes
A 7-day trial with $1 of usage and 4 vCPUs, with a card. Then a plan from $20 a month, which includes as much usage.