English
Docs menu

Agent Sandboxes docs · Tasks

Computer use: Chrome on a desktop in a sandbox

Updated · 5 min read

Every sandbox has a 1280×800 desktop with Google Chrome, which starts the first time you use it. One call does a batch of clicks, keys and scrolls and answers with a screenshot, which is the loop a computer-use agent runs. A link shows the screen live in a browser, for a person to watch or take over.

In short
  • ControlDesktop: actions in order (click, move, drag, scroll, type, press keys, wait), then one PNG screenshot with the cursor's position.
  • Free until used: the desktop and Chrome start at the first ControlDesktop or desktop link, in some seconds, and cost nothing before.
  • A live view: CreateLink with desktop, or view_only to watch without touching; the console's Desktop tab shows it too.
  • Chrome for agents: no first-run screens, no sign-in, sync, password manager or translate prompts.

Step 1: Take a screenshot

ControlDesktop with no actions answers a screenshot alone. The first call starts the desktop, which takes some seconds; the next ones answer at once.

Python
shot = sandboxes.control_desktop(sandbox_id)
with open("screen.png", "wb") as png:
    png.write(shot.screenshot.image)
print(shot.screenshot.width, shot.screenshot.height)  # 1280 800
TypeScript
import { writeFile } from "node:fs/promises";

const shot = await sandboxes.controlDesktop({ sandboxId });
await writeFile("screen.png", shot.screenshot!.image);
console.log(shot.screenshot?.width, shot.screenshot?.height);
Go
shot, err := client.ControlDesktop(ctx, &sandboxes.ControlDesktopRequest{SandboxID: id})
if err != nil {
	log.Fatal(err)
}
if err := os.WriteFile("screen.png", shot.Screenshot.Image, 0o644); err != nil {
	log.Fatal(err)
}
fmt.Println(shot.Screenshot.Width, shot.Screenshot.Height)
Rust
let shot = sandboxes
    .control_desktop(ControlDesktopRequest { sandbox_id: id.clone(), ..Default::default() })
    .await?;
if let Some(screen) = shot.screenshot {
    std::fs::write("screen.png", &screen.image)?;
    println!("{} {}", screen.width, screen.height);
}

Step 2: Open a page and act on it

Send the actions a person would make. They run in order, and the answer's screenshot shows the screen as they left it; wait_ms gives a page time to load before it.

Python
from pethost.sandboxes import Click, DesktopAction

shot = sandboxes.control_desktop(sandbox_id, actions=[
    DesktopAction(press_keys="ctrl+l"),
    DesktopAction(type_text="https://example.com"),
    DesktopAction(press_keys="Return"),
    DesktopAction(wait_ms=2000),
    DesktopAction(click=Click(x=640, y=400)),
])
print(shot.done_count, shot.failure.name, shot.screenshot.cursor_x, shot.screenshot.cursor_y)
# 5 UNSPECIFIED 640 400
TypeScript
const page = await sandboxes.controlDesktop({
  sandboxId,
  actions: [
    { pressKeys: "ctrl+l" },
    { typeText: "https://example.com" },
    { pressKeys: "Return" },
    { waitMs: 2000 },
    { click: { x: 640, y: 400 } },
  ],
});
console.log(page.doneCount, page.failure, page.screenshot?.cursorX, page.screenshot?.cursorY);
Go
page, err := client.ControlDesktop(ctx, &sandboxes.ControlDesktopRequest{SandboxID: id, Actions: []sandboxes.DesktopAction{
	{Action: sandboxes.DesktopActionPressKeys("ctrl+l")},
	{Action: sandboxes.DesktopActionTypeText("https://example.com")},
	{Action: sandboxes.DesktopActionPressKeys("Return")},
	{Action: sandboxes.DesktopActionWaitMs(2000)},
	{Action: sandboxes.DesktopActionClick{X: 640, Y: 400}},
}})
if err != nil {
	log.Fatal(err)
}
fmt.Println(page.DoneCount, page.Failure, page.Screenshot.CursorX, page.Screenshot.CursorY)
Rust
let act = |action| DesktopAction { action: Some(action) };
let page = sandboxes
    .control_desktop(ControlDesktopRequest {
        sandbox_id: id.clone(),
        actions: vec![
            act(DesktopActionAction::PressKeys("ctrl+l".into())),
            act(DesktopActionAction::TypeText("https://example.com".into())),
            act(DesktopActionAction::PressKeys("Return".into())),
            act(DesktopActionAction::WaitMs(2000)),
            act(DesktopActionAction::Click(Click { x: 640, y: 400, ..Default::default() })),
        ],
    })
    .await?;
println!("{} {:?}", page.done_count, page.failure);

Step 3: Watch it live, or hand it to a person

CreateLink with desktop gives a page that shows the screen live, where a person can click and type; with view_only they watch only. It expires after an hour unless ttl_seconds says otherwise.

Python
from pethost.sandboxes import DesktopLink

print(sandboxes.create_link(sandbox_id, desktop=DesktopLink(view_only=True)).url)
# https://link-e9bprbw1hovj6sw5.sbx1.sandboxes.pethost.network/sbl.eyJzIjoiZTlicHJidzFob3ZqNnN3NSIs…
TypeScript
const live = await sandboxes.createLink({ sandboxId, desktop: { viewOnly: true } });
console.log(live.url);
Go
live, err := client.CreateLink(ctx, &sandboxes.CreateLinkRequest{SandboxID: id, Target: sandboxes.CreateLinkDesktop{ViewOnly: true}})
if err != nil {
	log.Fatal(err)
}
fmt.Println(live.URL)
Rust
let live = sandboxes
    .create_link(CreateLinkRequest {
        sandbox_id: id.clone(),
        target: Some(CreateLinkRequestTarget::Desktop(DesktopLink { view_only: true })),
        ..Default::default()
    })
    .await?;
println!("{}", live.url);
CLI
$ pethost sandbox open api-dev --desktop --view-only
✓ Opened the screen of e9bprbw1hovj6sw5 in your browser.

The actions

ActionWhat it does
clickAt x, y: button LEFT (the default), RIGHT or MIDDLE, count 2 for a double click, hold_keys held meanwhile, as shift or ctrl+alt.
move_mouseMoves the pointer to a point.
dragFrom the first point of path to the last, the left button held down.
scrollAt a point, down wheel clicks (negative scrolls up) and right (negative scrolls left).
type_textTypes the text.
press_keysKeys pressed together: Return, ctrl+c, ctrl+shift+t.
wait_msWaits, for a page or an animation.

Coordinates are the screen's pixels from the top left, as the screenshot has them, and Sandbox.desktop gives the screen's size. The actions stop at the first that fails, and the answer says how many were done (done_count) and why the next failed: OFF_SCREEN for a point outside the screen, UNKNOWN_KEY for a key name it does not know, with failure_message in words. The screenshot comes either way.

The desktop and its Chrome

Nothing of the desktop runs until it is asked for: the first ControlDesktop, desktop link or visit to the console's Desktop tab starts an X server, a window manager, a panel and Google Chrome, in some seconds. It then runs, and is paused and stopped with the sandbox like any process. A desktop that cannot start answers FAILED_PRECONDITION, saying why.

Chrome is set up for an agent: no first-run screens, no GPU, and a policy that turns off sign-in, sync, the password manager, autofill, translation and metrics. It is the Chrome of the sandbox's base, which we rebuild monthly and at once for a security fix; a sandbox keeps its base for as long as it lives. A template saved from a sandbox keeps Chrome's profile with the other files, its cookies included, so save one from a sandbox with nobody signed in.

A computer-use loop

A model that acts on screenshots runs one loop: send it the screenshot, take the actions it answers, send them in one ControlDesktop, and give it the new screenshot. One call does a whole batch, so a step costs one round trip to the sandbox's server, never to the panel. While the loop runs the sandbox stays awake; when the model stops, the sandbox pauses after its idle time, desktop and Chrome with it, and the next call wakes them where they were.

A screenshot is what a web page put on the screen: read it as data. A page can show text written to steer an agent, so give a sandbox that browses no more secrets and no wider egress than the task needs (secrets and egress).

When it does not work

  • The first call takes seconds: it started the desktop. The next ones answer at once.
  • failure is OFF_SCREEN: a point is outside 1280×800. Take the coordinates from the last screenshot.
  • failure is UNKNOWN_KEY: write keys by their X names, joined with +: Return, Escape, ctrl+l.
  • FAILED_PRECONDITION: the desktop did not start; its words say why. A sandbox short of memory or disk is the usual cause: a larger size helps.
  • RESOURCE_EXHAUSTED without LimitReached: a ControlDesktop counts as 16 of the sandbox's 64 calls at once. Send fewer at the same time.

Questions

Does every sandbox have a desktop?

Yes: a 1280×800 screen with Chrome, in base. It starts at first use and costs nothing until then.

How do I take a screenshot of a sandbox?

ControlDesktop with no actions answers a PNG screenshot with the cursor's position.

Can a person watch what the agent does?

Yes: a desktop link with view_only shows the screen live in a browser, and the console's Desktop tab does too.

Which browser is in the sandbox?

Google Chrome, stable, with policies that turn off sign-in, sync, the password manager, autofill and translation.

Try Pethost Agent Sandboxes

A 7-day trial with $1 of usage and 4 vCPUs, with a card. Then a plan from $20 a month, which includes as much usage.

Start the trial