Computer Use

Computer Use lets an agent inspect a screenshot and interact with a computer using clicks, typing, keys, scrolling, and dragging. Jian Remote supplies a native helper for the OS; the gateway coordinates the task and passes its result back into the conversation.

It operates the desktop session you authorize. It is not a browser-only automation environment, and it does not give the model a separate sandboxed copy of your desktop.

Requirements by platformCopy Anchor Link

Published worker packages ship platform helpers. Source builds need the appropriate helper build; installing the JavaScript CLI from an incomplete source checkout alone does not create a native helper.

macOS permissionsCopy Anchor Link

For the standalone worker, grant Jian Remote access in System Settings → Privacy & Security → Screen Recording and Accessibility. For an app-managed worker, grant the identity named by the application and macOS prompt. If permissions were changed, follow any restart instruction macOS gives and retry a screenshot and a simple input action separately.

Do not assume that granting access to Terminal grants it to the background worker. Permissions attach to the responsible app identity.

A typical interactionCopy Anchor Link

1. Choose the correct connected machine and verify it is online.
2. Ask the agent to inspect the screen before acting.
3. The agent calls remote_computer with action: screenshot and reads the returned dimensions.
4. It performs the relevant input and captures the screen again after meaningful changes.
5. It calls end_session when finished, clearing the visible Jian pointer overlay.

Example request:

On Mira workstation, inspect the open spreadsheet and tell me which columns it contains. Do not edit or save anything.

Then expand scope deliberately:

Fill the empty status cells with “Pending,” show me the result, and wait before saving.

Use Ask first when you want approvals for actions. See protection modes; Computer Use is still a machine tool subject to that policy.

Actions and coordinatesCopy Anchor Link

The supported action names are screenshot, left_click, right_click, double_click, mouse_move, type, key, scroll, left_click_drag, wait, and end_session.

Coordinates are integer pixels in the returned screenshot image, with the origin at the top-left. Use widthPx and heightPx, not an assumed physical-monitor resolution. The helper can scale a screenshot to fit the transport budget and provides scaling information where needed.

An individual task defaults to 30 seconds and can request up to 120 seconds. A workflow can require several tasks; a timeout is not proof that the intended UI change completed.

Locked, sleeping, and offlineCopy Anchor Link

Online status comes from the worker's polling. macOS additionally reports session states such as locked or display_asleep.

A post-login locked Mac can remain online, with shell and file tasks still available. The helper wakes the display and attempts Computer Use; OS restrictions and the actual tool result decide what succeeds. This does not promise that an agent can unlock the OS or bypass authentication.

Full system sleep, such as a sleeping laptop with its lid closed, prevents polling until the machine wakes. The pre-login/FileVault screen before the user has signed in is different from a locked session after login.

Choose the appropriate toolCopy Anchor Link

Prefer a supported API or CLI when it is more precise for the task. Computer Use is valuable when the work requires a visual application, a desktop state, or a workflow with no suitable programmatic interface.

Gateway-side jian-display is a separate mechanism: the server container can run a temporary authenticated Xvfb display for a command. It is not the owner's remote desktop or a persistent VNC session. See workspaces and tools.

TroubleshootingCopy Anchor Link

Implementation referenceCopy Anchor Link

Action/result contract and native helper source.