Computatrum, the desktop agent#

Computatrum is ComputaCo's desktop app that uses a person's own computer for them: they type a task into a small floating bar, and an agent moves the pointer and types while they watch, with a Stop button and a stop key always in reach. It runs on macOS, Windows and Linux. Its models are reached only through the Computaco API (the hosted API § Computatrum), so the app never holds a provider key, and every call is metered into the account's 5-hour and weekly windows.

The source is computatrum/ (All Rights Reserved, COMPUTACO INC.). This page is how it works, how it is kept safe, how it is tested and how it is released.

What the person sees#

• The command bar. A frameless, always-on-top bar at the top of the screen: one field ("What should I do on this computer?") and Run. Ctrl+Shift+Space (⌘⇧Space on macOS) brings it back from anywhere.
• While a task runs the bar docks in the bottom-right corner, out of the way of the app the agent works in, and shows the task, a large red Stop, and a live feed of the steps in plain words ("Enter Ada Lovelace as Name", "Asking you: Place order").
• The overlay. Before each pointer or element action, a ring and a label mark where it will land, for about half a second. The marker never takes input and is hidden before the action is sent.
• Pauses. The moment the person moves the mouse or types, the agent pauses ("Paused — you're using the computer") and sends nothing until they press Resume (or Ctrl+Shift+Return).
• Confirmations. A step that sends, posts, submits, deletes, buys, pays, transfers, installs or changes account settings stops and asks: Allow once or Deny. The card is never focused automatically, so a stray Enter in another app cannot approve a purchase.
• Hand-over. Passwords, one-time codes, card numbers, recovery phrases and API keys are never typed by the agent. It stops with "Your turn", the person types the secret, and presses Continue.
• The quota meter. The footer shows the 5-hour and weekly windows as bars, with the reset time on hover; a task that runs out says when the quota resets.
• Sign-in. "Sign in with the browser" starts a device sign-in: the bar shows a code, the browser opens the console's device page, and once the person approves it the app holds an agent token (cwt_…) that reaches only the model calls and their usage. A developer can paste a cw_live_ key instead.
• Stop. The Stop button, Escape in the bar, or the global key (Ctrl+Shift+Escape; ⌘⇧Escape on macOS; Ctrl+Alt+End on Windows, where Ctrl+Shift+Escape is Task Manager's) aborts at once: the action in flight is killed, any key or button it may have held is released, and nothing more is sent.

Architecture#

  command bar (src/ui/bar.*)  ── IPC (preload.cjs) ──  main process (src/main/main.mjs)
                                                         │  activity.mjs   pause on the person's pointer/keys
                                                         │  overlay.mjs    the marker window
                                                         │  store.mjs      the sign-in, in the OS keyring
                                                         ▼
                                     agent loop (src/agent/loop.mjs) ── risk.mjs (confirm / hand over)
                                       │ look            │ read            │ decide
                                       ▼                 ▼                 ▼
                       harness (src/harness/)   POST /v1/agent/vision   POST /v1/agent/chat
                       capture · a11y tree ·     (Workers AI vision)     (deepseek-flash)
                       input · settle · stale        └──── src/agent/api.mjs ────┘

One step of the loop.

1. Look: a fresh frame of the screen (at most 1280 px on the long edge), the accessibility tree where the platform has one (AT-SPI on Linux), and the focused window. Computatrum's own windows are painted out of every capture inside the harness, so the bar is never in a frame, a comparison or a model's view.
2. Read: when the screen changed since the last description, the frame goes as JPEG to POST /v1/agent/vision, which describes the windows, controls (with approximate coordinates), field contents and messages. This is the only place a screenshot leaves the computer.
3. Decide: POST /v1/agent/chat (deepseek-flash) gets the task, the accessibility outline, the description and the steps so far — text only — and answers one JSON decision: an action (click, press, set_text, type, key, scroll, drag, wait), locate (ask the vision model where something is), ask_user, done or fail. Every action carries an intent the person reads and a declared risk.
4. Check: risk.mjs asks for confirmation or hands the step over (§ safety).
5. Show: the overlay marks the target.
6. Act: the harness sends the action once, against the frame it was planned on; if the screen changed meanwhile it refuses as stale and nothing is sent. It waits for the screen to settle and answers what changed, which the next step reads.

Grounding prefers the accessibility tree: a control named in it is acted on by id (press, set_text, or a click at its centre), so coordinates come from the platform, not from the vision model's estimate. On a screen with no tree, the vision description and locate ground pixel clicks.

The harness: what came from CommandAGI#

The harness is CommandAGI's computer-use implementation (CommandAGI INC. is ComputaCo's parent; the reuse is authorised). Each file names its origin in its header.

Safety#

• The person's input wins. activity.mjs samples the cursor every 80 ms and the system idle time. A pointer move the agent did not make (it is not inside an action, and the pointer is not where the agent left it), or input less than a second ago while the agent has sent nothing for 1.5 s, pauses the task and disarms the harness. A move made while an action runs is caught when it ends. Watching is off while the agent waits on the person.
• Risky steps ask. The model declares a risk on every action, and risk.mjs checks the words of its intent and the control it targets as well; either is enough to ask. A model that forgets to declare a purchase is still asked.
• Secrets are never typed. Typing into a field the tree calls password text, into a field or with an intent naming a credential, or text with a secret's shape (a Luhn-valid card number, a key prefix, a 12–24-word phrase) is handed to the person, whatever the model says.
• Never into itself. The agent's keys are refused while the keyboard focus is on Computatrum; its clicks are refused inside Computatrum's windows. On Run the keyboard is given back to the app the person was last in.
• What leaves the computer. Frames go only to /v1/agent/vision; the chat model gets text. Nothing else is sent: no telemetry, no task history, no screenshots kept. The only network calls are the five agent routes. The sign-in is encrypted with the OS keyring (Keychain, DPAPI, libsecret) where there is one; otherwise it is a file only the user can read, and Settings says so.

Tests#

Nothing in the tests drives the owner's own desktop. computatrum/dev/virtual-display.mjs starts a private Xvfb (it picks its own display number and refuses the owner's), a private D-Bus session with its own AT-SPI bus, and Openbox as the window manager — a user-space copy fetched by dev/fetch-openbox.sh with apt-get download where Openbox is not installed. The computer is opened with requireDisplay set to that display, and the app with COMPUTATRUM_REQUIRE_DISPLAY and COMPUTATRUM_NO_BROWSER (a browser started from the virtual display could reach the owner's own). dev/toy-app.py is the scripted application: a GTK order form (Name, Email, Gift wrap, Next, a review page with Place order) and a login form, each writing its state to a file the test reads. dev/record.mjs records the display to video.webm in its own process.

cd computatrum
npm run setup                 # npm install, the Electron binary, Openbox for the tests
npm test                      # unit tests: risk, decisions, protocol, usage, device sign-in, 429, stop
npm run test:e2e              # the loop and the whole app in the virtual display, against dev/mock-api.mjs
COMPUTATRUM_API=https://dev.computaco.ai COMPUTATRUM_TOKEN=cwt_… npm run test:e2e   # the same on the real models
COMPUTATRUM_APP_BINARY=$PWD/dist/linux-arm64-unpacked/computatrum npm run test:e2e  # a packaged build

dev/mock-api.mjs serves the same contract as the hosted API, with a scripted planner in place of the chat model (it reads the accessibility outline the app sends) and a vision endpoint that checks the image is a real JPEG. It logs every request, and the tests assert that no image ever reaches the chat endpoint.

Measured on 2026-10-28 on this machine (aarch64, Ubuntu 24.04, Electron 44.7.2):

The real runs used deepseek-flash for planning and @cf/zai-org/glm-5.3-flash for vision, and a free dev account signed in through the device flow (approved from its console session). Screenshots from these runs are in computatrum/docs/screens/.

Building and releasing#

npm run dist:linux     # AppImage, tar.gz and deb in dist/
npm run dist:mac       # on macOS: universal dmg and zip
npm run dist:win       # on Windows: NSIS installer and zip

The Linux AppImage, tarball and deb build here (126 MB, 120 MB and 94 MB for arm64; the deb in 1 m 28 s with the rest, electron-builder fetching an arm64 fpm). The deb depends on xdotool, x11-apps, x11-xserver-utils, python3-gi and gir1.2-atspi-2.0. An AppImage cannot declare those: the bar says what is missing ("This computer is not ready: xdotool not found…").

.github/workflows/computatrum.yml builds every installer on hosted runners: Linux x64 and arm64 (AppImage, deb, tar.gz), macOS universal (dmg, zip) and Windows x64 (NSIS exe, zip), and uploads each as a workflow artifact. A tag computatrum-v<version> (it must be package.json's version) also publishes them to R2 with latest.json (scripts/publish-downloads.mjs), which https://computaco.ai/download/computatrum/ serves (cloud-api § Downloads); running it by hand can publish to the dev tier's bucket instead. Publishing needs the repository secret CLOUDFLARE_API_TOKEN (an account API token with Workers R2 Storage: Edit). Until a Developer ID exists the macOS app is ad-hoc signed without the hardened runtime (build/unsigned-mac.yml), so Apple silicon runs it once the person allows it in Gatekeeper; each signing step runs only when its secret below is set.

What remains for signed releases:

• macOS signing and notarization. A "Developer ID Application" certificate of COMPUTACO INC. (CSC_LINK, CSC_KEY_PASSWORD) and notarization credentials (APPLE_ID, APPLE_APP_SPECIFIC_PASSWORD, APPLE_TEAM_ID, or an App Store Connect API key), then -c.mac.notarize=true. The hardened runtime and entitlements are already configured.
• macOS pointer tool. The harness's macOS input uses cliclick for the pointer; it must be bundled (built universal and signed) into build/bin/mac/, which ships as resources/bin and is put first on the PATH. The first run needs the person to grant Accessibility and Screen Recording in System Settings → Privacy & Security; an onboarding screen for that is not built yet.
• Windows signing. An Authenticode certificate (WIN_CSC_LINK, WIN_CSC_KEY_PASSWORD) or Azure Trusted Signing; unsigned builds meet SmartScreen.
• Not yet run on macOS or Windows. Their capture and input commands come from CommandAGI and are unit-checked as written; neither OS has run Computatrum. The accessibility tree is read on Linux only (macOS AX and Windows UI Automation are not built), so there the agent grounds with vision alone.
• Updates. No update feed is configured (electron-updater and a publish target).
• Wayland. Input on Wayland uses ydotool, which needs ydotoold and /dev/uinput; the overlay's shaped window is X11-only.