Skip to main content
The StateSet Computer Use Agent now runs on three interchangeable engines and can drive commerce APIs and the GUI in a single loop.

​
Engines

Select the engine per job — from the dashboard, or with --engine {claude,openai,nsr} on the CLI. All three share the cost and summary path, so switching engines does not change how a job is accounted for or reported.

​
Choosing between them

The two vision engines and NSR are different shapes, and the difference decides which to reach for. In a vision-based agent every UI step is an API round-trip: a screenshot goes out, reasoning and a tool call come back. Cost and latency therefore scale with the number of steps, not with how hard the decision was — clicking through four screens of a form costs four round-trips whether or not any of them required judgement. NSR executes those same steps locally, with no API call. So the useful pattern is not “pick one engine” but separate the decision from the execution: let a vision engine handle the step that genuinely needs judgement, and let NSR carry the deterministic navigation around it.
All three share the cost and summary path, so a job that switches engines is still accounted for and reported the same way. That is what makes mixing them practical rather than a special case.

​
Deploying NSR’s perception

NSR does not read pixels through a model API. On the web it takes real DOM element boxes from the accessibility tree; on a desktop it uses a local ONNX UI detector, pointed at by NSR_UI_MODEL_PATH. The detector recognises ten element classes, in a fixed order: button, text_field, link, checkbox, dropdown, menu, tab, icon, label, dialog. Building and validating one:
Then run with it:
The class order is fixed and positional. A detector retrained with the classes in a different order loads without error and then mislabels every element — a button read as a text_field. If you build your own detector, keep the order exactly as above.

​
NSR web capability

NSR drives a headless Playwright browser with no X display and no API calls. A bridge implements the two methods NSR calls — execute and take_screenshot — feeding the a11y_first perception real DOM element boxes rather than pixels. On the realistic vertical benchmark, action-task accuracy went from 0% to 80% (12/15). The remaining misses are the three hardest multi-step purchase and transfer flows. NSR also gained:
  • Real <select> dropdown support (select_option).
  • Perceive-and-scroll to reach below-the-fold elements.
  • Expanded money-action steps so multi-step finance flows complete.

​
Commerce MCP stack

One MCPManager aggregates 25 commerce tools across three servers, so the agent uses APIs where they exist and falls back to the GUI where they don’t — API-first, GUI-fallback, in one loop. Each is a FastMCP stdio server with an agent preset. The Dsco server implements a faithful OAuth2 client-credentials lifecycle per the v3 Standard API guide — mint, cache to expires_in, bearer, and refresh on 401.
Multi-server aggregation required an MCPManager handshake fix; the multi-server plus computer-use loop is verified live.

​
OSWorld benchmarking

A real OSWorld score is runnable locally for all three engines, with a Docker/KVM run path.
The runner ships a docker.yaml config and an osworld_os_smoke.jsonl of 8 real os-domain tasks drawn from OSWorld’s evaluation examples — terminal and file operations, settings, and one infeasible task.
The OSWorld Docker provider runs with NET_ADMIN but no tun device, which costs QEMU its port forwarding and leaves the VM unreachable. The runner injects /dev/net/tun and privileged for the OSWorld image — this requires /dev/net/tun on the host. With it the VM boots in about 24 seconds; without it, readiness times out.
NSR has been validated end to end on the real Dockerized OSWorld VM, scored by OSWorld’s own evaluator. There is also a cross-domain benchmark grader for computer-use-preview.
Last modified on August 29, 2026