# Harmony browser agent Source: https://docs.ideharmony.com/concepts/harmony-browser-agent The **browser surface** you can see in Harmony is not automatically a browser an agent can control. On the app feature branch, the agent-driven path is **experimental and off by default**; this page describes implemented code, not a released job-application workflow or permission to visit a real account. The closed model tool set includes `browser_open`, `browser_observe`, `browser_click`, `browser_type`, `browser_select`, `browser_scroll`, `browser_wait` and a **gated** `browser_submit`. Each call requires the verified launch/session binding and its own capability approval. The model cannot name its workspace, principal, session or target. `browser_upload` also has a packet-only executor and a Desktop owner-review/active-packet authority now; its worker catalogue is selected at initialization, **not** dynamically recomputed when a later packet is approved. A listed tool is not permission to upload: the live executor rechecks the exact approved packet and lease. Downloads are intercepted by the backend, not offered as a model-selected path or executable tool. All agent-browser modes remain **off by default**. ## Two backends, neither an ambient browser tab | Backend | How it is selected | Ownership and current limit | |---|---|---| | **Standalone** | Explicit `harmonyBrowser: 'standalone'`; absent or unknown mode resolves to `off` | Daemon-managed, sandboxed Chromium with a guarded network path and owned target, not the user's default profile. The default handle now composes popup denial, CDP screenshot capture, per-session download quarantine, physical lease teardown and durable receipts; a failed closure remains unconfirmed for repair. | | **Embedded** | Explicit `harmonyBrowser: 'embedded'` **and** `browser.embeddedBackend.enabled: true` **and** a qualified authenticated host invoker | Desktop provisions an isolated blank guest, arms request policy/CDP before navigation, checks the verified pane and routes host-owned operations. The second latch defaults `false`. Real Electron 44 offscreen, sandbox-on local-fixture checks now cover act, approved PDF upload, download quarantine, crash notice across the authenticated host pipe and tiled full-page CDP capture (38/38 in the specialist lane). This is **not** a packaged/onscreen release proof or independent screenshot-text agreement. | The ordinary [browser surface](/docs/reference/feature-status) has its own Preview status. That does **not** imply either agent backend is enabled. A guarded network path is not a guarantee against every DNS rebinding or browser exploit. The embedded connection-level DNS/pinning risk, full assembled app and supported-platform path still need qualification. The real-Electron full-page fixture shows below-fold pixels, but scrolling tiles are not atomic on dynamic or sticky pages; visual/text agreement remains `unassessed`. ## Lease first, then observe After an approved `browser_open` returns an actual surface, the daemon mints an ownership receipt bound to the verified launch, principal, workspace, task, session and surface (and a host-verified pane when embedded). The default lease lasts ten minutes; a foreign workspace, expired or revoked lease, transferred target or result from another surface cannot act or observe. A page-provided surface id or UI "active tab" is not ownership proof. Release, expiry and session close journal durable transitions and request teardown of the exact target. A closure is **confirmed only after a backend acknowledgement**; failures stay unconfirmed for bounded GC/repair. A durable receipt is not proof that an outside site accepted a form. Browser lifecycle spans (`navigate`, `settle`, `act`, `upload`, `download`, `handoff`, `deny`) now append redacted, bounded records to the **existing** activity ledger, not a second tracking log. A scoped, cursor-based projection is available for an authorized UI to read; a live UI read path is not yet wired, and a lifecycle receipt is not a page's acceptance of an action. `browser_observe` produces a bounded, typed envelope: DOM projection, hash, provenance, truncation state, visible light-DOM element refs (role, accessible name, opaque ref id) and a generation token. Page text and accessible names are **`untrusted-page` data, not instructions or approval**; the Pi worker separately frames browser tool text as `UNTRUSTED_PAGE_DATA`. The Harness context compiler now classifies page, download and clipboard fragments and quotes/hashes them only in the volatile tail. In a hostile-page request-manifest test, its stable instruction prefix and tool catalogue did not change. **That compiler currently feeds an inspector projection, not Pi's provider request constructor**: these checks do not prove provider-wire serialization or that a model will never follow hostile text. A page's URL or instructions cannot grant host authority or become the agent's general memory. ### What a generation token does not do A fresh observation replaces the previous generation. Its token expires (normally within 30 seconds and never beyond the lease) and belongs to the exact receipt, surface and daemon-verified scope. Every act or upload must present **that generation and an element ref from it**; no model-supplied CSS selector, JavaScript or arbitrary node is accepted. A fresh DOM read and paired screenshot recheck the ref before an effect. A changed page, missing screenshot, disagreement, stale token, forged ref, foreign lease or active human takeover refuses. Each effect invalidates its generation, so the next act needs another observation. The token alone is **not permission**: the separate capability approval and, for form values, the user packet still apply. Standalone CDP screenshot capture is composed. Real embedded CDP act/capture and full-page local-fixture pixels have now been exercised with sandbox on; independent OCR, atomic dynamic-page capture and packaged-app parity remain unproven. Iframes, shadow roots and incomplete challenge scans are barriers reported to the user, not quietly traversed. ### Packet-bound form values and files For `browser_type` and `browser_select`, the model supplies a `valueId`, not raw field text. The daemon resolves it from a **host-verified, scope- and origin-bound user packet**. An absent or mismatched fact still refuses with `NALA_RUNTIME_TOOL_MISSING_FACT` and performs **no browser effect**: ask the person, never infer an answer or consent from page text. With the separate application-ledger/browser gates explicitly on, the daemon resolves values only from the exact active, MAIN-approved JobPacket. Ordinary approved facts project as text; a human-answered fact projects only after the matching draft revision is approved and its ledger receipt recorded. A select without a typed option still asks. Selecting a backend alone supplies no arbitrary facts. A refused field can create a restart-durable, typed **FactRequest** in the human inbox with an A2A notice whose delivery state is receipt-checked (`queued` is not `delivered`). TUI or Desktop answers create a **new draft JobPacket revision**, with who/when provenance. The original approved packet remains active until Desktop MAIN's existing owner-reviewed packet approval accepts the new digest; only then can the ledger move `blocked(missing_fact)` → `prepared` and a fresh observation permit retry. Standalone TUI can draft an answer but cannot silently approve it. An unanswered or ambiguously owned application stays blocked. Answers are not placed in page text, A2A bodies or generic task notes by this revision path. Generic `browser_click` cannot tick consent boxes or press submit, apply, pay or delete controls. Each act has a capability approval and an idempotency key; a lost acknowledgement is `OUTCOME_UNKNOWN`, never an instruction to click again. `browser_upload` takes a visible file-input ref and a file ID from a content-addressed JobPacket. The Desktop packet review now shows applicant facts/provenance, PDF name/size/MIME/hash and forbidden-invent fields; Approve or Revoke crosses sender-validated MAIN IPC and an owner-bound daemon authority. The executor reopens the approved file once, checks PDF MIME, size and SHA-256, stages private bytes and rechecks approval/page freshness. The private staged PDF is retained for the exact lease until physical close so Chromium can read it for a later fixture POST. The model cannot choose a path or provide bytes. **Release hold:** the Pi worker's `browser_upload` catalogue is fixed at initialization, before a packet becomes active after `browser_open`; it can be advertised too early and is not dynamically added/revoked for later model turns. The executor still refuses an unapproved/revoked/foreign packet. Do not call this a qualified end-user upload flow merely because the review UI exists. Downloads, when intercepted, go to per-session quarantine with bounded PDF, plain-text or CSV policy, content hashes and `untrusted-download` references. They are not executed, auto-imported into memory or exposed as trusted content. Standalone uses CDP quarantine before navigation; the host-owned embedded `will-download` path was exercised in a sandboxed local Electron fixture. Neither result is a live-site or packaged-build file-safety guarantee. ## Human control and consequential actions A positive heuristic signal for a login, credential, passkey, MFA or CAPTCHA challenge hands the **already live leased surface** to a human and blocks agent browser calls. A negative heuristic is **not evidence** that the page is safe. Login and verification belong to the person; the agent must never bypass them. Returning control requires a host-verified local-user decision. Without a runtime-wide pause callback, this pauses browser tools, not every model turn. The TUI's `/browser status|handoff|release|approvals` now reaches a handshake-bound operator RPC over the **same** lease and ApprovalService; inline y/n prompts show origin/action/redacted diff. `/browser resume` still requires independent host-verified local-user proof and refuses where no verifier is available; typing the command or holding a pipe token is not that proof. CLI `harmony browser` one-shot sockets can refuse if unbound. `browser_submit` is in the closed tool catalogue, but `submitArmed` defaults `false`: the tool **refuses before an effect**. Even in an explicitly armed fixture, a host-verified user-intent packet, live DOM/pre-submit field diff, fresh ApprovalService decision, host deny policy and kill-switch gate any consequential action. Desktop now has a redacted, field-by-field pre-submit review surface for a pending gate request, but the production browser bootstrap still supplies **no trusted form reader, host intent-packet resolver or armed submit gate**. The visible modal cannot manufacture such a request; changing `submitArmed` alone does not make submit usable. A gate ticket or button click is **not a submission receipt**; no page text can authorize a real-world application. There was **no real job submission** in this slice. ## The application ledger is not an autopilot The durable **application ledger** has landed in the app branch under the separate `jobs.applicationLedger.enabled: false` gate (store schema v9, `nala.jobs.{list,get,upsert,transition}` and read-only `harmony jobs list|show`). It deduplicates canonical HTTPS job URL and employer-scoped requisition ID, keeps at most 20 slots per workflow, and persists state, evidence references, consumed idempotency keys and restart pointers. The ledger itself **never submits a form**. It is separate from general agent memory. Its states distinguish `discovered`, `prepared`, `awaiting_user`, `authorized`, `outcome_unknown`, `submitted_verified`, `blocked` (MFA, CAPTCHA, missing fact or denylist) and `withdrawn`. `authorized` needs host-verified user approval; `outcome_unknown` is the durable attempt fence, **never automatically retried**. `submitted_verified` requires a host verifier to match owner-scoped confirmation observation evidence after that fence. The production daemon has **not wired a host approval/confirmation artifact reader to the ledger**, so an enabled ledger can prepare records and reconcile an approved missing-fact revision but cannot mint a verified submission through its normal RPC path. Even a matched page confirmation is evidence of what the page showed, not independent proof that an employer processed an application. Neither a tool return, a screenshot alone nor a confident model message is a verified outcome. This is **not an available autonomous application service**. No real application should be submitted without explicit user authorization. A **20-encounter local-fixture dogfood run** used a scripted model, the real daemon bootstrap and sandboxed Chromium, with an explicitly fixture-armed submit policy and test-only approvals/preview text: 18 distinct application rows, 13 fixture-verified submissions, four blocks (MFA, CAPTCHA, missing fact, denylist), one `outcome_unknown` lost acknowledgement never retried, and two duplicate encounters skipped. It killed/restarted the daemon child after encounter 10 and resumed from receipts without a second POST. These are **fixture outcomes**, not an enabled application service or proof of employer processing. A current missing fact can now be answered into a draft packet, but it still needs MAIN approval before a retry. Harmony Pro's gated identity handoff is distinct from this local ledger. Owner review, site-side incarnation import and HMAC-authenticated return are implemented behind default-OFF gates; a verified return appends a new local incarnation rather than overwriting history. Terminal `/cloud handoff` and `/cloud return` only direct the user to Desktop's review/Bring Home path — they do not transfer or verify identity themselves. The website Command Center has a read-only, owner-scoped live activity view and an imported-identity card when the separate Cloud gates permit it; neither is proof of a running or exactly resumed Cloud provider. See the [continuity boundary](/docs/concepts/persistent-agents-and-memory) for what can and cannot cross. ## Related - [Persistent agents and memory](/docs/concepts/persistent-agents-and-memory) - [Approvals and permissions](/docs/concepts/permissions-and-approvals) - [Feature status](/docs/reference/feature-status) - [Glossary](/docs/reference/glossary)