Harmony browser agent

The browser surface you can see in Harmony is not automatically a browser an agent can control. On the app feature branch, the agent-driven path is experimental and off by default; this page describes implemented code, not a released job-application workflow or permission to visit a real account.

The closed model tool set includes browser_open, browser_observe, browser_click, browser_type, browser_select, browser_scroll, browser_wait and a gated browser_submit. Each call requires the verified launch/session binding and its own capability approval. The model cannot name its workspace, principal, session or target. browser_upload also has a packet-only executor and a Desktop owner-review/active-packet authority now; its worker catalogue is selected at initialization, not dynamically recomputed when a later packet is approved. A listed tool is not permission to upload: the live executor rechecks the exact approved packet and lease. Downloads are intercepted by the backend, not offered as a model-selected path or executable tool. All agent-browser modes remain off by default.

Two backends, neither an ambient browser tab

The ordinary browser surface has its own Preview status. That does not imply either agent backend is enabled. A guarded network path is not a guarantee against every DNS rebinding or browser exploit. The embedded connection-level DNS/pinning risk, full assembled app and supported-platform path still need qualification. The real-Electron full-page fixture shows below-fold pixels, but scrolling tiles are not atomic on dynamic or sticky pages; visual/text agreement remains unassessed.

Lease first, then observe

After an approved browser_open returns an actual surface, the daemon mints an ownership receipt bound to the verified launch, principal, workspace, task, session and surface (and a host-verified pane when embedded). The default lease lasts ten minutes; a foreign workspace, expired or revoked lease, transferred target or result from another surface cannot act or observe. A page-provided surface id or UI "active tab" is not ownership proof. Release, expiry and session close journal durable transitions and request teardown of the exact target. A closure is confirmed only after a backend acknowledgement; failures stay unconfirmed for bounded GC/repair. A durable receipt is not proof that an outside site accepted a form.

Browser lifecycle spans (navigate, settle, act, upload, download, handoff, deny) now append redacted, bounded records to the existing activity ledger, not a second tracking log. A scoped, cursor-based projection is available for an authorized UI to read; a live UI read path is not yet wired, and a lifecycle receipt is not a page's acceptance of an action.

browser_observe produces a bounded, typed envelope: DOM projection, hash, provenance, truncation state, visible light-DOM element refs (role, accessible name, opaque ref id) and a generation token. Page text and accessible names are untrusted-page data, not instructions or approval; the Pi worker separately frames browser tool text as UNTRUSTED_PAGE_DATA. The Harness context compiler now classifies page, download and clipboard fragments and quotes/hashes them only in the volatile tail. In a hostile-page request-manifest test, its stable instruction prefix and tool catalogue did not change. That compiler currently feeds an inspector projection, not Pi's provider request constructor: these checks do not prove provider-wire serialization or that a model will never follow hostile text. A page's URL or instructions cannot grant host authority or become the agent's general memory.

What a generation token does not do

A fresh observation replaces the previous generation. Its token expires (normally within 30 seconds and never beyond the lease) and belongs to the exact receipt, surface and daemon-verified scope. Every act or upload must present that generation and an element ref from it; no model-supplied CSS selector, JavaScript or arbitrary node is accepted. A fresh DOM read and paired screenshot recheck the ref before an effect. A changed page, missing screenshot, disagreement, stale token, forged ref, foreign lease or active human takeover refuses. Each effect invalidates its generation, so the next act needs another observation. The token alone is not permission: the separate capability approval and, for form values, the user packet still apply. Standalone CDP screenshot capture is composed. Real embedded CDP act/capture and full-page local-fixture pixels have now been exercised with sandbox on; independent OCR, atomic dynamic-page capture and packaged-app parity remain unproven. Iframes, shadow roots and incomplete challenge scans are barriers reported to the user, not quietly traversed.

Packet-bound form values and files

For browser_type and browser_select, the model supplies a valueId, not raw field text. The daemon resolves it from a host-verified, scope- and origin-bound user packet. An absent or mismatched fact still refuses with NALA_RUNTIME_TOOL_MISSING_FACT and performs no browser effect: ask the person, never infer an answer or consent from page text. With the separate application-ledger/browser gates explicitly on, the daemon resolves values only from the exact active, MAIN-approved JobPacket. Ordinary approved facts project as text; a human-answered fact projects only after the matching draft revision is approved and its ledger receipt recorded. A select without a typed option still asks. Selecting a backend alone supplies no arbitrary facts.

A refused field can create a restart-durable, typed FactRequest in the human inbox with an A2A notice whose delivery state is receipt-checked (queued is not delivered). TUI or Desktop answers create a new draft JobPacket revision, with who/when provenance. The original approved packet remains active until Desktop MAIN's existing owner-reviewed packet approval accepts the new digest; only then can the ledger move blocked(missing_fact) → prepared and a fresh observation permit retry. Standalone TUI can draft an answer but cannot silently approve it. An unanswered or ambiguously owned application stays blocked. Answers are not placed in page text, A2A bodies or generic task notes by this revision path. Generic browser_click cannot tick consent boxes or press submit, apply, pay or delete controls. Each act has a capability approval and an idempotency key; a lost acknowledgement is OUTCOME_UNKNOWN, never an instruction to click again.

browser_upload takes a visible file-input ref and a file ID from a content-addressed JobPacket. The Desktop packet review now shows applicant facts/provenance, PDF name/size/MIME/hash and forbidden-invent fields; Approve or Revoke crosses sender-validated MAIN IPC and an owner-bound daemon authority. The executor reopens the approved file once, checks PDF MIME, size and SHA-256, stages private bytes and rechecks approval/page freshness. The private staged PDF is retained for the exact lease until physical close so Chromium can read it for a later fixture POST. The model cannot choose a path or provide bytes. Release hold: the Pi worker's browser_upload catalogue is fixed at initialization, before a packet becomes active after browser_open; it can be advertised too early and is not dynamically added/revoked for later model turns. The executor still refuses an unapproved/revoked/foreign packet. Do not call this a qualified end-user upload flow merely because the review UI exists.

Downloads, when intercepted, go to per-session quarantine with bounded PDF, plain-text or CSV policy, content hashes and untrusted-download references. They are not executed, auto-imported into memory or exposed as trusted content. Standalone uses CDP quarantine before navigation; the host-owned embedded will-download path was exercised in a sandboxed local Electron fixture. Neither result is a live-site or packaged-build file-safety guarantee.

Human control and consequential actions

A positive heuristic signal for a login, credential, passkey, MFA or CAPTCHA challenge hands the already live leased surface to a human and blocks agent browser calls. A negative heuristic is not evidence that the page is safe. Login and verification belong to the person; the agent must never bypass them. Returning control requires a host-verified local-user decision. Without a runtime-wide pause callback, this pauses browser tools, not every model turn. The TUI's /browser status|handoff|release|approvals now reaches a handshake-bound operator RPC over the same lease and ApprovalService; inline y/n prompts show origin/action/redacted diff. /browser resume still requires independent host-verified local-user proof and refuses where no verifier is available; typing the command or holding a pipe token is not that proof. CLI harmony browser one-shot sockets can refuse if unbound.

browser_submit is in the closed tool catalogue, but submitArmed defaults false: the tool refuses before an effect. Even in an explicitly armed fixture, a host-verified user-intent packet, live DOM/pre-submit field diff, fresh ApprovalService decision, host deny policy and kill-switch gate any consequential action. Desktop now has a redacted, field-by-field pre-submit review surface for a pending gate request, but the production browser bootstrap still supplies no trusted form reader, host intent-packet resolver or armed submit gate. The visible modal cannot manufacture such a request; changing submitArmed alone does not make submit usable. A gate ticket or button click is not a submission receipt; no page text can authorize a real-world application. There was no real job submission in this slice.

The application ledger is not an autopilot

The durable application ledger has landed in the app branch under the separate jobs.applicationLedger.enabled: false gate (store schema v9, nala.jobs.{list,get,upsert,transition} and read-only harmony jobs list|show). It deduplicates canonical HTTPS job URL and employer-scoped requisition ID, keeps at most 20 slots per workflow, and persists state, evidence references, consumed idempotency keys and restart pointers. The ledger itself never submits a form. It is separate from general agent memory.

Its states distinguish discovered, prepared, awaiting_user, authorized, outcome_unknown, submitted_verified, blocked (MFA, CAPTCHA, missing fact or denylist) and withdrawn. authorized needs host-verified user approval; outcome_unknown is the durable attempt fence, never automatically retried. submitted_verified requires a host verifier to match owner-scoped confirmation observation evidence after that fence. The production daemon has not wired a host approval/confirmation artifact reader to the ledger, so an enabled ledger can prepare records and reconcile an approved missing-fact revision but cannot mint a verified submission through its normal RPC path. Even a matched page confirmation is evidence of what the page showed, not independent proof that an employer processed an application. Neither a tool return, a screenshot alone nor a confident model message is a verified outcome. This is not an available autonomous application service. No real application should be submitted without explicit user authorization.

A 20-encounter local-fixture dogfood run used a scripted model, the real daemon bootstrap and sandboxed Chromium, with an explicitly fixture-armed submit policy and test-only approvals/preview text: 18 distinct application rows, 13 fixture-verified submissions, four blocks (MFA, CAPTCHA, missing fact, denylist), one outcome_unknown lost acknowledgement never retried, and two duplicate encounters skipped. It killed/restarted the daemon child after encounter 10 and resumed from receipts without a second POST. These are fixture outcomes, not an enabled application service or proof of employer processing. A current missing fact can now be answered into a draft packet, but it still needs MAIN approval before a retry.

Harmony Cloud's gated identity handoff is distinct from this local ledger. Owner review, site-side incarnation import and HMAC-authenticated return are implemented behind default-OFF gates; a verified return appends a new local incarnation rather than overwriting history. Terminal /cloud handoff and /cloud return only direct the user to Desktop's review/Bring Home path — they do not transfer or verify identity themselves. The website Command Center has a read-only, owner-scoped live activity view and an imported-identity card when the separate Cloud gates permit it; neither is proof of a running or exactly resumed Cloud provider. See the continuity boundary for what can and cannot cross.

Related

• Persistent agents and memory
• Approvals and permissions
• Feature status
• Glossary