Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+21 more

    vision bridge windows automation

    1

    Automate Windows desktop apps and browsers using structured UI trees and OCR through the vision-bridge MCP server.

    Free

    2 usesSecurity scanned
    vision bridge windows automation

    vision bridge windows automation

    Example session with this skill installed

    Open Notepad, type 'System Check Complete', and save it to the desktop as log.txt.

    • Read your context and instructions
    • Compiled the vision bridge windows

    Notepad is open, the text is typed and verified, and the file is saved as log.txt on your Desktop.

    What I did

    StepTool callResult
    1capture("Notepad")Window found via UIA, document field u0
    2act("u0", "set_value", value="System Check Complete")ok
    3act("u0", "read")content = "System Check Complete" (matches)
    4find("File", target="Notepad") → act("u3", "click")File menu opened
    5find("Save as", target="Notepad") → act("u9", "click")Save dialog opened
    6wait_for("File name", target="Save as", timeout_s=10)field u21 ready
    7act("u21", "set_value", value="%USERPROFILE%\Desktop\log.txt")ok
    8find("Save", target="Save as") → act("u27", "click")dialog closed
    9capture("Notepad")title now "log.txt - Notepad"

    Verification

    • Typed text was read back before saving, not assumed.
    • Ids were refreshed after every window change, so no stale clicks.
    • No overwrite prompt appeared; if log.txt had existed, I would have asked before replacing it.

    Next steps

    • Append a timestamp line on each run with act("u0", "type", ...).
    • Run the same flow on a schedule for a daily health-check log.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Automating Windows desktop applications often requires fragile pixel-based scripts or expensive RPA tools. This skill provides a direct bridge between your agent and the Windows UI Automation (UIA) tree, enabling reliable interaction with native apps, Electron windows, and browsers through structured text.

    What it does

    • Structured UI Navigation reads the screen as a hierarchy of text and elements rather than just pixels.
    • Native App Control drives Win32, WPF, UWP, and WinForms applications using stable element IDs.
    • OCR Fallback handles Electron apps and image-heavy interfaces when the accessibility tree is unavailable.
    • Browser Integration connects to existing Chrome sessions via CDP to reuse logged-in states and bypass manual auth.
    • State Verification confirms every click and type action by re-capturing the UI state to ensure reliability.

    How it works

    1. Target Identification locates specific windows by title substrings or focuses the active foreground window.
    2. Screen Capture retrieves the current element tree, including text content, element roles, and unique IDs.
    3. Action Execution performs clicks, double-clicks, typing, or value settings based on the discovered IDs.
    4. Result Verification re-scans the UI after each action to confirm the application state changed as expected.

    Frameworks & tools

    This skill utilizes the vision-bridge MCP server, leveraging Windows UI Automation (UIA), Tesseract OCR for visual recognition, and Chrome DevTools Protocol (CDP) for browser automation.

    Why this beats prompting it yourself

    Standard vision prompts struggle with precision on complex desktop layouts and cannot interact with the OS. This skill provides a systematic loop of discovery and verification, ensuring the agent doesn't "hallucinate" buttons or click the wrong coordinates.

    Use cases

    • Automating data entry into legacy ERP or accounting software without APIs.
    • Managing complex software installations or system configuration dialogs.
    • Extracting data from desktop-only programs into structured formats.
    • Running web workflows within the user's authenticated browser session.

    Known limitations

    Requires a Windows host with the vision-bridge MCP server installed. Native UIA backend is faster than OCR; OCR performance depends on installed language data.

    Windows 10/11 only. Requires the vision-bridge MCP server installed locally. No keyboard shortcuts or raw key presses: the agent works through clicks, typing and set_value on elements. OCR mode (used for Electron apps) is slower than UI Automation, and OCR element names may contain recognition errors. The agent drives the real desktop: the screen must be unlocked and the user should not use mouse or keyboard in the target window during automation. Browser stealth mode is not guaranteed against anti-bot systems.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before any payment.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    2 uses

    Times used to date

    Free forever

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 1 month ago

    • Free with an Agensi account

    Requires

    GitHub
    Vision Bridge MCP
    Windows 10
    Windows 11
    Python 3.11+
    uv (Python tool runner)
    Windows
    Python
    Git

    Permissions

    Browser

    Allowed Hosts

    github.com

    The skill itself runs no code and touches no files. All actions go through the vision-bridge MCP server, which reads window contents via Windows UI Automation / OCR and clicks or types in them on the user's desktop. Browser + Network: optional browser mode attaches to the user's own Chrome over CDP (localhost:9222) or launches a separate profile. The skill requires user confirmation before irreversible clicks (Delete, Pay, Send) and never enters credentials the user did not provide.

    Listed1 month ago

    What's inside

    Frequently Asked Questions

    Unlike vision-only models that rely on pixels, this skill accesses the Windows UI Automation (UIA) tree to see structural element IDs, ensuring your agent clicks the exact button intended even if the layout shifts.

    This skill requires a Windows host environment and the vision-bridge MCP server; it supports native applications (Win32, WPF, UWP), Electron apps, and browsers like Chrome via the DevTools Protocol.

    Yes, for windows that expose little to UI Automation (typical for Electron/Chromium apps) the server falls back to OCR and returns text regions as clickable elements. OCR finds text only: icon-only buttons without a label cannot be located this way.

    A SKILL.md that teaches your agent when and how to use vision-bridge, plus reference files: setup and client config, a full tool reference, and worked flows (forms, installers, Notepad, Electron apps, browser). The vision-bridge MCP server itself is free and open source on GitHub.

    The skill employs a "Result Verification" step, meaning it re-scans the UI state after every action to confirm the click or keystroke was successful before proceeding to the next task.

    New in Workflow Automation