name: screen-pilot description: Use ScreenPilot for visual desktop GUI automation. Trigger when an agent needs to select a desktop window, observe labeled UI elements, reason over a screenshot, click/type/scroll/key-press by visual target id, or install/start the local ScreenPilot helper app. license: MIT compatibility: macOS with Accessibility permission; ScreenPilot local helper app allowed-tools: run_command metadata: author: DDM Team version: 2.1.0

ScreenPilot

ScreenPilot automates desktop apps through an Observe-Think-Act loop:

1. Select a target window.
2. Observe a screenshot annotated with numeric UI element IDs.
3. Choose the intended element ID, optionally with vision reasoning.
4. Execute click, type, scroll, key press, or wait instructions.

The CLI auto-checks the local helper app at http://127.0.0.1:8080/api/health. If ScreenPilot is not running, it starts an installed app. If the app is missing on macOS, it downloads the latest ScreenPilot-macos-arm64.zip release from zhouchangui/bgwndsvr, installs it to ~/Applications/ScreenPilot.app, and starts it.

Commands

Run commands from this skill directory.

# List windows
node scripts/query.mjs --mode select

# Lock a window by title or process name
node scripts/query.mjs --mode select --query "WeChat"

# Capture an annotated screenshot for the locked window
node scripts/query.mjs --mode observe

# Ask a vision model which element to use
node scripts/query.mjs --mode think --query "Find the message input box"

# Click, type, scroll, key press, or run a JSON sequence
node scripts/query.mjs --mode act --target 15 --action click
node scripts/query.mjs --mode act --target 12 --action type_text --text "Hello"
node scripts/query.mjs --mode act --json '[{"type":"wait","duration":1000},{"type":"key_press","keys":["Enter"]}]'

observe writes the latest annotated image to temp/latest_observe.jpg. Read that image before choosing a target_id.

Configuration

• SCREENPILOT_BASE_URL: defaults to http://127.0.0.1:8080
• SCREENPILOT_TOKEN: overrides the local token file
• SCREENPILOT_RELEASE_REPO: defaults to zhouchangui/bgwndsvr
• SCREENPILOT_APP_PATH: overrides the app path used for startup
• OPENROUTER_API_KEY: required only for think mode
• OPENROUTER_MODEL: optional model override for think

Without SCREENPILOT_TOKEN, the CLI reads macOS token candidates under ~/Library/Application Support/.../token.conf, then falls back to the current dev token.

Actions

| Action | Parameters | Description | | :--- | :--- | :--- | | click | target_id | Single left click | | double_click | target_id | Double left click | | type_text | target_id, text | Click target and type text | | scroll | target_id, direction, amount | Scroll up or down | | key_press | keys | Key list such as ["Enter"] or ["Meta+v"] | | wait | duration | Wait milliseconds |

Operating Notes

• On first macOS launch, the unsigned helper app may require manual approval in Privacy & Security.
• Window enumeration and input simulation require ScreenPilot to have Accessibility permission.
• Element IDs change after layout changes; run observe again after navigation, animation, or refresh.
• If an action fails, first re-run observe; if focus is wrong, click a neutral area or re-select the window.