---
name: screen-pilot
description: Use ScreenPilot for visual desktop GUI automation. Trigger when an agent needs to select a desktop window, observe labeled UI elements, reason over a screenshot, click/type/scroll/key-press by visual target id, or install/start the local ScreenPilot helper app.
license: MIT
compatibility: macOS with Accessibility permission; ScreenPilot local helper app
allowed-tools: run_command
metadata:
  author: DDM Team
  version: 2.1.0
---

# ScreenPilot

ScreenPilot automates desktop apps through an Observe-Think-Act loop:

1. Select a target window.
2. Observe a screenshot annotated with numeric UI element IDs.
3. Choose the intended element ID, optionally with vision reasoning.
4. Execute click, type, scroll, key press, or wait instructions.

The CLI auto-checks the local helper app at `http://127.0.0.1:8080/api/health`. If ScreenPilot is not running, it starts an installed app. If the app is missing on macOS, it downloads the latest `ScreenPilot-macos-arm64.zip` release from `zhouchangui/bgwndsvr`, installs it to `~/Applications/ScreenPilot.app`, and starts it.

## Commands

Run commands from this skill directory.

```bash
# List windows
node scripts/query.mjs --mode select

# Lock a window by title or process name
node scripts/query.mjs --mode select --query "WeChat"

# Capture an annotated screenshot for the locked window
node scripts/query.mjs --mode observe

# Ask a vision model which element to use
node scripts/query.mjs --mode think --query "Find the message input box"

# Click, type, scroll, key press, or run a JSON sequence
node scripts/query.mjs --mode act --target 15 --action click
node scripts/query.mjs --mode act --target 12 --action type_text --text "Hello"
node scripts/query.mjs --mode act --json '[{"type":"wait","duration":1000},{"type":"key_press","keys":["Enter"]}]'
```

`observe` writes the latest annotated image to `temp/latest_observe.jpg`. Read that image before choosing a `target_id`.

## Configuration

- `SCREENPILOT_BASE_URL`: defaults to `http://127.0.0.1:8080`
- `SCREENPILOT_TOKEN`: overrides the local token file
- `SCREENPILOT_RELEASE_REPO`: defaults to `zhouchangui/bgwndsvr`
- `SCREENPILOT_APP_PATH`: overrides the app path used for startup
- `OPENROUTER_API_KEY`: required only for `think` mode
- `OPENROUTER_MODEL`: optional model override for `think`

Without `SCREENPILOT_TOKEN`, the CLI reads macOS token candidates under `~/Library/Application Support/.../token.conf`, then falls back to the current dev token.

## Actions

| Action | Parameters | Description |
| :--- | :--- | :--- |
| `click` | `target_id` | Single left click |
| `double_click` | `target_id` | Double left click |
| `type_text` | `target_id`, `text` | Click target and type text |
| `scroll` | `target_id`, `direction`, `amount` | Scroll `up` or `down` |
| `key_press` | `keys` | Key list such as `["Enter"]` or `["Meta+v"]` |
| `wait` | `duration` | Wait milliseconds |

## Operating Notes

- On first macOS launch, the unsigned helper app may require manual approval in Privacy & Security.
- Window enumeration and input simulation require ScreenPilot to have Accessibility permission.
- Element IDs change after layout changes; run `observe` again after navigation, animation, or refresh.
- If an action fails, first re-run `observe`; if focus is wrong, click a neutral area or re-select the window.