Desktop GUI automation via OpenInterpreter — mouse, keyboard, screenshot, and OCR control for native macOS/Linux applications. Three modes: Library (Claude reasons, OI executes), OS subprocess (full autonomous computer use), and Local agent (Ollama, offline). This skill should be used when interacting with desktop apps that have no CLI or API, automating GUI workflows, reading screen content via OCR, or controlling mouse/keyboard.
インストールする前に、エージェントに与えられる指示の中身を確認できます。
Desktop control for Claude Code via OpenInterpreter (62k stars, AGPL-3.0). Mouse, keyboard, screenshot, and OCR primitives backed by pyautogui + pytesseract.
| Mode | LLM | Script | Best For |
|---|---|---|---|
| Library | Claude Code (native) | Individual scripts below | Surgical GUI actions — Claude sees screenshots, reasons, dispatches actions |
| OS subprocess | Claude API (via OI) | oi_os_mode.py | Full autonomous computer use — delegate entire GUI tasks |
| Local agent | Ollama (offline) | oi_os_mode.py --local | Offline computer use, no API costs, privacy-sensitive tasks |
Use Library mode by default. Use OS subprocess to delegate self-contained GUI tasks. Use Local agent when offline or to avoid API costs.
Run once:
~/.claude/skills/open-interpreter/scripts/oi_install.sh
Installs open-interpreter[os] via uv, verifies pyautogui and tesseract, checks macOS permissions.
macOS permissions (one-time, manual):
Verify permissions:
python3 ~/.claude/skills/open-interpreter/scripts/oi_permission_check.py
The core pattern for GUI automation:
1. Take screenshot → oi_screenshot.py
2. Read PNG → Claude Read tool (native vision)
3. Decide action → Claude reasoning
4. Execute action → oi_click.py / oi_type.py
5. Verify → Take another screenshot
6. Loop until done
oi_screenshot.py — Capture screen, return file path with Retina metadata
python3 ~/.claude/skills/open-interpreter/scripts/oi_screenshot.py
python3 ~/.claude/skills/open-interpreter/scripts/oi_screenshot.py --region 0,0,800,600
python3 ~/.claude/skills/open-interpreter/scripts/oi_screenshot.py --active-window
Output (3 lines):
/tmp/oi_screenshot_1708789200.png
SCALE_FACTOR=2
SCREEN_SIZE=1512x982
oi_click.py — Mouse click by coordinates or OCR text
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --x 450 --y 300
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --x 900 --y 600 --image-coords
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --text "Submit"
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --x 450 --y 300 --double
python3 ~/.claude/skills/open-interpreter/scripts/oi_click.py --x 450 --y 300 --right
--image-coords: auto-divides by Retina scale factor (use when coordinates come from screenshot image pixels)--text: OCR-based — screenshots, finds text via pytesseract, clicks center of matchoi_type.py — Keyboard input
python3 ~/.claude/skills/open-interpreter/scripts/oi_type.py --text "hello world"
python3 ~/.claude/skills/open-interpreter/scripts/oi_type.py --key enter
python3 ~/.claude/skills/open-interpreter/scripts/oi_type.py --hotkey command space
python3 ~/.claude/skills/open-interpreter/scripts/oi_type.py --text "search" --method typewrite
--method typewrite: character-by-character (use when clipboard is needed for other purposes)--hotkey: AppleScript on macOS for reliable modifier key handlingoi_find_text.py — OCR screen reading
python3 ~/.claude/skills/open-interpreter/scripts/oi_find_text.py --text "Submit"
python3 ~/.claude/skills/open-interpreter/scripts/oi_find_text.py --text "Price" --screenshot /tmp/ss.png
Returns JSON array: [{"text": "Submit", "x": 450, "y": 300, "w": 80, "h": 24, "confidence": 95}]
oi_computer.py — Unified dispatch for all actions
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py screenshot
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py click --x 450 --y 300
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py type --text "hello"
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py find --text "Submit"
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py scroll --clicks 3
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py mouse-position
python3 ~/.claude/skills/open-interpreter/scripts/oi_computer.py screen-size
macOS Retina displays render at 2x (or 3x) scaling. Screenshot image pixels differ from screen coordinates:
| Metric | Example (14" MBP) |
|---|---|
| Image pixels (screenshot) | 3024 x 1964 |
| Screen coordinates (pyautogui) | 1512 x 982 |
| Scale factor | 2x |
When estimating click targets from a screenshot image, use --image-coords on oi_click.py to auto-divide by the scale factor. The oi_screenshot.py output includes SCALE_FACTOR metadata.
For self-contained GUI tasks, delegate to OI's full agent loop:
python3 ~/.claude/skills/open-interpreter/scripts/oi_os_mode.py "Open Calculator and compute 2+2"
python3 ~/.claude/skills/open-interpreter/scripts/oi_os_mode.py --provider anthropic "Change the desktop wallpaper"
OI runs its own screenshot → analyze → act loop using the Claude API. Requires ANTHROPIC_API_KEY.
Run OI with a local vision model via Ollama:
python3 ~/.claude/skills/open-interpreter/scripts/oi_os_mode.py --local "What apps are open?"
Prerequisites:
ollama serveollama pull llama3.2-visionLimitation: Local models use OI's classic code-execution mode, not the screenshot-driven OS Mode (which requires Claude 3.5 Sonnet). Local mode generates and executes code to accomplish GUI tasks rather than using pixel-level screenshot analysis.
python3 scripts/oi_type.py --hotkey command space
sleep 0.5
python3 scripts/oi_type.py --text "Calculator"
sleep 0.3
python3 scripts/oi_type.py --key enter
python3 scripts/oi_screenshot.py > /tmp/ss_meta.txt
python3 scripts/oi_find_text.py --text "Total" --screenshot "$(head -1 /tmp/ss_meta.txt)"
python3 scripts/oi_click.py --text "Save"
python3 scripts/oi_click.py --text "Email"
python3 scripts/oi_type.py --text "user@example.com"
python3 scripts/oi_type.py --key tab
python3 scripts/oi_type.py --text "password123"
pyautogui.FailSafeException (enabled by default)[oi] click at (450, 300) button=left| Symptom | Fix |
|---|---|
oi_screenshot.py returns black image | Grant Screen Recording permission to terminal app |
oi_click.py / oi_type.py no effect | Grant Accessibility permission to terminal app |
| OCR finds no text | Verify tesseract: which tesseract && tesseract --version |
| Retina coordinates off by 2x | Use --image-coords flag on oi_click.py |
oi_find_text.py low confidence | Try larger text, ensure screen is not obstructed |
| OS Mode hangs | Verify ANTHROPIC_API_KEY is set, check OI stderr output |
| Local mode fails | Verify Ollama running (ollama list) and model pulled |
| File | Contents |
|---|---|
references/computer-api.md | OI Computer API reference — mouse, keyboard, display, clipboard |
references/os-mode.md | OS Mode usage, provider configuration, agent loop architecture |
references/safety-and-permissions.md | macOS permissions guide, safety model, failsafe configuration |
まだレビューはありません。使ってみた感想をお寄せください。
概要と使いどころ
Search academic papers, build literature reviews, and synthesize research findings — combines Exa MCP (research_paper category, arxiv filtering) with arxiv-mcp-server for paper discovery, download, and deep analysis. Triggers on academic paper, literature review, research synthesis, arxiv, find papers, scholarly search.
日本語の概要は準備中です。原文の説明を表示しています。
Automates browser interactions for web testing, form filling, screenshots, and data extraction. Use when the user needs to navigate websites, interact with web pages, fill forms, take screenshots, test web applications, or extract information from web pages.
日本語の概要は準備中です。原文の説明を表示しています。
This skill should be used when creating, auditing, or maintaining AGENTS.md files and Codex CLI configuration for any project—including initializing AGENTS.md for cross-agent compatibility (Codex, Cursor, Copilot, Devin, Jules, Amp, Gemini CLI), generating config.toml or .rules files, scaffolding .agents/skills/, converting CLAUDE.md to AGENTS.md, or auditing existing agent configs for bloat and staleness. Complementary to codex-orchestrator (which executes subagents; this skill creates the config files they consume).
日本語の概要は準備中です。原文の説明を表示しています。
Academic research skill for Biblical Hebrew, Semitic linguistics, cuneiform studies, and comparative Ancient Near Eastern research. Provides Sefaria API for Hebrew Bible, CDLI/ORACC for cuneiform databases, and web discovery via Omnisearch, Exa, Firecrawl, and Obscura for finding scholarly sources across JSTOR, Perseus, Persée, Google Scholar, and academia.edu. Triggers on Hebrew quotes, cuneiform, Sefaria, ANE research, Minoan, search for scholarship, find papers, literature review, scholarly search, academic search, Genesis/Tehom, Ugaritic, Talmudic sources, extract from PDF, OCR academic.
日本語の概要は準備中です。原文の説明を表示しています。
Build comprehensive ARCHITECTURE.md files following matklad's canonical guidelines — bird's-eye views, ASCII/Mermaid diagrams, codemaps, invariants, and layer boundaries. Triggers on document the architecture, create ARCHITECTURE.md, map this codebase, architectural overview.
日本語の概要は準備中です。原文の説明を表示しています。
Generate physically-based atmospheric scattering shaders — sky domes, planetary atmospheres, LUT-optimized pipelines, depth-aware post-processing. Four modes — sky-dome, atmosphere-post, planet, lut. Triggers on atmospheric scattering, sky shader, sunset rendering, planet atmosphere, Rayleigh scattering, Mie scattering, volumetric sky, sky dome, atmosphere post-processing, aerial perspective, transmittance LUT, sky rendering, realistic sky, planetary rendering.
日本語の概要は準備中です。原文の説明を表示しています。