GitHub Copilot app Automation

Use this skill when the task is about driving, testing, mapping, or experimenting with the GitHub Copilot app. Prefer app/session APIs first, then app-native canvases/automations, and use GUI Accessibility only when the visible desktop UI itself must be exercised.

Where to Run This

This skill's power depends heavily on where the agent runs:

Context	Session/app APIs	Capture reliability	Best for
Inside the GitHub Copilot app agent (recommended)	Available for the host instance. They do not select a separately launched persona.	High when the agent launches, drives, and captures the persona by process ID	End-to-end screenshot capture with a dedicated persona
GitHub Copilot CLI (terminal)	Not available	Low/manual — a human must navigate the app, and its window must be on the same macOS Space as the terminal	Read-only mapping (map-app.sh), the dynamic screenshots.sh list/embed flow, and one-off captures of an already-visible window

For course screenshots, drive this from inside the app, but do not use the host instance as the screenshot target. Launch the dedicated persona below, then use process-scoped Accessibility and capture. The shell scripts here run in both contexts because the app agent is built on Copilot CLI.

Regardless of context: pixels still come from screencapture (there is no pure render/export API yet, so the window must be visible), Screen Recording permission is required, captures must come from a sanitized training account on the training fork, and advanced features (Agent Merge, cloud automations) stay policy/billing-gated.

For each screenshot, launch a new app process from the persistent, signed-in demo persona instead of creating a new persona or reusing an open process:

SK=.github/skills/github-copilot-app-automation/sample_codes/macos-accessibility
PERSONA="demo"
# The local clone of the training fork, not the upstream course repository.
REPO="$HOME/copilot-app-for-beginners"
COPILOT_PID="$(bash "$SK/prepare-persona.sh" "$PERSONA" "$REPO" 60)"

This keeps the existing login and persona data, starts a separate app process, sets only that window to a standard 1920x1080-point frame, resets the display zoom and zooms in three times (the zoom is measured and confirmed), enables and verifies Streamer Mode, verifies that the account is signed in, verifies the exact local repository, dismisses transient banners, and returns its process ID. If the repository is missing, the pre-step opens the exact folder through the native folder picker and verifies it before capture. A setup failure keeps the process for inspection and does not return a successful process ID. Do not select a target by app name when another Copilot instance is open. Do not use full screen: it can draw the menu bar and title bar over the app content.

To keep working on other displays during capture, set COPILOT_CAPTURE_DISPLAY=builtin for both prepare-persona.sh and capture-window.sh. The window then opens on the built-in laptop display and zooms in twice instead of three times, because the laptop frame is smaller. After setup, focus goes back to the app that you used before. The pointer is not moved, and the capture does not activate the persona. Do not type during the first launch seconds, and keep the laptop display on its current Space.

Before preparing the visible state, extract the Markdown around the image reference:

python3 "$SK/screenshot-context.py" <chapter>/README.md <image-name>

Use the section heading, instructions before the image, and instructions after the image to define the exact screen, open menu, selected item, and callouts. Do not capture a generic approximation. Confirm that current Accessibility labels agree with the text. If the app label changed, update the directly related course text before capture.

Add numbered red callouts when one screenshot shows two or more controls that the reader must select in order. The nearby instructions must identify each number, for example, "select + (callout 1), then select Add GitHub repository (callout 2)." Do not add callouts to a single-action screenshot, an output example, or an evidence screenshot. To point at one item without a sequence, such as a menu item to select, use a red highlight box (--box) instead. To point at one small status or control, such as a status pill, use a red arrow (--arrow TAIL_X:TAIL_Y:HEAD_X:HEAD_Y). Recreate the callouts, boxes, or arrows of an existing image when you retake it, at the new positions of the same controls.

Keep the process ID and use it for all UI operations and capture. Do not use navigate_to or other app/session APIs to prepare this window because those APIs can route to the instance that hosts the agent. Drive the persona through Accessibility selectors scoped to its process ID.

An isolated HOME does not isolate credentials stored in the macOS Keychain. The persona can still show the current GitHub account name and avatar. Process-specific capture automatically derives the visible profile name from the accessibility tree and replaces that text with Copilot Dev. If an account handle or repository owner matches the normalized profile name, it replaces that text with copilotdev, including inside paths and branch names. The macOS account name and machine name, which terminal prompts and paths can show, become copilotdev and copilot-dev-mac. It preserves the avatar and unrelated people and organizations. On a Settings screen, it also removes the displayed app version. Never hard-code a person's name or an app version. Never treat the persona name alone as proof that the capture is sanitized. Zoom in on every replaced span: a partial replacement can leave letters of the private name next to the new text.

Non-Interactive / Hidden-Session Rule

If the user asks for a fresh, separate, or hidden Copilot App instance/session:

• Stay on app/session APIs: list_projects, create_session, send_session_message, get_session, and filesystem verification.
• Do not call navigate_to, foreground open -a, menu-bar actions, folder pickers, or Accessibility clicks unless the user explicitly asks to see the App UI.
• Do not make the user select a folder or project. If a project cannot be registered through available non-interactive surfaces, report that limitation instead of opening a visible picker.
• For local folders, first make the target folder a distinct repo root when needed (git init in the target folder) so the App can distinguish it from a parent repository.
• If a hidden registration attempt is necessary, prefer open -g -j -a "GitHub Copilot" <folder> and immediately verify with list_projects; if it does not register the project, stop and report the gap.
• A screenshot of a hidden/minimized/background-only session is not possible with macOS screencapture; screenshot capture requires a visible window or a future App-provided render/export API. Do not switch the user's current App window just to satisfy a hidden screenshot request.

Operating Model

Work in layers, from safest to most brittle:

1. Session orchestration tools: list projects/sessions, create isolated sessions, message sessions, inspect sessions, navigate to sessions.
2. App-native surfaces: canvases, automations, project settings, MCP, skills, issue/PR sessions.
3. Copilot CLI remote control: useful for CLI sessions via GitHub.com/GitHub Mobile, not for native app GUI control.
4. macOS Accessibility/UI scripting: use AppleScript/System Events for named controls such as Message, Select model, and Open changes.
5. Screenshot capture: use Accessibility window bounds plus screencapture to save PNGs, then convert to WebP for web delivery.
6. Low-level GUI clicking: avoid unless the user explicitly approves and no named accessibility control exists.

Safety Rules

• Create a separate session for experiments unless the user explicitly wants the current session modified.
• For hidden/separate-session tasks, never navigate the user's visible App window unless explicitly requested.
• Never submit a prompt, create a PR, merge, delete, approve permissions, or enable Agent Merge without explicit user intent.
• For GUI probes, start read-only: enumerate windows, controls, roles, names, and actions.
• For composer tests, type a clearly marked draft and clear it without pressing Return/Enter.
• Avoid pixel-coordinate clicks. Target controls by accessibility role/name/action.
• Do not persist secrets or private UI data in repo files. Store scratch probes in /tmp or the session artifact directory.
• Screenshots may contain private repo names, prompts, diffs, terminal output, or tokens. Capture only the requested app state and save course-ready images under an intentional assets folder.
• Expect transient app/window states. Re-discover the window and AXWebArea before each UI operation.
• If Accessibility is denied, stop and ask the user to grant access to the automation caller. Do not try to bypass macOS privacy controls.
• If screencapture fails or returns a privacy prompt, the automation caller also needs macOS Screen Recording permission.

Known Local App Facts

Observed on this machine:

• App path: /Applications/GitHub Copilot.app
• Bundle id: com.github.githubapp
• Bundle version: read dynamically with map-app.sh; do not rely on a stored version
• Executable: /Applications/GitHub Copilot.app/Contents/MacOS/github
• Binary type: native macOS Mach-O arm64
• Frameworks observed include AppKit and WebKit.
• The desktop app is not obviously Electron/Chromium; do not assume Playwright/CDP control is available.

Known Accessibility Map

After macOS Accessibility permission was enabled, System Events could address the app as process GitHub Copilot.

Useful exposed controls (app 1.1.26; see app-ui-map.md for the full map):

Control	Role	Use
GitHub Copilot	AXWindow	Main app window
GitHub Copilot	AXWebArea	Main webview-backed app content
Sidebar	AXGroup	Sidebar region
Toggle sidebar, Command + B	AXCheckBox	Collapse/expand sidebar
New, Pull requests, Issues, Automations, Customize	AXButton	Sidebar views (Quick links)
More	AXPopUpButton	Sidebar menu with Edit sidebar...
New project or session	AXPopUpButton	Projects + menu
New session in <project>	AXButton	Project-specific new session entry (visible on hover or focus)
Message	AXTextArea	Prompt composer
Project: <project>	AXComboBox	Composer project picker
Workspace: <choice>, branch: <branch>	AXPopUpButton	Where to work menu
Toggle panel, Command + Option + B	AXCheckBox	Review panel
Add tab, Command + T	AXPopUpButton	Review panel tab menu

See macos-accessibility.md for scripts and caveats.

Preferred Workflows

Map the App Safely

1. Confirm the app is running and discover bundle/process state.
2. Enumerate projects and sessions with app/session tools.
3. Run a read-only Accessibility window/menu probe.
4. Enumerate AXWebArea children.
5. Save findings to session artifacts, not the repo, unless the user asked for durable docs.

Use:

• probe-readonly.applescript
• automation-map.md

Test Composer Control Without Submitting

1. Create a separate idle session.
2. Navigate to that session.
3. Focus AXTextArea name=Message.
4. Set a draft value directly.
5. Read the value back.
6. Clear it with Command+A and Delete.
7. Verify the returned cleared-value is empty.

Use type-and-clear-draft.applescript.

Capture Screenshots for Course Assets

Only use this workflow when the user has agreed the App can be visible or when the target state is already visible in a dedicated App window. Capture from a sanitized training account on the training fork — never the user's real private projects.

1. Run screenshot-context.py for the referenced image and turn the nearby course text into a concrete capture checklist. If you retake an existing image, view the old image first and list its callouts, boxes, and crop.
2. Launch a new process from the persistent demo persona with prepare-persona.sh. The pre-step sets a standard 1920x1080-point window, resets zoom, zooms in three times, and enables and verifies Streamer Mode. It fails closed if Streamer Mode or the zoom cannot be verified. Pass the exact local training repository and keep the returned process ID. Turn off Siri before you capture: its waveform orb can appear next to a focused text field, and the capture script fails when it sees the orb.
3. Map that process with COPILOT_PID="$COPILOT_PID" map-app.sh so steps match the current app build.
4. Navigate that process to the exact state with Accessibility controls and let spinners settle. Use prepare-copilot-state.swift for states that a named action cannot create: focus shows a hover-only control (such as the + on the Chats row), highlight-menu-item selects a menu item with the keyboard (web menus ignore synthetic pointer moves), set-value fills a field without pressing Return, and park moves the pointer and focus away so no hover effects or focus rings remain.
5. Find the final coordinates of each control with locate-copilot-element.swift, then capture the window owned by that process ID and convert it to WebP. Pass callouts in instruction order, highlight boxes, and an optional crop, all in final 1920x1080 coordinates:

   bash sample_codes/macos-accessibility/capture-window.sh \
     <chapter>/assets <base-name> 40 "$COPILOT_PID"
   
   bash sample_codes/macos-accessibility/capture-window.sh \
     00-setup/assets app-add-project 40 "$COPILOT_PID" \
     --callout 1:372:338 --callout 2:585:437
   
   bash sample_codes/macos-accessibility/capture-window.sh \
     06-canvases/assets app-open-repo-issues-canvas 40 "$COPILOT_PID" \
     --box 1287:266:1551:309 --crop 352:0:1920:700
   

6. For callout images, confirm that every badge is next to its control, does not cover a label or icon, and matches the instruction order. The standard style is a 26px-radius #ff594b circle with a white number. Keep each badge at least its radius plus 2 px from the image edge.
7. Keep PNG as the source artifact; use WebP in web/course pages.
8. Confirm that full-window files are exactly 1920x1080. A crop keeps the size of its crop rectangle.
9. Confirm that the image has a 2px #cccccc border inside its edges, and that no window corner shows black pixels. The app window has transparent rounded corners. Put an image on white before any RGB conversion or resize, or the corners turn black. capture-window.sh stops if it finds dark corner pixels.
10. Confirm that the three macOS window buttons do not show in the top-left corner and that the sidebar icon is in their place. capture-window.sh does this with hide-window-controls.py.
11. Confirm that the output keeps the avatar, shows Copilot Dev instead of a person's profile name, and uses copilotdev for matching personal repository owners, paths, and branch names.
12. For Settings screenshots, confirm that no app version is visible.
13. Review the screenshot for other private data before committing or publishing.
14. Restore any persona state that you changed for the capture (hidden projects, installed plugins or MCP servers, test automations or sessions), then close the exact process with cleanup-persona.sh only after the capture is verified. Keep the signed-in persona for the next screenshot.

To work through the course's pending shots, discover them dynamically and process one at a time (never hardcode the list): screenshots.sh list -> screenshots.sh next -> capture -> screenshots.sh embed. See missing-screenshots.md.

Use:

• screenshot-capture.md — full method, permissions, and the macOS Spaces constraint
• missing-screenshots.md — the generic, one-at-a-time capture workflow (shots are discovered dynamically, never hardcoded)
• screenshots.sh — dynamically list / next / embed the chapters' <!-- app-screenshot: ... --> placeholders
• screenshot-context.py — extracts the course section that defines a screenshot's required state
• prepare-persona.sh — launches a new process from a signed-in persona, verifies the exact local repository, and returns the process ID
• ensure-streamer-mode.swift — enables and verifies Streamer Mode before screenshot preparation
• cleanup-persona.sh — stops one exact screenshot process and preserves the signed-in persona
• launch-persona.sh — low-level launcher used by prepare-persona.sh
• control-copilot-window.swift — sets the 1920x1080-point capture frame and the measured 150% capture zoom (125% on the built-in display) for one exact app process
• control-copilot-ui.swift — presses one named Accessibility control in an exact app process
• prepare-copilot-state.swift — prepares transient states: focus a hover-only control, highlight a menu item, set a field value, dismiss banners, and park the pointer
• locate-copilot-element.swift — prints the final 1920x1080 coordinates of named controls for callouts, boxes, and crops
• find-private-identities.swift — derives profile names and matching personal repository owners without hard-coded identities
• sanitize-screenshot.sh — replaces identity text while preserving avatars; caches identities per process because dialogs can hide the sidebar
• app-ui-map.md — sanitized factual UI map (menus, sidebar, views, composer controls), regenerable via map-app.sh
• capture-window.sh — recommended: process-filtered CoreGraphics window capture + WebP, with --callout, --box, --arrow, and --crop
• finalize-screenshot.py — enforces 1920x1080 output, adds the required 2px #cccccc inside border, and crops
• hide-window-controls.py — removes the macOS window buttons and moves the sidebar icon into their place, so screenshots do not show the operating system
• add-step-callouts.py — adds ordered red step badges, red highlight boxes, and red arrows to a finalized screenshot
• find-copilot-window.swift — lists on-screen Copilot windows with their window and process IDs
• map-app.sh — version-stamped UI map for grounding steps and diffing app updates
• capture-copilot-window.sh — older Accessibility-rectangle fallback

Drive App Work Without GUI Scripting

Prefer session orchestration:

• list_projects to find a project id.
• create_session to create a new isolated session.
• send_session_message to give that session work.
• get_session to inspect path and metadata.
• navigate_to only when the user explicitly wants to see the session.

See session-orchestration.md.

Use Canvases for Structured Human-Agent Work

For test boards, dashboards, course trackers, or visible state, use canvas surfaces instead of GUI clicking. Canvases are bidirectional and can expose agent-callable actions. See canvas-and-automation.md.

Decision Table

Goal	Best mechanism
Learn how sessions behave	Create separate sessions and compare outputs
Type/clear a prompt in the desktop UI	macOS Accessibility Message text area
Click a named app button	Accessibility by role/name/action
Capture a course screenshot	Accessibility window bounds + screencapture PNG + WebP conversion
Launch recurring experiments	App automations/workflows
Build a visible experiment dashboard	Canvas extension
Steer a CLI session away from machine	Copilot CLI remote control
Test PR/issue lifecycle	App session APIs plus GitHub issue/PR tools
Full native UI regression testing	Appium Mac2 after explicit setup

Source References

• GitHub Copilot app overview: https://docs.github.com/en/copilot/concepts/agents/github-copilot-app
• Agent sessions: https://docs.github.com/en/copilot/how-tos/github-copilot-app/agent-sessions
• Canvas extensions: https://docs.github.com/en/copilot/how-tos/github-copilot-app/working-with-canvas-extensions
• Automations: https://docs.github.com/en/copilot/how-tos/github-copilot-app/using-automations
• Copilot CLI remote control: https://docs.github.com/en/copilot/concepts/agents/copilot-cli/about-remote-control
• Apple UI scripting: https://developer.apple.com/library/archive/documentation/LanguagesUtilities/Conceptual/MacAutomationScriptingGuide/AutomatetheUserInterface.html