Computer control

The computer tool drives the machine outside the browser: desktop applications, installers, native dialogs, anything that is not a web page.

For a web page, use browser instead. It reads the DOM and clicks by element number, so it is faster, cheaper in tokens and it does not miss.

The fast path needs no screenshot

This is the one that matters, and the only one that works if you cannot see images.

1. windows lists every open window, with its size, position, application and which screen it is on. The focused one is marked *, and the screen your last screenshot came from is marked <- captured. Minimised windows are listed separately, by title: they are still open, and focus_window or restore_window brings one back.

   On a multi-monitor setup this is the line that matters. A window on screen 2 cannot be clicked with coordinates read off a capture of screen 1: those numbers are valid only for the screen they came from, so you would click a real point on the wrong monitor. Either capture that screen first, or act by ref, which needs no coordinates at all and does not care which monitor the window sits on. A window straddling two monitors is attributed to the one showing most of it, which is the one a human would name.

2. focus_window { "window": "Notepad" } brings one to the front, matching a piece of its title.

3. read returns a numbered map of that window's controls, plus the static text on it:

   ref_1 <document> Text editor = the current contents
   ref_7 <tab> notes.txt
   ref_13 <menuitem> File
   ref_21 <button> Minimise
   

4. click, double_click, right_click and fill take a ref instead of coordinates. focus gives a control focus without acting on it.

5. read again to see what changed.

Acting by ref goes through the OS accessibility API. It calls the control directly instead of simulating someone aiming at it, so it is deterministic, it cannot land a pixel off, and it works even when the window is not in front. The result says which pattern was used (invoke, toggle, select, expand, setvalue, rangevalue, default action) or tells you it had to fall back to a real click.

Refs come only from the latest read or find. Every read renumbers them, and changing window throws them away. Read again after acting, exactly as with the browser tool.

read uses UI Automation, so it is wired for Windows today. Elsewhere, use the pixel path below; windows still works everywhere.

When the window is too big to read

read stops at 300 controls and 120 lines of static text, and says so when it truncates. On a large Electron app or an office suite you will hit that, and you will pay for it every time.

find { "text": "Save" } runs the same read and returns only the controls whose label matches. Use it whenever you already know what you are looking for. It renumbers refs like any other read, and the numbers it hands back are usable immediately.

What a ref means depends on the action

Every one of these acts on the element, not on a guess:

Action with a ref	What happens
left_click	its automation pattern, or a real click if it exposes none
right_click, double_click, triple_click, middle_click	a real click at its centre
mouse_move (or hover)	the pointer moves onto it and nothing is clicked
scroll	it is scrolled into view through its own pattern
left_click_drag	it is the grab point, to_x and to_y are the drop
fill	text into a field, or a number into a slider or spinner
focus	focus only, no action

A slider or a spinner is set, not clicked. click on one is refused, and the refusal names its minimum, maximum and current value. Use fill with the number you want. A click would have landed at the centre of the control's rectangle, which is halfway along its travel, and reported success.

scroll on a ref is the only way to reach an off-screen control without a screenshot. read skips anything scrolled out of view, so a control below the fold does not exist as far as the tree is concerned. Scroll a nearby ref into view, then read again.

The pixel path, for what the tree cannot see

Hand-drawn interfaces expose nothing: games, canvases, poorly tagged Electron apps. When read comes back empty, fall back to looking.

1. screens lists the monitors: rank, id, size, position, scale, name.
2. screenshot captures one (screen picks it by id, rank or name; default is the primary) and returns it as an image, 1280px wide by default. That capture sets the coordinate system.
3. mouse_move, left_click, left_click_drag, scroll at x,y read off that image.

Coordinates are always pixels of the last screenshot. Never desktop pixels. The tool converts, including on a display scaled to 150% and on a mixed setup where a 4K sits next to a 1080p. Pointing before any screenshot is refused on purpose: without a capture there is no shared coordinate system, and treating the numbers as desktop pixels would work on a simple setup and fail silently everywhere else.

max_width raises the resolution when you genuinely cannot read what you need. It costs tokens in proportion.

Moving the windows themselves

move_window takes window plus any of x, y, width, height, in desktop pixels. A maximised window is restored first, because Windows ignores a placement request on one and would report a success that changes nothing.

minimize_window, maximize_window and restore_window do what they say. close_window is the cross, not a kill: the application gets the request, may put up a save prompt, and may refuse. Call windows again to see whether it actually went, and read the screen if a dialog appeared. To really kill a process, that is shell_exec.

All of them take a piece of the title, like focus_window. An exact title wins over a partial match, and an ambiguous piece is refused with the list of candidates rather than acted on: closing the wrong window is not recoverable.

Dragging

left_click_drag does the whole gesture in one call: press, move in steps, release. It holds briefly at each end, because HTML5, Electron and most file managers treat a press that moves immediately as a click and drop nothing. hold_ms changes that pause, button makes it a right or middle drag, which is how CAD and 3D tools orbit and pan.

For anything the single call cannot express, mouse_down and mouse_up hold a button across several calls, exactly as key_down and key_up do for the keyboard:

• drawing a path on a canvas, or a lasso selection
• reading a slider's value part way through its travel
• dropping onto a target that has to open under the pointer first
• a drag with a modifier: key_down Shift, mouse_down, mouse_move, mouse_up, key_up

Keys

type types into whatever has focus, up to 5000 characters. For more than that, or for anything with awkward characters, put it on the clipboard with write_clipboard and press Control+v: there is no length limit that way, and no per-character timing to go wrong.

key presses one key or a chord.

Both tell you which window they went to. Read that line. A keystroke goes to the front window, not to the one you have in mind, and on a multi-screen setup the window you were thinking of is often on the other screen with something else in front. If the reported focus is not what you expected, nothing you typed went where you thought: focus_window first, then type again. Typing into LaRuche's own window is refused outright.

Named keys: Enter, Tab, Escape, Backspace, Delete, Space, Home, End, PageUp, PageDown, Up, Down, Left, Right, F1 to F12, or a single character. Chord with Control+c, Shift+Tab, Alt+F4, Meta+r. repeat presses several times, hold_ms holds each press.

key_down and key_up hold a key while other gestures happen, which is what a game needs and what a drag with a modifier needs.

Anything left held is released for you after a minute, and release_all releases everything on demand. Release deliberately anyway: a key still down changes what every later click means, including the user's own, and the automatic release is a safety net for an abandoned mission, not a substitute for finishing the gesture.

The clipboard

read_clipboard and write_clipboard move text in and out. This is the shortest route into and out of a native application, and it is the only way to read what a Control+c actually copied.

Waiting

wait { "ms": 1500 } pauses while an installer advances, a menu animates open, or a dialog appears. Use it instead of taking a screenshot you do not need: a capture costs tokens and a wait costs nothing.

What the human sees

The screen being driven gets an amber frame, a floating panel that names each action as it happens, and a ring that follows the cursor and pulses on every click. The cursor glides to its target instead of teleporting, text appears character by character, and a screenshot triggers a brief flash once the capture is taken. It all fades a few seconds after LaRuche stops acting. The panel can be dragged anywhere and stays where the user puts it.

glow: false removes the decoration, animate: false acts instantly on a long sequence, speed above 1 slows the motion down for a demonstration. Acting by ref shows the panel line without moving the mouse at all, because nothing needs to be aimed at.

What it refuses, and why

Two refusals are by design, not bugs to work around:

• LaRuche's own windows. A click there would let you answer your own approval prompts, which would make every safeguard in the node decorative. The check is by process id, not by window title.
• The human took the mouse back. If the pointer has moved more than 30 pixels from where the tool left it, and less than 45 seconds have passed, the next gesture is refused and nothing is done. Say so and wait. Repeating the same call takes control back deliberately, so it is not a deadlock, but do not repeat it reflexively: someone is using the machine. type is checked too, even though it aims at no point: a long burst of typing is exactly what someone reaching for the mouse wants to stop.

A window running as administrator cannot be driven at all. Windows blocks synthetic input across that privilege boundary, silently: the click is sent, does nothing, and returns no error, which is indistinguishable from aiming badly. focus_window and type warn you when the target is elevated, and the window actions refuse outright. This cannot be worked around and should not be. Say that the window has to be driven by hand, or that LaRuche has to be restarted as administrator, and stop trying.

The abort shortcut

Ctrl+Alt+Shift+H stops everything, immediately. It is registered globally as soon as the tool is first used, and it works whether or not the glow is on. Pressing it releases every held key and button, refuses the call in flight, and interrupts a long burst of typing part way through, reporting how many characters had already landed.

If a call comes back saying the user aborted, they took the machine back on purpose. Say so and stop. Do not retry the gesture, and do not look for another way to do the same thing: that is the one response the shortcut exists to prevent.

Approval comes in two classes. Approving one observing action (screens, screenshot, cursor_position) does not approve a click, and the first acting call is asked separately.

The whole tool can be switched off on a node with LARUCHE_COMPUTER=0. If every call comes back saying GUI control is disabled, that is why, and it is the operator's decision: say so rather than looking for another way in.

When not to use it

• A web page: browser.
• Moving, renaming, converting or reading files: shell_exec or file_*. Driving a file manager by clicking is slower and fails in more ways.
• Anything a command line can do: run the command. Clicking through a settings dialog to read a value that systeminfo prints is wasted work and wasted tokens.

The right use is the case with no other door: a native application with no CLI, an installer, a dialog the OS put on top of everything.