Desktop

The desktop capability drives native Windows applications. It addresses controls through Windows UI Automation, the accessibility layer behind every Windows app. It names up to four things about a control: its accessible Name, its ControlType, the application's own AutomationId and the window it sits inside.

#The locator

A DesktopLocator is the accessibility tree's own language. There is no coordinate field, no rectangle and no positional index anywhere in the type, because none of those is an identity: a coordinate survives no resize, no DPI change and no theme change.

const locator = {
  controlType: 'Button',
  name: 'Eight',
  automationId: 'num8Button',
  within: { controlType: 'Window', name: 'Calculator' },
};

within scopes the search to one top-level window. Its name is matched as a substring, because a browser puts its active tab in its own window title and the title moves. A target's name, by contrast, is matched exactly. after addresses the first element of the locator's ControlType that follows a named anchor, which is how a value that changes every run is read from the stable label beside it.

#The actions

ctx.desktop is a DesktopFacade with ten verbs:

• engine() reports which bridge answered and whether an interactive desktop is present.
• read(locator) returns the control's Name, ControlType and enabled flag, plus its toggle state for a checkable control.
• press(locator) invokes the control through UIA's Invoke pattern.
• text(locator) returns the element's accessible Name as a string, which for a display or a counter is its value.
• setValue(locator, value) writes through UIA's Value pattern.
• activate(locator) brings the window to the front.
• closeWindow(locator) closes a window through the Window pattern.
• select(locator, option) chooses a named option of a dropdown.
• toggle(locator) flips a checkable control through the Toggle pattern.
• selectItem(locator) selects the matched element through its own SelectionItem pattern.

The engine is the windows-uia bridge, a PowerShell-driven UIA provider. The platform never injects synthetic input. Every action is an accessibility API the application itself implements, so there is no SendKeys and no coordinate click to fall back on.

#When a control cannot be found

Every miss is a named SystemException, never a null. DesktopRefusalCode is a closed set of seven:

• DESKTOP_UIA_UNAVAILABLE, no bridge in this runtime.
• DESKTOP_SESSION_INTERACTIVE_REQUIRED, a bridge but no interactive desktop.
• DESKTOP_APP_NOT_RUNNING, the desktop presents no application window at all.
• DESKTOP_TARGET_NOT_RENDERED, the window is up but its content has not rendered yet.
• DESKTOP_MULTIPLE_MATCHES, more than one element matches, refused rather than taking the first.
• DESKTOP_TARGET_NOT_FOUND, nothing matches, or the locator is not a locator at all.
• DESKTOP_VALUE_BEARING_NOT_ADDRESSABLE, the element's accessible Name is its value, so the contract refuses to locate by that name.

The runner adds two more when a facade was never supplied: DESKTOP_NOT_DECLARED and DESKTOP_UNAVAILABLE.

#The window must be active

A Windows application publishes a collapsed accessibility tree until it is the active window. Notepad and Calculator both enumerate their window and offer no content tree until brought forward, which is why a run that starts with another window in front finds none of its controls and refuses DESKTOP_TARGET_NOT_FOUND against an application that is plainly live.

Make activate the first step:

import { defineAutomation } from '../framework';

export default defineAutomation({
  id: 'calculator-eight-press',
  displayName: 'Press eight on Calculator',
  capabilities: ['desktop'],
  url: '',
  getTransaction: async () => ({ id: 'one', payload: {} }),
  processTransaction: async (ctx) => {
    await ctx.desktop.activate({ controlType: 'Window', name: 'Calculator' });
    await ctx.desktop.press({
      controlType: 'Button',
      name: 'Eight',
      within: { controlType: 'Window', name: 'Calculator' },
    });
  },
});

#Things that catch people out

Declare 'desktop'. Both the runner and the Orchestrator gate on it. A desktop run also needs a robot on an interactive session. A service, or a task set to run whether or not a user is logged on, has no desktop at all.

A control whose accessible Name is its value cannot be located by that name. Address it by the application-owned AutomationId, or by an after relation to a stable anchor, or the contract refuses by name.