🖥️ Computer API Reference

February 11, 2026 · View on GitHub

Overview

The Computer module provides comprehensive desktop automation capabilities including mouse operations,keyboard input, screen capture, and window management. It enables automated UI testing and RPA workflows.

📚 Tutorial

Computer Use Guide

Automate desktop applications

📋 Requirements

  • Requires windows_latest image for computer use features

Computer

Computer module for desktop UI automation. Provides comprehensive desktop automation capabilities including mouse, keyboard,window management, application management, and screen operations.

Constructor

public Computer(Session session)

Methods

startApp

public ProcessListResult startApp(String startCmd, String workDirectory, String activity)

public ProcessListResult startApp(String startCmd, String workDirectory)

public ProcessListResult startApp(String startCmd)

Starts an application with the given command, optional working directory and optional activity.

Parameters:

  • startCmd (String): The command to start the application
  • workDirectory (String): working directory for the application
  • activity (String): activity name to launch (for mobile apps). Defaults to empty string.

Returns:

  • ProcessListResult: ProcessListResult containing the list of processes started and error message if any.

stopAppByPName

public AppOperationResult stopAppByPName(String pname)

Stops an application by process name.

Parameters:

  • pname (String): The process name of the application to stop

Returns:

  • AppOperationResult: AppOperationResult containing success status and error message if any

stopAppByPID

public AppOperationResult stopAppByPID(int pid)

Stops an application by process ID.

Parameters:

  • pid (int): The process ID of the application to stop

Returns:

  • AppOperationResult: AppOperationResult containing success status and error message if any

stopAppByCmd

public AppOperationResult stopAppByCmd(String stopCmd)

Stops an application by stop command.

Parameters:

  • stopCmd (String): The command to stop the application

Returns:

  • AppOperationResult: AppOperationResult containing success status and error message if any

listVisibleApps

public ProcessListResult listVisibleApps()

Lists all applications with visible windows. Returns detailed process information for applications that have visible windows,including process ID, name, command line, and other system information. This is useful for system monitoring and process management tasks.

Returns:

  • ProcessListResult: ProcessListResult containing list of visible applications with detailed process information

getInstalledApps

public InstalledAppListResult getInstalledApps(boolean startMenu, boolean desktop, boolean ignoreSystemApps)

public InstalledAppListResult getInstalledApps()

Gets the list of installed applications.

Parameters:

  • startMenu (boolean): Whether to include start menu applications. Defaults to true.
  • desktop (boolean): Whether to include desktop applications. Defaults to false.
  • ignoreSystemApps (boolean): Whether to ignore system applications. Defaults to true.

Returns:

  • InstalledAppListResult: InstalledAppListResult containing list of installed apps and error message if any

clickMouse

public BoolResult clickMouse(int x, int y, MouseButton button)

public BoolResult clickMouse(int x, int y)

public BoolResult clickMouse(int x, int y, String button)

Clicks the mouse at the specified screen coordinates.

Parameters:

  • x (int): X coordinate in pixels (0 is left edge of screen)
  • y (int): Y coordinate in pixels (0 is top edge of screen)
  • button (MouseButton): Mouse button to click. Options: - MouseButton.LEFT: Single left click - MouseButton.RIGHT: Right click (context menu) - MouseButton.MIDDLE: Middle click (scroll wheel) - MouseButton.DOUBLE_LEFT: Double left click Defaults to MouseButton.LEFT

Returns:

  • BoolResult: BoolResult Object containing: - success (boolean): Whether the click succeeded - data (Boolean): True if successful, null otherwise - errorMessage (String): Error description if failed

Throws:

  • IllegalArgumentException: If button is not one of the valid options

Behavior:

  • Clicks at the exact pixel coordinates provided
  • Does not move the mouse cursor before clicking
  • For double-click, use MouseButton.DOUBLE_LEFT
  • Right-click typically opens context menus

Note:

  • Coordinates are absolute screen positions, not relative to windows
  • Use getScreenSize() to determine valid coordinate ranges
  • Consider using moveMouse() first if you need to see cursor movement

moveMouse

public BoolResult moveMouse(int x, int y)

Moves the mouse to the specified coordinates.

Parameters:

  • x (int): X coordinate
  • y (int): Y coordinate

Returns:

  • BoolResult: BoolResult Result object containing success status and error message if any

Note:

  • Moves the cursor smoothly to the target position
  • Does not click after moving
  • Use getCursorPosition() to verify the new position

dragMouse

public BoolResult dragMouse(int fromX, int fromY, int toX, int toY, MouseButton button)

public BoolResult dragMouse(int fromX, int fromY, int toX, int toY)

public BoolResult dragMouse(int fromX, int fromY, int toX, int toY, String button)

Drags the mouse from one point to another.

Parameters:

  • fromX (int): Starting X coordinate
  • fromY (int): Starting Y coordinate
  • toX (int): Ending X coordinate
  • toY (int): Ending Y coordinate
  • button (MouseButton): Mouse button to use. Defaults to LEFT

Returns:

  • BoolResult: BoolResult containing success status and error message if any

scroll

public BoolResult scroll(int x, int y, ScrollDirection direction, int amount)

public BoolResult scroll(int x, int y)

public BoolResult scroll(int x, int y, String direction, int amount)

Scrolls the mouse wheel at the specified coordinates.

Parameters:

  • x (int): X coordinate
  • y (int): Y coordinate
  • direction (ScrollDirection): Scroll direction. Defaults to UP
  • amount (int): Scroll amount. Defaults to 1

Returns:

  • BoolResult: BoolResult containing success status and error message if any

getCursorPosition

public OperationResult getCursorPosition()

Gets the current cursor position.

Returns:

  • OperationResult: OperationResult Result object containing cursor position data with keys 'x' and 'y', and error message if any

Note:

  • Returns the absolute screen coordinates
  • Useful for verifying mouse movements
  • Position is in pixels from top-left corner (0, 0)

inputText

public BoolResult inputText(String text)

Types text into the currently focused input field.

Parameters:

  • text (String): The text to input. Supports Unicode characters

Returns:

  • BoolResult: BoolResult Object with success status and error message if any

Note:

  • Requires an input field to be focused first
  • Use clickMouse() or UI automation to focus the field
  • Supports special characters and Unicode

pressKeys

public BoolResult pressKeys(List<String> keys, boolean hold)

public BoolResult pressKeys(List<String> keys)

Presses the specified keys.

Parameters:

  • keys (List): List of keys to press (e.g., Arrays.asList("Ctrl", "a"))
  • hold (boolean): Whether to hold the keys. Defaults to false

Returns:

  • BoolResult: BoolResult Result object containing success status and error message if any

Note:

  • Key names are case-sensitive
  • When hold=true, remember to call releaseKeys() afterwards
  • Supports modifier keys like Ctrl, Alt, Shift
  • Can press multiple keys simultaneously for shortcuts

releaseKeys

public BoolResult releaseKeys(List<String> keys)

Releases the specified keys.

Parameters:

  • keys (List): List of keys to release (e.g., Arrays.asList("Ctrl", "a"))

Returns:

  • BoolResult: BoolResult Result object containing success status and error message if any

Note:

  • Should be used after pressKeys() with hold=true
  • Key names are case-sensitive
  • Releases all keys specified in the list

getScreenSize

public OperationResult getScreenSize()

Gets the screen size and DPI scaling factor.

Returns:

  • OperationResult: OperationResult Result object containing screen size data with keys 'width', 'height', and 'dpiScalingFactor', and error message if any

Note:

  • Returns the full screen dimensions in pixels
  • DPI scaling factor affects coordinate calculations on high-DPI displays
  • Use this to determine valid coordinate ranges for mouse operations

screenshot

public OperationResult screenshot()

Takes a screenshot of the current screen.

Returns:

  • OperationResult: OperationResult Result object containing the path to the screenshot and error message if any

Note:

  • Returns an OSS URL to the screenshot image
  • Screenshot captures the entire screen
  • Useful for debugging and verification
  • Image format is typically PNG

betaTakeScreenshot

public ScreenshotBytesResult betaTakeScreenshot(String format)

public ScreenshotBytesResult betaTakeScreenshot()

Takes a screenshot of the Computer and returns raw binary image data (beta).

This API uses the MCP tool `screenshot` (wuying_capture) and returns raw binary image data. The backend also returns the captured image dimensions (width/height in pixels), which are exposed on ScreenshotBytesResult.width and ScreenshotBytesResult.height. The backend metadata fields `type` and `mime_type` are exposed on ScreenshotBytesResult.type and ScreenshotBytesResult.mimeType.

Parameters:

  • format (String): The desired image format (default: "png"). Supported: "png", "jpeg", "jpg"

Returns:

  • ScreenshotBytesResult: ScreenshotBytesResult Object containing the screenshot image data (bytes) and metadata including type, mimeType, width, and height when provided by the backend

Throws:

  • IllegalArgumentException: If format is invalid

Supported formats:

  • "png"
  • "jpeg" (or "jpg")

listRootWindows

public WindowListResult listRootWindows(int timeoutMs)

public WindowListResult listRootWindows()

Lists all root windows.

Parameters:

  • timeoutMs (int): Timeout in milliseconds. Defaults to 3000

Returns:

  • WindowListResult: WindowListResult Result object containing list of windows and error message if any

getActiveWindow

public WindowInfoResult getActiveWindow(int timeoutMs)

public WindowInfoResult getActiveWindow()

Gets the currently active window.

Parameters:

  • timeoutMs (int): Timeout in milliseconds. Defaults to 3000

Returns:

  • WindowInfoResult: WindowInfoResult Result object containing active window info and error message if any

Note: Java version requires timeoutMs parameter, while Python version does not.

activateWindow

public BoolResult activateWindow(int windowId)

Activates the specified window.

Parameters:

  • windowId (int): The ID of the window to activate

Returns:

  • BoolResult: BoolResult Result object containing success status and error message if any

Note:

  • The window must exist in the system
  • Use listRootWindows() to get available window IDs
  • Activating a window brings it to the foreground

closeWindow

public BoolResult closeWindow(int windowId)

Closes the specified window.

Parameters:

  • windowId (int): The ID of the window to close

Returns:

  • BoolResult: BoolResult Result object containing success status and error message if any

Note:

  • The window must exist in the system
  • Use listRootWindows() to get available window IDs
  • Closing a window terminates it permanently

maximizeWindow

public BoolResult maximizeWindow(int windowId)

Maximizes the specified window.

Parameters:

  • windowId (int): The ID of the window to maximize

Returns:

  • BoolResult: BoolResult Result object containing success status and error message if any

Note:

  • The window must exist in the system
  • Maximizing expands the window to fill the screen
  • Use restoreWindow() to return to previous size

minimizeWindow

public BoolResult minimizeWindow(int windowId)

Minimizes the specified window.

Parameters:

  • windowId (int): The ID of the window to minimize

Returns:

  • BoolResult: BoolResult Result object containing success status and error message if any

Note:

  • The window must exist in the system
  • Minimizing hides the window in the taskbar
  • Use restoreWindow() or activateWindow() to bring it back

restoreWindow

public BoolResult restoreWindow(int windowId)

Restores the specified window.

Parameters:

  • windowId (int): The ID of the window to restore

Returns:

  • BoolResult: BoolResult Result object containing success status and error message if any

Note:

  • The window must exist in the system
  • Restoring returns a minimized or maximized window to its normal state
  • Works for windows that were previously minimized or maximized

resizeWindow

public BoolResult resizeWindow(int windowId, int width, int height)

Resizes the specified window.

Parameters:

  • windowId (int): The ID of the window to resize
  • width (int): New width of the window
  • height (int): New height of the window

Returns:

  • BoolResult: BoolResult Result object containing success status and error message if any

Note:

  • The window must exist in the system
  • Width and height are in pixels
  • Some windows may have minimum or maximum size constraints

fullscreenWindow

public BoolResult fullscreenWindow(int windowId)

Makes the specified window fullscreen.

Parameters:

  • windowId (int): The ID of the window to make fullscreen

Returns:

  • BoolResult: BoolResult containing success status and error message if any

Note:

  • The window must exist in the system
  • Fullscreen mode hides window borders and taskbar
  • Different from maximizeWindow() which keeps window borders
  • Press F11 or ESC to exit fullscreen in most applications

focusMode

public BoolResult focusMode(boolean on)

Toggles focus mode on or off.

Parameters:

  • on (boolean): True to enable focus mode, False to disable it

Returns:

  • BoolResult: BoolResult containing success status and error message if any

Note:

  • Focus mode helps reduce distractions by managing window focus
  • When enabled, may prevent background windows from stealing focus
  • Behavior depends on the window manager and OS settings

💡 Best Practices

  • Verify screen coordinates before mouse operations
  • Use appropriate delays between UI interactions
  • Handle window focus changes properly
  • Take screenshots for verification and debugging
  • Use keyboard shortcuts for efficient automation
  • Clean up windows and applications after automation