HostAgent: Desktop Orchestrator

November 12, 2025 · View on GitHub

HostAgent serves as the centralized control plane of UFO². It interprets user-specified goals, decomposes them into structured subtasks, instantiates and dispatches AppAgent modules, and coordinates their progress across the system. HostAgent provides system-level services for introspection, planning, application lifecycle management, and multi-agent synchronization.

Architecture Overview

Operating atop the native Windows substrate, HostAgent monitors active applications, issues shell commands to spawn new processes as needed, and manages the creation and teardown of application-specific AppAgent instances. All coordination occurs through a persistent state machine, which governs the transitions across execution phases.

![HostAgent Architecture](../../img/hostagent2.png)
Figure: HostAgent architecture showing the finite state machine, processing pipeline, and interactions with AppAgents through the Blackboard pattern.

Core Responsibilities

Task Decomposition

Given a user's natural language input, HostAgent identifies the underlying task goal and decomposes it into a dependency-ordered subtask graph.

Example: User request "Extract data from Word and create an Excel chart" becomes:

  1. Extract table from Word document
  2. Create chart in Excel with extracted data
![Task Decomposition](../../img/decomposition.png)
Figure: HostAgent decomposes user requests into sequential subtasks, assigns each to the appropriate application, and orchestrates AppAgents to complete them in dependency order.

Application Lifecycle Management

For each subtask, HostAgent inspects system process metadata (via UIA APIs) to determine whether the target application is running. If not, it launches the program and registers it with the runtime.

AppAgent Instantiation

HostAgent spawns the corresponding AppAgent for each active application, providing it with task context, memory references, and relevant toolchains (e.g., APIs, documentation).

Task Scheduling and Control

The global execution plan is serialized into a finite state machine (FSM), allowing HostAgent to enforce execution order, detect failures, and resolve dependencies across agents. See State Machine Details for the FSM architecture.

Shared State Communication

HostAgent reads from and writes to a global blackboard, enabling inter-agent communication and system-level observability for debugging and replay.

Key Characteristics

  • Scope: Desktop-level orchestrator (system-wide, not application-specific)
  • Lifecycle: Single instance per session, persists throughout task execution
  • Hierarchy: Parent agent that manages multiple child AppAgents
  • Communication: Owns and coordinates the shared Blackboard
  • Control: 7-state finite state machine with 4-phase processing pipeline

Execution Workflow

sequenceDiagram
    participant User
    participant HostAgent
    participant Blackboard
    participant AppAgent1
    participant AppAgent2
    
    User->>HostAgent: "Extract Word table, create Excel chart"
    HostAgent->>HostAgent: Decompose into subtasks
    HostAgent->>Blackboard: Write subtask 1
    HostAgent->>AppAgent1: Create/Get Word AppAgent
    AppAgent1->>AppAgent1: Execute Word task
    AppAgent1->>Blackboard: Write result 1
    AppAgent1-->>HostAgent: Return FINISH
    
    HostAgent->>Blackboard: Read result 1
    HostAgent->>Blackboard: Write subtask 2
    HostAgent->>AppAgent2: Create/Get Excel AppAgent
    AppAgent2->>Blackboard: Read result 1
    AppAgent2->>AppAgent2: Execute Excel task
    AppAgent2->>Blackboard: Write result 2
    AppAgent2-->>HostAgent: Return FINISH
    
    HostAgent->>HostAgent: Verify completion
    HostAgent-->>User: Task completed

Deep Dive Topics

Input and Output

HostAgent Input

InputDescriptionType
User RequestNatural language task descriptionString
Application InformationActive application metadataList of Dicts
Desktop ScreenshotsVisual context of desktop stateImage
Previous Sub-TasksCompleted subtask historyList of Dicts
Previous PlanPlanned future subtasksList of Strings
BlackboardShared memory spaceDictionary

HostAgent Output

OutputDescriptionType
ObservationDesktop screenshot analysisString
ThoughtReasoning processString
Current Sub-TaskActive subtask descriptionString
MessageInformation for AppAgentString
ControlLabelSelected application indexString
ControlTextSelected application nameString
PlanFuture subtask sequenceList of Strings
StatusAgent state (CONTINUE/ASSIGN/FINISH/etc.)String
CommentUser-facing informationString
QuestionsClarification requestsList of Strings
BashSystem command to executeString

Example Output:

{
    "Observation": "Desktop shows Microsoft Word with document open containing a table",
    "Thought": "User wants to extract data from Word first",
    "Current Sub-Task": "Extract the table data from the document",
    "Message": "Starting data extraction from Word document",
    "ControlLabel": "0",
    "ControlText": "Microsoft Word - Document1",
    "Plan": ["Extract table from Word", "Create chart in Excel"],
    "Status": "ASSIGN",
    "Comment": "Delegating table extraction to Word AppAgent",
    "Questions": [],
    "Bash": ""
}

Architecture & Design:

Configuration:

System Integration:

API Reference

:::agents.agent.host_agent.HostAgent

Summary

HostAgent is the desktop-level orchestrator that:

  • Decomposes tasks and coordinates AppAgents
  • Operates at system level, not application level
  • Uses a 7-state FSM: CONTINUE → ASSIGN → AppAgent → CONTINUE → FINISH
  • Executes a 4-phase pipeline: DATA_COLLECTION → LLM → ACTION → MEMORY
  • Creates, caches, and reuses AppAgent instances
  • Provides shared Blackboard memory for all agents
  • Maintains single instance per session managing multiple AppAgents

Next Steps:

  1. Read State Machine for FSM details
  2. Read Processing Strategy for pipeline architecture
  3. Read Command System for available desktop operations
  4. Read AppAgent for application-level execution