Harness Runtime: Tools, Agent Loop, Middleware, Memory

July 4, 2026 · View on GitHub

Continues from README.md: tool registry, agent loop, middleware, and memory/stores.

Tool Registry

The tool registry owns available tools and their schemas.

pub struct ToolRegistry<State, Ctx = ()> {
    tools: HashMap<ToolName, Arc<dyn Tool<State, Ctx>>>,
}

#[async_trait]
pub trait Tool<State, Ctx = ()>: Send + Sync {
    fn name(&self) -> &str;
    fn description(&self) -> &str;
    fn schema(&self) -> ToolSchema;

    async fn call(
        &self,
        state: &State,
        ctx: &mut RunContext<Ctx>,
        call: ToolCall,
    ) -> Result<ToolResult>;
}

Tool schema requirements:

name
description
JSON schema compatible input shape
optional output schema
safety metadata
timeout override
retry override
model-visible flag for each argument
injected-runtime argument declarations that are hidden from model schemas
side-effect and idempotency metadata
confirmation policy for destructive operations
artifact output policy

Tool call requirements:

id
name
arguments
provider metadata
originating model call id
validation status
retry attempt

Tool result requirements:

tool_call_id
name
content
raw structured value
elapsed time
error flag
artifact references
user-visible summary
redacted event payload

Tool names should default to ASCII snake_case. The registry should reject duplicate names and invalid names.

Agent Loop

The default loop is the LangChain-style model-tool loop:

input messages
  -> build request
  -> call model
  -> if assistant has tool calls:
       validate tool calls
       execute tools
       append tool messages
       repeat
  -> final assistant message

Detailed lifecycle:

Create RunConfig and RunContext.
Load short-term memory for thread_id if configured.
Normalize input into messages.
Apply prompt templates and dynamic context.
Select model.
Select exposed tools.
Run before_model middleware, including prompt/cache-layout guards and pre-call compression.
Invoke or stream the model through wrap_model middleware.
Run on_model_delta middleware for streamed chunks.
Run after_model middleware, including post-call compression and summary persistence.
Emit model events and append assistant message.
If tool calls exist, validate name, schema, and limits.
Run before_tool middleware per call.
Execute tools — concurrently when the turn has two or more calls and no tool-wrap (ToolMiddleware) middleware is registered (wrap middleware holds &mut RunContext across each call, so it forces the serial path); results always fold back in original call order.
Run on_tool_delta middleware for tool progress streams.
Run after_tool middleware per result.
Append tool messages.
Repeat until no tool calls remain.
Validate structured output if configured.
Persist short-term memory.
Emit final event and return AgentRun.

Hard limits:

max_model_calls
max_tool_calls
max_concurrency
wall-clock timeout
per-call timeout
retry budget

The loop must fail closed when a limit is reached.

Middleware

Middleware is the main extension point for behavior that cuts across providers, tools, and graph nodes.

#[async_trait]
pub trait Middleware<State, Ctx = ()>: Send + Sync {
    async fn before_model(
        &self,
        state: &State,
        ctx: &mut RunContext<Ctx>,
        request: &mut ModelRequest,
    ) -> Result<()>;

    async fn on_model_delta(
        &self,
        state: &State,
        ctx: &mut RunContext<Ctx>,
        delta: &mut ModelDelta,
    ) -> Result<()>;

    async fn after_model(
        &self,
        state: &State,
        ctx: &mut RunContext<Ctx>,
        response: &mut ModelResponse,
    ) -> Result<()>;

    async fn before_tool(
        &self,
        state: &State,
        ctx: &mut RunContext<Ctx>,
        call: &mut ToolCall,
    ) -> Result<()>;

    async fn on_tool_delta(
        &self,
        state: &State,
        ctx: &mut RunContext<Ctx>,
        delta: &mut ToolDelta,
    ) -> Result<()>;

    async fn after_tool(
        &self,
        state: &State,
        ctx: &mut RunContext<Ctx>,
        result: &mut ToolResult,
    ) -> Result<()>;

    async fn on_error(
        &self,
        state: &State,
        ctx: &mut RunContext<Ctx>,
        error: &TinyAgentsError,
    ) -> Result<()>;
}

Middleware ordering is stable and explicit. Middleware runs in registration order for before_* hooks, registration order for streaming delta hooks, and reverse order for after_* hooks. Wrap hooks should surround the full model or tool operation when middleware needs setup, streaming inspection, and teardown as one unit.

Built-in middleware candidates:

tracing middleware
retry middleware
timeout middleware
model fallback middleware
token-bucket rate limiter middleware
prompt cache layout guard middleware
message trimming middleware
summarization middleware
context compression middleware
transcript compression middleware
retrieval compression middleware
streaming delta compression middleware
output compression middleware
context editing middleware
tool allowlist middleware
dynamic tool selection middleware
guardrail middleware
PII detection/redaction middleware
human-in-the-loop middleware
shell/filesystem privilege boundary middleware
structured output validator
rate limiter

Wrap hooks should exist in addition to before/after hooks. A wrap hook receives a request plus a handler and can call the handler, replace the request, retry, fallback to another model/tool, short-circuit with a response, or return a control command. Before/after hooks are simpler and should remain available for common mutation and observation cases.

Memory And Stores

Memory and storage are related but not the same feature. memory owns conversation semantics. store owns persistence backends.

Memory has two layers conceptually:

short-term memory: thread-scoped conversation state
long-term store: cross-thread application data

Short-term memory:

keyed by thread_id
loaded before an agent loop
updated after successful loop completion
optionally trimmed or summarized
useful for conversation continuity

Stores:

available through RunContext
namespaced
typed where possible
usable by tools and middleware
not automatically injected into prompts unless middleware does it
reusable by memory, event recording, tool artifacts, and web UIs

Suggested traits:

#[async_trait]
pub trait ShortTermMemory<State>: Send + Sync {
    async fn load(&self, thread_id: &ThreadId) -> Result<Option<State>>;
    async fn save(&self, thread_id: &ThreadId, state: &State) -> Result<()>;
}

The storage layer should be a separate harness feature:

#[async_trait]
pub trait Store: Send + Sync {
    async fn get(&self, key: StoreKey) -> Result<Option<StoreValue>>;
    async fn put(&self, key: StoreKey, value: StoreValue) -> Result<()>;
    async fn delete(&self, key: StoreKey) -> Result<()>;
    async fn scan(&self, prefix: StoreKeyPrefix) -> Result<Vec<StoreRecord>>;
}

#[async_trait]
pub trait AppendStore: Send + Sync {
    async fn append(&self, stream: StoreStream, value: StoreValue) -> Result<StoreOffset>;
    async fn read_from(&self, stream: StoreStream, offset: StoreOffset) -> Result<Vec<StoreRecord>>;
}

pub enum StoreValue {
    Json(serde_json::Value),
    Bytes(Vec<u8>),
    Text(String),
}

Initial store backends:

InMemoryStore: deterministic tests and examples.
JsonlStore: append-only local development, replayable event logs, and cheap debugging.
FileStore: local artifacts such as tool outputs, provider payload snapshots, and prompt fixtures.
MongoStore: durable application/runtime records for server deployments.

Later store backends:

SQLite for single-node durable local apps.
Postgres for multi-tenant production apps.
S3-compatible blob store for large artifacts.
Redis for short-lived cache/session data.

Store data classes:

run records
thread records
normalized messages
event envelopes
tool call records
model call records
structured outputs
user/application memory
tool artifacts and blobs

Backend selection should be per store namespace:

let stores = StoreRegistry::new()
    .register("events", JsonlStore::new("./data/events.jsonl"))
    .register("threads", MongoStore::new(mongo, "threads"))
    .register("artifacts", FileStore::new("./data/artifacts"));

Store events should flow through harness::events or the registry event bus:

store.read
store.write
store.append
store.delete
store.error

Sensitive store fields must support redaction before event emission.