Chapter 2: Realtime API Fundamentals

Welcome to Chapter 2: Realtime API Fundamentals. In this part of OpenAI Realtime Agents Tutorial: Voice-First AI Systems, you will build an intuitive mental model first, then move into concrete implementation details and practical production tradeoffs.

Realtime systems are event systems first and model systems second. Reliability comes from mastering session state and event flow.

Learning Goals

By the end of this chapter, you should be able to:

• map session lifecycle states clearly
• choose the right transport for your product constraints
• debug event-ordering and reconnect issues
• plan around current GA and deprecation timelines

Session Lifecycle

A practical lifecycle looks like this:

1. client requests short-lived session credentials
2. client opens realtime transport
3. session configuration is set (modalities, instructions, tool policy)
4. user input is committed to conversation
5. responses and tool events stream continuously
6. session is gracefully closed or resumed after reconnect

Transport Selection

Transport	Best For	Tradeoff
WebRTC	browser voice UX and low-latency media	more moving parts in signaling/media handling
WebSocket	server-side pipelines and custom clients	you own more media/runtime behavior

Event Design Principles

• treat every event as observable and traceable
• avoid relying on undocumented ordering assumptions
• make unknown events non-fatal in client handlers
• keep event handlers idempotent when possible

GA and Beta Timeline Note

OpenAI deprecation docs currently list February 27, 2026 as the Realtime beta interface shutdown date. New builds should target GA semantics and avoid beta-only behavior.

Practical Debug Framework

When issues occur, classify first:

• connection issue (transport/session establishment)
• protocol issue (event ordering/invalid payload)
• model issue (quality/content)
• tool/runtime issue (external dependency failures)

This prevents wasting time tuning prompts for transport bugs.

Realtime Contract Checklist

• schema validation for incoming/outgoing events
• reconnect strategy with bounded retries
• timeout and cancellation behavior for long operations
• structured logging for session, event type, and outcome

Source References

• OpenAI Realtime Guide
• OpenAI API Deprecations
• openai/openai-realtime-agents Repository

Summary

You now understand the realtime lifecycle and have a framework for protocol-level debugging and migration-safe implementation.

Next: Chapter 3: Voice Input Processing

Source Code Walkthrough

src/app/types.ts

The TranscriptItem interface in src/app/types.ts handles a key part of this chapter's functionality:

  toolLogic?: Record<
    string,
    (args: any, transcriptLogsFiltered: TranscriptItem[], addTranscriptBreadcrumb?: (title: string, data?: any) => void) => Promise<any> | any
  >;
  // addTranscriptBreadcrumb is a param in case we want to add additional breadcrumbs, e.g. for nested tool calls from a supervisor agent.
  downstreamAgents?:
    | AgentConfig[]
    | { name: string; publicDescription: string }[];
}

export type AllAgentConfigsType = Record<string, AgentConfig[]>;

export interface GuardrailResultType {
  status: "IN_PROGRESS" | "DONE";
  testText?: string; 
  category?: ModerationCategory;
  rationale?: string;
}

export interface TranscriptItem {
  itemId: string;
  type: "MESSAGE" | "BREADCRUMB";
  role?: "user" | "assistant";
  title?: string;
  data?: Record<string, any>;
  expanded: boolean;
  timestamp: string;
  createdAtMs: number;
  status: "IN_PROGRESS" | "DONE";
  isHidden: boolean;
  guardrailResult?: GuardrailResultType;
}

This interface is important because it defines how OpenAI Realtime Agents Tutorial: Voice-First AI Systems implements the patterns covered in this chapter.

src/app/types.ts

The Log interface in src/app/types.ts handles a key part of this chapter's functionality:

  instructions: string;
  tools: Tool[];
  toolLogic?: Record<
    string,
    (args: any, transcriptLogsFiltered: TranscriptItem[], addTranscriptBreadcrumb?: (title: string, data?: any) => void) => Promise<any> | any
  >;
  // addTranscriptBreadcrumb is a param in case we want to add additional breadcrumbs, e.g. for nested tool calls from a supervisor agent.
  downstreamAgents?:
    | AgentConfig[]
    | { name: string; publicDescription: string }[];
}

export type AllAgentConfigsType = Record<string, AgentConfig[]>;

export interface GuardrailResultType {
  status: "IN_PROGRESS" | "DONE";
  testText?: string; 
  category?: ModerationCategory;
  rationale?: string;
}

export interface TranscriptItem {
  itemId: string;
  type: "MESSAGE" | "BREADCRUMB";
  role?: "user" | "assistant";
  title?: string;
  data?: Record<string, any>;
  expanded: boolean;
  timestamp: string;
  createdAtMs: number;
  status: "IN_PROGRESS" | "DONE";
  isHidden: boolean;

This interface is important because it defines how OpenAI Realtime Agents Tutorial: Voice-First AI Systems implements the patterns covered in this chapter.

src/app/types.ts

The ServerEvent interface in src/app/types.ts handles a key part of this chapter's functionality:

}

export interface ServerEvent {
  type: string;
  event_id?: string;
  item_id?: string;
  transcript?: string;
  delta?: string;
  session?: {
    id?: string;
  };
  item?: {
    id?: string;
    object?: string;
    type?: string;
    status?: string;
    name?: string;
    arguments?: string;
    role?: "user" | "assistant";
    content?: {
      type?: string;
      transcript?: string | null;
      text?: string;
    }[];
  };
  response?: {
    output?: {
      id: string;
      type?: string;
      name?: string;
      arguments?: any;
      call_id?: string;

This interface is important because it defines how OpenAI Realtime Agents Tutorial: Voice-First AI Systems implements the patterns covered in this chapter.

src/app/types.ts

The LoggedEvent interface in src/app/types.ts handles a key part of this chapter's functionality:

}

export interface LoggedEvent {
  id: number;
  direction: "client" | "server";
  expanded: boolean;
  timestamp: string;
  eventName: string;
  eventData: Record<string, any>; // can have arbitrary objects logged
}

// Update the GuardrailOutputZod schema to use the shared ModerationCategoryZod
export const GuardrailOutputZod = z.object({
  moderationRationale: z.string(),
  moderationCategory: ModerationCategoryZod,
  testText: z.string().optional(),
});

export type GuardrailOutput = z.infer<typeof GuardrailOutputZod>;

This interface is important because it defines how OpenAI Realtime Agents Tutorial: Voice-First AI Systems implements the patterns covered in this chapter.

How These Components Connect

flowchart TD
    A[TranscriptItem]
    B[Log]
    C[ServerEvent]
    D[LoggedEvent]
    E[based]
    A --> B
    B --> C
    C --> D
    D --> E