Judges

LLM-as-judge implementations. Each judge implements the Judge interface.

Interface

interface Judge {
    name: string
    initialize(config: JudgeConfig): Promise<void>
    evaluate(input: JudgeInput): Promise<JudgeResult>
    getPromptForQuestionType(questionType: string, providerPrompts?: ProviderPrompts): string
    getModel(): LanguageModel
}

Adding a Judge

1. Create src/judges/myjudge.ts
2. Implement Judge interface
3. Register in src/judges/index.ts
4. Add to JudgeName type in src/types/judge.ts
5. Add default model in src/utils/models.ts (DEFAULT_JUDGE_MODELS)

Required returns:

• initialize() - Set up client with API key and model
• evaluate() - Return { score: 0|1, label: "correct"|"incorrect", explanation: string }
• getPromptForQuestionType() - Return prompt string for question type
• getModel() - Return the initialized LanguageModel

Use these helpers from ./base.ts:

• buildJudgePrompt(input) - Builds full prompt from JudgeInput
• parseJudgeResponse(text) - Extracts JudgeResult from LLM response

Models

Add model config in src/utils/models.ts:

interface ModelConfig {
    id: string
    provider: "openai" | "anthropic" | "google"
    displayName: string
    supportsTemperature: boolean
    defaultTemperature: number
    maxTokensParam: "maxTokens" | "max_completion_tokens"
    defaultMaxTokens: number
}

Provider-Specific Prompts

Providers can override judge prompts. See providers/README.md.

Existing Judges

Judge	SDK	Default Model
openai	@ai-sdk/openai	gpt-4o
anthropic	@ai-sdk/anthropic	sonnet-4
google	@ai-sdk/google	gemini-2.5-flash