LLM-RFP

April 5, 2026 · View on GitHub

A tool for choosing the best LLM for your workload using a competitive "Request for Proposal" process — where the models themselves make the case for why they should be selected.

Concept

You have a project (e.g., a podcast, a coding workflow, a writing pipeline) and you want to find the best LLM for it. Instead of manually benchmarking models, let them pitch themselves.

How It Works

  1. User creates an RFP describing their workload, requirements, and expectations.
  2. RFP is sent to a pool of ~10 candidate models via the OpenRouter API. Each model receives the RFP and must explain why it is particularly well-suited to the workload.
  3. (Optional) Competitive layer: The user specifies which LLM they currently use. Each candidate model is told what the incumbent is and asked to argue why they could do it better.
  4. LLM-as-a-Judge evaluation: A judge model reviews all pitches and produces a shortlist of the top 3.
  5. User receives results: The top 3 pitches along with the judge's analysis of whether any are worth exploring or potentially better than the current model, considering the specific workload.

Candidate Selection Variations

  • Manual: User specifies which models to include in the candidate pool.
  • Automated: The assistant polls the OpenRouter API and selects a pool of ~10 candidate models based on the workload description (e.g., filtering by context window, pricing, modality support).

Features

  • Slash commands to support user onboarding and RFP creation
  • Results saved to the repo for future reference and comparison
  • OpenRouter API integration for broad model access

Stack

  • OpenRouter API

Status

Early concept / experiment.