Experiments & Evaluations Index

April 17, 2026 · View on GitHub

Index Experiments Evaluations POC Templates

Repository Index

This repository serves as an index of experimental projects, evaluations, proof-of-concepts, templates, patterns, and exploratory ideas related to AI/LLM development and workflows.

About this Index

This collection brings together various experimental repositories exploring AI agent workflows, LLM capabilities, evaluation frameworks, and development patterns. These repositories represent hands-on experiments, proof-of-concepts, benchmarking efforts, and reusable templates for AI-driven development.


Quick Reference: Evaluations

RepositoryCategoryKey Finding
Whisper Fine-Tune Accuracy EvalSpeechSmaller models improve with fine-tuning; larger models degrade unless handling code-switching
One-Shot Transcription Microphone EvalSpeechEnvironment matters more than equipment cost for STT accuracy
Transcription Cleanup EvalSpeechCompares cloud models on single-step transcription + cleanup
Audio Understanding Bitrate EvalSpeechMP3 bitrate has negligible effect on audio-LLM transcription above ~16 kbps; Gemini 3 Flash Preview leads accuracy, Voxtral leads accuracy-per-second
Whisper WPM Background Noise EvalSpeechSpeaking pace and background noise impact on Whisper accuracy
Long Form Audio EvalSpeechLong-form audio transcription evaluation
Local ASR STT BenchmarkSpeechLocal ASR/STT benchmarking
Hebrew Image Generation EvalImageHebrew text rendering in AI image generation
Bias Censorship Eval TestsLLMTesting for bias and censorship in LLMs

Quick Reference: Experiments

RepositoryCategoryDescription
Voice Cloning Difference TestSpeechHow training data duration affects voice cloning quality
Text Cleanup Fine-Tuning SetSpeechDataset for training AI to clean up STT transcripts
Voice Cleanup Prompt ExperimentSpeechComparing OpenAI vs Gemini for transcript cleanup
Impact Bond Policy SimulatorMulti-AgentSimulating stakeholder reactions to policy proposals
Peace In The Middle EastMulti-AgentAI simulation of geopolitical dialogue
Weird AI Experiment IdeatorMulti-AgentBlind multi-pass review for generating experiment ideas
LLM Long Codegen TestLLMTesting long-form code generation
Single Shot Brevity TrainingLLMTraining for concise responses

AI Agent Development

Agent Workflows & Patterns

RepositoryLinkDescriptionDate
Agent Handover DemoView RepoDemonstration of agent handover patterns2025
Agent Network Expander TemplateView RepoTemplate for expanding agent networks2025
Agent Task Repo Pattern With MCPView RepoRepository pattern for agent tasks using MCP2025
AI Agent UNView RepoAI agent unified namespace2025
AI Agent Workspace SpecView RepoAgent workspace specificationMar 2025
Weird AI Experiment IdeatorView RepoCrewAI multi-agent system using blind multi-pass review to generate creative AI experiment ideas2025

Development Templates

RepositoryLinkDescriptionDate
AI Assistant TemplateView RepoTemplate for AI assistant development2025
AI Dev Prompts ExampleView RepoExample development prompts for AI2025
AI Development TemplateView RepoGeneral AI development template2025

LLM Evaluation & Benchmarking

LLM Capabilities & Testing

RepositoryLinkDescriptionDate
Bias Censorship Eval TestsView RepoEvaluation tests for bias and censorship2025
LLM Evaluation PromptsView RepoPrompts for evaluating LLMs2025
Assistant Self IdeationView RepoDemo of AI "self ideation" in practiceFeb 2025

LLM Experiments

RepositoryLinkDescriptionDate
LLM Council TemplateView RepoPersonality-based LLM council deliberation template with PDF + podcast digest2026
LLM Experiment NotebookView RepoNotebook of LLM experiments2025
LLM Long Codegen TestView RepoTesting long-form code generation2025
LLM Max Token LengthView RepoMaximum token length explorationFeb 2025
Single Shot Brevity TrainingView RepoTraining for concise single-shot responses2025
One Prompt AI BookView RepoExperiment in generating content from single prompts2025
Long AI Prompting ExperimentView RepoTesting experiments with extended prompts2025

Hugging Face Spaces

RepositoryLinkDescriptionDate
Single Shot Brevity TrainingView SpaceBrevity training interface2025
LLM Long Code Generation ExperimentView SpaceLong-form code generation experiment2025
Max Output Tokens AnalysisView SpaceMaximum output tokens analysisFeb 2025

Speech-to-Text & Audio Processing

STT Benchmarks & Evaluation

RepositoryLinkDescriptionDate
Local ASR STT BenchmarkView RepoLocal ASR and STT benchmarking2025
Long Form Audio EvalView RepoEvaluation of long-form audio transcription2025
Personal STT BenchmarkingView RepoPersonal speech-to-text benchmarking2025
STT Voice Note EvaluationView RepoEvaluation of STT for voice notes2025
Whisper WPM Background Noise EvalView RepoEvaluating how speaking pace and background noise affect Whisper ASR accuracy2025
Whisper Fine-Tune Accuracy EvalView RepoGUI tool for comparing fine-tuned vs original Whisper models using WER metrics with whisper.cpp/Vulkan acceleration2025
One-Shot Transcription Microphone EvalView RepoMicrophone benchmarking for STT—found environment matters more than equipment cost across 10 mics and 15 samples2025
Transcription Cleanup EvalView RepoEvaluates cloud audio models (GPT-4o, Gemini, Voxtral, Qwen) on single-step transcription with cleanupDec 2025
Audio Understanding Bitrate EvalView RepoMP3 bitrate × 12 OpenRouter audio-LLMs × 4 dictation samples (240 API calls). Finds bitrate above 16 kbps barely matters for Gemini/Voxtral; OpenAI GPT-Audio models occasionally refuse verbatim-transcription and respond conversationallyApr 2026

Hugging Face Spaces

RepositoryLinkDescriptionDate
Single Podcast ASR EvalView SpaceSingle podcast ASR evaluation2025
STT ComparisonView SpaceSpeech-to-text comparison tool2025
Local STT Eval One SampleView SpaceLocal STT evaluation with single samples2025
Whisper Fine-Tune EvalView SpaceInteractive evaluation of fine-tuned Whisper models2025
Single Shot ASR EvalView SpaceSingle-shot ASR evaluation tool2025

Hugging Face Datasets

RepositoryLinkDescriptionDate
Podcast ASR EvaluationView DatasetDataset for podcast ASR evaluation2025
Whisper Fine-Tune One Shot EvalView DatasetWER and accuracy evaluation comparing fine-tuned Whisper (Tiny, Base, Small, Medium) vs stock models on 1 hour of audio, inference on Modal A1002025
Audio Understanding Bitrate EvalView DatasetBitrate × model sweep across 12 OpenRouter audio-LLMs; WER + latency per call, with source WAVs and MP3 variantsApr 2026

Audio Samples & Resources

RepositoryLinkDescriptionDate
Microphone Audio SamplesView RepoCollection of microphone audio samples2025

Audio Processing Experiments

RepositoryLinkDescriptionDate
Crying Baby Audio ScrubView RepoAudio processing for baby noise removal2025
Audio Context Pipeline ModelView RepoNotes and model for audio context pipelineApr 2025
Voice Cleanup Prompt ExperimentView RepoComparing OpenAI (Whisper+GPT-4) vs Gemini for transcript cleanup2025
Voice Cloning Difference TestView RepoExperiment testing how training data duration (1/3/5 min) affects one-shot voice cloning quality2025
Text Cleanup Fine-Tuning SetView RepoDataset and tooling for training AI to automatically clean up STT transcripts2025

Image Generation & Visual AI

Image Generation Evaluation

RepositoryLinkDescriptionDate
Hebrew Image Generation EvalView RepoEvaluation of AI image generation models for Hebrew text rendering2025

Specialized Applications

Multi-Agent Simulations

RepositoryLinkDescriptionDate
Impact Bond Policy SimulatorView RepoCrewAI multi-agent framework simulating stakeholder reactions to Pay-for-Success impact bond proposals2025
Peace In The Middle EastView RepoExperimental multi-agent AI system simulating geopolitical dialogue with state and non-state actors2025

OSINT & Intelligence

RepositoryLinkDescriptionDate
OSINT Missile Intelligence AgentView RepoOSINT-focused intelligence agent2025

Data Analysis

RepositoryLinkDescriptionDate
GHG EBITDA CorrelationsView RepoAnalysis of greenhouse gas and EBITDA correlations2025

Testing & Documentation

Test Repositories

RepositoryLinkDescriptionDate
Test Markdown DocsView RepoTest repository for markdown documentation2025
Test System PromptsView RepoTest repository for system prompts2025

IndexLinkDescription
Speech & ASR EvaluationsView RepoComprehensive index of speech recognition and ASR evaluation studies

Note: This is a focused index covering experimental AI/LLM development projects. For a higher-level collection of all repository indexes and other projects, see the GitHub Master Index.

Author

Daniel Rosehill Contact: public@danielrosehill.com Website: danielrosehill.com

Proof of Concepts — AI Self-Ideation

RepositoryLinkDescriptionDate
AI Agent Ideation AgentView RepoIdeation agent that generates ideas for AI agents2025
AI Assistant IdeatorView RepoStreamlit app for ideating AI assistants2025
Claude Space Self-ideatorView RepoClaude Code ideating ideas for new applications for Claude Code workspaces2025

Proof of Concepts — Context & Interview Workflows

RepositoryLinkDescriptionDate
Agentic Context Development Interview DemoView RepoDemonstration of chained LLM agent workflow for generating personal contextual data2025
AI Interview Workflow V2View RepoAI interview workflow, version 22025
My LLM Context Repo PublicView RepoA context repo for experimenting with LLM models (public version)2025

Proof of Concepts — Multi-Agent Panels & Decision Making

RepositoryLinkDescriptionDate
AI Agent Virtual Panel ConfigsView RepoSets of "panels" for testing virtual AI persona voting bodies and thinking groups2025
Claude AI ConferenceView RepoAI experiment: panel + TTS, mini conference/symposium2025
Panel Of ClaudeView RepoExploratory Claude model: multiple agents mimicking a panel debate2025
Claude Change My ViewView RepoCMV with AI (pattern/template)2025
Claude Decision Evaluation FrameworkView RepoClaude Code model for decision evaluation2025

Proof of Concepts — Research & Report Generation

RepositoryLinkDescriptionDate
All About MCPView RepoExample repository for agentic AI generated long form report generation2025
Claude Deep Research ModelView RepoRepo model for an iterative deep research model with voice pipeline2025
Claude Georeaction ResearcherView RepoClaude template for analysing global sentiment/reaction to a geopolitical issue2025
Geopol Forecaster POCView RepoExperimental prediction analysis for real world events2025

Proof of Concepts — Other

RepositoryLinkDescriptionDate
AI ResumeView RepoNotes/test for creating a resume/CV specifically intended for AI agents2025
Claude Agent Picker PatternView RepoPattern/idea for the "too many subagents" problem2025
Policy VisualiserView RepoVisualise how different countries approach policy challenges with Gemini identifying clusters2025
System Prompt FactoryView RepoA system prompt generation UI combining model and user characteristics2025
System Prompt Generation ConfigurationsView RepoSystem prompts for using AI tools to generate and improve system prompts2025
The Jerusalem Odyssey TextView RepoA 100 page book manuscript generated from a single prompt using Sonnet 3.72025

Additional Evaluations

RepositoryLinkDescriptionDate
Gemini 3.1 Lite Audio Understanding EvalView RepoVoice recording for testing with TTS/cloning2025

Additional Experiments

RepositoryLinkDescriptionDate
Better Web Design IncView RepoAgentic web design crew specialising in offbeat designs and client dissatisfaction!2025
Gemini Body Language AnalystView RepoTest app "vibe coded" in Google AI Studio: analyse body language from photo plus context2025
LLM DetectiveView RepoAgent that tries to probe other models' capabilities with conversation2025
LLM WarsView Repo(Experiment) LLMs argue "who's better" in a podcast2025
Natural Language Relationship DefinitionView RepoExperiment: trying to create a database schema using natural language2025
No BS AI System PromptView RepoSystem prompt for a blunt AI assistant that gets to the point2025
No Wheel InventionsView RepoSlash command for Claude to encourage avoiding reinventing the wheel2025
Two AIs TalkView RepoExperiment: two AI agents, each one thinks the other is a liar2025
Weird AI BotsView RepoSome configurations for offbeat AI roleplay characters just for fun2025

Data Visualization

RepositoryLinkDescriptionDate
Agentic AI Architecture VisualisationView RepoFramework-agnostic data model and visualizations mapping the moving pieces of agentic AI systems2025