CI/CD Integration

June 3, 2026 · View on GitHub

Problem: How do you prevent broken AI agents from reaching production? Manual testing doesn't scale, and LLM outputs are non-deterministic.

Solution: EvalView integrates with CI/CD pipelines to automatically run agent regression tests on every PR and block merges when behavior degrades. It provides a GitHub Action, PR comments with alerts, job summaries, proper exit codes, and retry-resilient posting.

EvalView is CLI-first. You can run it locally or add to CI.

Quick Start (GitHub Actions)

Copy this to .github/workflows/evalview.yml:

name: EvalView Regression Check

on:
  pull_request:
  push:
    branches: [main]

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Check for regressions
        id: evalview
        uses: hidai25/eval-view@v0.8.0
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          fail-on: REGRESSION

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: evalview-report
          path: evalview-report.html
          if-no-files-found: ignore

      - name: Post PR comment
        if: github.event_name == 'pull_request' && always()
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          pip install evalview -q
          evalview ci comment --results ${{ steps.evalview.outputs.results-file }}

Setup:

Add OPENAI_API_KEY to your repository secrets
Run evalview snapshot locally to create your baseline
Commit .evalview/golden/ to git

That's it. Every PR now gets a regression check, a PR comment, and a job summary.

PR Comments

evalview ci comment generates a markdown comment and posts it to your PR. It auto-detects the PR context from GitHub Actions environment variables.

What you see on a clean PR

## ✅ EvalView: PASSED

| Metric | Value |
|--------|-------|
| Tests | 5/5 unchanged (100%) |

What you see when something breaks

## ❌ EvalView: REGRESSION

> **Alerts**
> - 💸 Cost spike: \$0.02 → \$0.08 (+300%)
> - ⏱️ Latency spike: 1.5s → 4.2s (+180%)
> - 🤖 Model/runtime update: declared, high confidence

| Metric | Value |
|--------|-------|
| Tests | 3/5 unchanged (60%) |
| Regressions | 1 |
| Tools Changed | 1 |
| Model Runtime Signal | declared (high confidence) |

### Changes from Baseline
- ❌ **search-flow**: score -15.0, 1 tool change(s)
- ⚠️ **create-flow**: 1 tool change(s)

<details>
<summary>... and 3 more change(s)</summary>

- ⚠️ **update-flow**
- ⚠️ **delete-flow**
- ⚠️ **list-flow**

</details>

[View full report](https://github.com/yourorg/yourrepo/actions/runs/12345)

---
*Generated by EvalView*

Features

Feature	Description
Cost spike alerts	Warns when total cost increases >50% from baseline
Latency spike alerts	Warns when total latency increases >50% from baseline
Model/runtime callout	Shows declared or suspected upstream updates with evidence
Collapsible overflow	First 5 changes shown inline, rest collapsed in `<details>`
Comment deduplication	Updates existing EvalView comment instead of creating duplicates
Retry with backoff	Retries `gh` CLI calls up to 3x with exponential backoff
Job summary	Writes results to `$GITHUB_STEP_SUMMARY` for Actions UI visibility

CLI Options

evalview ci comment                              # Auto-detect results + post
evalview ci comment --results path/to/file.json  # Specific results file
evalview ci comment --dry-run                    # Preview without posting
evalview ci comment --no-update                  # Always create new comment

Three Comment Formats

evalview ci comment auto-detects the results format:

Source	Command	Comment
`evalview check --json`	Regression detection	Diffs, alerts, model changes
`evalview run`	Full evaluation	Scores, pass/fail, cost
`evalview generate`	Test generation	Discovered tools, coverage gaps

Job Summary

Every CI run writes results to $GITHUB_STEP_SUMMARY, making them visible directly in the GitHub Actions UI — no need to click into the PR comment or download artifacts.

The job summary contains the same markdown as the PR comment: status, metrics, alerts, and diffs.

The job summary is written by the evalview ci comment step in your workflow. The recommended Quick Start workflow above includes this step. If you omit it, no job summary will appear.

Review Generated Suites in PRs

If you use evalview generate, every run writes a machine-readable suite report:

tests/generated/generated.report.json

Turn that into a PR comment:

evalview ci comment --results tests/generated/generated.report.json

The generated-suite comment includes:

discovered tools
draft behavior paths
coverage gaps
approval instructions for snapshot --approve-generated

Recommended flow:

evalview generate --agent http://localhost:8000
evalview ci comment --results tests/generated/generated.report.json
evalview snapshot tests/generated --approve-generated

GitHub Action Reference

Inputs

Input	Description	Default
`mode`	`check` (regression detection) or `run` (full evaluation)	`check`
`openai-api-key`	OpenAI API key for LLM-as-judge	-
`anthropic-api-key`	Anthropic API key (optional)	-
`test-path`	Path to test cases directory	`tests`
`config-path`	Path to config file	`.evalview/config.yaml`
`fail-on`	Statuses that fail CI (comma-separated)	`REGRESSION`
`strict`	Fail on any change	`false`
`diff`	Compare against golden baselines (run mode)	`false`
`contracts`	Check MCP contracts for drift	`false`
`filter`	Filter tests by name pattern	-
`max-workers`	Parallel workers	`4`
`max-retries`	Retry failed tests	`2`
`fail-on-error`	Fail workflow on test failure	`true`
`generate-report`	Generate HTML report	`true`
`python-version`	Python version	`3.11`

Outputs

Output	Description
`results-file`	Path to JSON results
`report-file`	Path to HTML report
`total-tests`	Total tests run
`passed-tests`	Passed/unchanged count
`failed-tests`	Failed/regressed count
`regressions`	Regressions detected (check mode)
`tools-changed`	Tests with tool changes (check mode)
`pass-rate`	Pass rate percentage
`status`	Overall: `passed`, `regression`, `tools_changed`, `output_changed`

Configurable Strictness

Control what fails your CI:

# Default: only fail on score drops
evalview check --fail-on REGRESSION

# Stricter: also fail on tool changes
evalview check --fail-on REGRESSION,TOOLS_CHANGED

# Strictest: fail on any change
evalview check --strict

Exit Codes

Scenario	Exit Code
All tests pass, all PASSED	0
All tests pass, only warn-on statuses	0 (with warnings)
Any test fails OR any fail-on status	1
Execution errors (network, timeout)	2

Pre-Push Hooks (Zero CI Config)

evalview install-hooks

Regression checks run before every git push. No CI pipeline needed — works purely locally.

GitLab CI

evalview:
  image: python:3.11
  script:
    - pip install evalview
    - evalview check --json --fail-on REGRESSION > results.json
  variables:
    OPENAI_API_KEY: $OPENAI_API_KEY
  artifacts:
    paths:
      - results.json

CircleCI

version: 2.1
jobs:
  evalview:
    docker:
      - image: python:3.11
    steps:
      - checkout
      - run: pip install evalview
      - run: evalview check --json --fail-on REGRESSION

Golden Traces — Setting up baselines
CLI Reference — All commands
Evaluation Metrics — Scoring system
YAML Schema — Test case format