Regression Test: Golden Dataset Management

Manage golden test datasets for LLM prompt regression testing using Promptfoo. Ensures prompt changes don't degrade output quality.

One eval tells you today's score. A kept set tells you what your change did. There is a recorded three-run walkthrough of exactly that, runnable with no API key, at https://github.com/7alexhale5-rgb/alexhale-skills/tree/main/example

Prerequisite: npm install -g promptfoo (check with which promptfoo)

---

Commands

/regression-test init

Scaffold regression testing infrastructure in the current project.

1. Create directory structure:

   mkdir -p .promptfoo/golden
   

2. Generate .promptfoo/promptfooconfig.yaml (the path matters: tests: "golden/*.yaml" below resolves relative to this file, and /regression-test run looks for it here):

   description: "Golden dataset regression tests"
   # Without this, promptfoo has no prompt to send and every case scores an empty string.
   # The vars come from the golden files that `/regression-test add` writes.
   prompts:
     - |
       [{"role":"system","content":"{{system_prompt}}"},{"role":"user","content":"{{user_input}}"}]
   providers:
     - id: anthropic:messages:claude-sonnet-5
       config:
         temperature: 0
   tests: "golden/*.yaml"
   defaultTest:
     assert:
       - type: llm-rubric
         value: "Output should match the expected behavior described in the test case"
   

3. Add .promptfoo/results/ to .gitignore if not already present. Ignore the results, commit the goldens: a golden dataset nobody else can see is not a shared baseline, and the whole point is that your teammates and your CI run the same cases you do. If a prompt in a golden case is genuinely sensitive, keep that one case out of the repo rather than hiding the dataset.

4. Report: "Initialized .promptfoo/golden/ — run /regression-test add <name> to capture test cases."

---

/regression-test add <name>

Capture a prompt+response pair as a golden test case.

1. Parse <name> from user input (slug format, e.g., auth-flow, error-handling)

2. Prompt user for:

   • System prompt (or detect from current project's CLAUDE.md/skill)
   • User input (the test prompt)
   • Expected behavior (natural language description of good output)
   • Vars (optional template variables)

3. Write golden test case:

   # .promptfoo/golden/{name}.yaml
   - description: "{name}"
     vars:
       system_prompt: |
         {system prompt content}
       user_input: |
         {user input}
     assert:
       - type: llm-rubric
         value: |
           {expected behavior description}
       - type: not-contains
         value: "error"
   

4. Report: "Added golden test {name} — {N} total test cases."

---

/regression-test run

Execute Promptfoo evaluation against all golden datasets.

1. Check prerequisites:

   which promptfoo 2>&1  # must return path
   ls .promptfoo/golden/*.yaml 2>/dev/null | wc -l  # must be > 0
   

2. Run evaluation:

   cd {project_root}
   promptfoo eval --config .promptfoo/promptfooconfig.yaml \
     --output .promptfoo/results/eval-$(date +%Y%m%d-%H%M%S).json \
     --no-cache 2>&1
   

3. Parse results JSON:

   • Count PASS/FAIL per test case
   • Extract failure details (expected vs actual)

4. Report:

   Regression Test Results
   ├─ Total: {N} test cases
   ├─ Pass: {N} ({%})
   ├─ Fail: {N} ({%})
   └─ Duration: {N}s
   
   Failures:
   - {test_name}: {assertion_type} — expected "{expected}", got "{actual}"
   

5. If all pass: "All golden tests pass — safe to ship." If failures: "Regressions detected — review before shipping."

---

/regression-test report

Show regression trends over the last 5 evaluations.

1. Read result files:

   ls -t .promptfoo/results/eval-*.json | head -5
   

2. For each result file, extract:

   • Date (from filename)
   • Pass/fail counts
   • Any new failures vs previous run

3. Report:

   Regression Trends (last 5 evals)
   ├─ 2026-03-31 14:30: 12/12 PASS ✓
   ├─ 2026-03-30 09:15: 11/12 PASS (1 fail: auth-flow)
   ├─ 2026-03-29 16:45: 12/12 PASS ✓
   ├─ 2026-03-28 11:00: 10/12 PASS (2 fail)
   └─ 2026-03-27 08:30: 12/12 PASS ✓
   
   Trend: STABLE (4/5 clean)
   

---

Where to wire it in

The value shows up when the run is automatic, not when you remember to type it.

Moment	What to do
After changing any prompt	/regression-test run before you commit. This is the one that catches most regressions.
Before a release	Run it in CI and fail the build on a new failure. promptfoo eval returns non-zero when an assertion fails.
After a model or version change	Re-run the whole set. A provider upgrade is a silent prompt change.
Weekly, on a schedule	/regression-test report to see whether quality is drifting rather than breaking.

If your agent harness supports hooks or a pre-commit step, put the run there. A check that depends on discipline is a check you will lose.

---

Graceful Degradation

Component	If Missing	Fallback
promptfoo CLI	Not installed	Prompt user: npm install -g promptfoo
.promptfoo/ directory	Not initialized	Prompt: /regression-test init
Golden datasets	No .yaml files	Skip with note: "No golden tests — add with /regression-test add"
Results directory	Missing	Create on first run