Eval Reporter

You are generating evaluation reports from model benchmark results.

Your Goal

Given evaluation outputs and metrics:

1. Write structured metrics.json with all measurements
2. Write a beautiful HTML report directly (no template — you design the page)
3. Save everything to results/YYYY-MM-DD_modelname/

Workflow

Step 1: Collect Data

Gather from the previous phases:

• Model profile from research phase: name, URL, description, architecture, params, license, notable features
• Stage results: smoke test pass/fail, quality outputs, performance metrics
• Generated artifacts: images, audio, text files in artifacts/
• Timing and cost information
• Device info: GPU, VRAM, framework versions

Step 2: Write metrics.json

Run the metrics writer:

python .claude/skills/eval-reporter/scripts/metrics_writer.py \
  --output results/YYYY-MM-DD_modelname/metrics.json \
  --model MODEL_ID \
  --model-type MODEL_TYPE \
  --provider PROVIDER \
  --gpu GPU_NAME \
  --json-data '{"stages": {...}}'

Or construct the JSON directly following references/report-format.md.

Step 3: Write HTML Report

Write the HTML yourself. Do not use a template. You are an ML engineer writing a report for a technical audience. Design the page to be informative, honest, and beautiful.

Consult references/report-format.md for design guidelines (CSS palette, component examples).

Required sections:

1. Header — Model name (linked to HuggingFace page), type badge, date, provider/GPU
2. Model Overview — Architecture, parameter count, license, key features, 1-2 sentence description from model card
3. Execution Environment — GPU, VRAM, framework versions, provider
4. Smoke Test — Pass/fail, load time, any issues
5. Quality Results — Test inputs paired with outputs:
   • TTS: show input text above each <audio> player
   • Image gen: show prompt above each <img>
   • LLM: show prompt and response together
   • Always show what went IN and what came OUT
6. Performance — Key metrics table, comparison to published benchmarks if known
7. Conclusion — Your honest assessment: strengths, weaknesses, comparison to claimed performance, practical recommendations
8. Reproduction — Link to workspace/run.py, note the provider and cost

Design principles:

• Dark theme preferred (see references/report-format.md for CSS palette)
• Responsive layout, readable at any width
• Use semantic HTML — <table>, <audio>, <img>, not div soup
• Reference artifacts by relative path (not base64)

Step 4: Organize Result Directory

results/YYYY-MM-DD_modelname/
├── report.html          # Human-readable report (GitHub Pages ready)
├── metrics.json         # Structured data for comparison
├── artifacts/           # Generated outputs (images, audio, text)
│   ├── output_001.png
│   ├── output_002.wav
│   └── ...
└── workspace/           # Reproduction scripts (committed)
    └── run.py           # The inference script used

Step 5: Update Index Page

Regenerate the top-level index.html (report listing for GitHub Pages):

python .claude/skills/eval-reporter/scripts/generate_index.py

Important

• Consult references/report-format.md for the metrics.json schema and CSS design reference
• Images should be referenced by relative path in HTML (not base64, to keep file size small)
• Audio files: use <audio controls> tags in HTML
• Always include the run.py script for reproducibility
• Be honest in assessments — document failures and limitations clearly
• The report should be self-contained and understandable without reading metrics.json