dotnet-ci-benchmarking

Continuous benchmarking guidance for detecting performance regressions in CI pipelines. Covers baseline file management with BenchmarkDotNet JSON exporters, GitHub Actions workflows for artifact-based baseline comparison, regression detection patterns with configurable thresholds, and alerting strategies for performance degradation.

Version assumptions: BenchmarkDotNet v0.14+ for JSON export, GitHub Actions runner environment. Examples use actions/upload-artifact@v4 and actions/download-artifact@v4.

Scope

• Baseline file management with BenchmarkDotNet JSON exporters
• GitHub Actions workflows for artifact-based baseline comparison
• Regression detection with configurable thresholds
• Alerting strategies for performance degradation

Out of scope

• BenchmarkDotNet setup and benchmark class design -- see [skill:dotnet-benchmarkdotnet]
• Performance architecture patterns -- see [skill:dotnet-performance-patterns]
• Profiling tools (dotnet-counters, dotnet-trace, dotnet-dump) -- see [skill:dotnet-profiling]
• OpenTelemetry metrics and distributed tracing -- see [skill:dotnet-observability]
• Composable CI/CD workflow design -- see [skill:dotnet-gha-patterns]

Cross-references: [skill:dotnet-benchmarkdotnet] for benchmark class setup and JSON exporter configuration, [skill:dotnet-observability] for correlating benchmark regressions with runtime metrics changes, [skill:dotnet-gha-patterns] for composable workflow patterns (reusable workflows, composite actions, matrix builds).

---

Baseline File Management

BenchmarkDotNet JSON Export

BenchmarkDotNet's JSON exporter produces machine-readable results for automated comparison. Configure the exporter in benchmark classes:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Exporters.Json;

[JsonExporterAttribute.Full]
[MemoryDiagnoser]
public class CriticalPathBenchmarks
{
    [Benchmark(Baseline = true)]
    public void ProcessOrder() { /* ... */ }

    [Benchmark]
    public void ProcessOrderOptimized() { /* ... */ }
}

```text

Or configure via custom config for all benchmark classes:

```csharp

using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Exporters.Json;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;

var config = ManualConfig.Create(DefaultConfig.Instance)
    .AddJob(Job.ShortRun)  // fewer iterations for CI speed
    .AddExporter(JsonExporter.Full)
    .WithArtifactsPath("./benchmark-results");

BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, config);

```json

### JSON Export Structure

The exported JSON file (`*-report-full.json`) contains structured benchmark results:

```json

{
  "Title": "CriticalPathBenchmarks",
  "Benchmarks": [
    {
      "FullName": "MyApp.Benchmarks.CriticalPathBenchmarks.ProcessOrder",
      "Statistics": {
        "Mean": 1234.5678,
        "Median": 1230.1234,
        "StandardDeviation": 15.234,
        "StandardError": 4.812
      },
      "Memory": {
        "BytesAllocatedPerOperation": 1024,
        "Gen0Collections": 0.0012,
        "Gen1Collections": 0,
        "Gen2Collections": 0
      }
    }
  ]
}

```text

Key fields for regression comparison:

| Field                               | Purpose                                         |
| ----------------------------------- | ----------------------------------------------- |
| `Statistics.Mean`                   | Average execution time (nanoseconds)            |
| `Statistics.Median`                 | Middle execution time (more robust to outliers) |
| `Statistics.StandardDeviation`      | Measurement variability                         |
| `Memory.BytesAllocatedPerOperation` | GC allocation per operation                     |

### Baseline Storage Strategies

| Strategy                         | Pros                                   | Cons                                                            | Best For                                |
| -------------------------------- | -------------------------------------- | --------------------------------------------------------------- | --------------------------------------- |
| Git-committed baseline file      | Versioned, auditable, no external deps | Repo size grows; must update deliberately                       | Small benchmark suites, stable hardware |
| GitHub Actions artifacts         | No repo bloat; automatic retention     | 90-day default retention; cross-workflow access requires tokens | Large benchmark suites, shared runners  |
| External storage (S3/Azure Blob) | Unlimited history; cross-repo sharing  | Extra infrastructure; credential management                     | Multi-repo benchmark comparison         |

This skill focuses on the **GitHub Actions artifact** strategy as the default. For composable workflow patterns and
reusable actions, see [skill:dotnet-gha-patterns].

---

## GitHub Actions Benchmark Workflow

### Basic Benchmark Workflow

```yaml

name: Benchmarks

on:
  pull_request:
    paths:
      - 'src/**'
      - 'benchmarks/**'
  workflow_dispatch:

permissions:
  contents: read
  actions: read # required for artifact download

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '8.0.x'

      - name: Run benchmarks
        run: dotnet run -c Release --project benchmarks/MyBenchmarks.csproj -- --exporters json

      - name: Upload benchmark results
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-results-${{ github.sha }}
          path: benchmarks/BenchmarkDotNet.Artifacts/results/
          retention-days: 90

```text

### Baseline Comparison Workflow

This workflow downloads the baseline from a previous run and compares against current results:

```yaml

name: Benchmark Regression Check

on:
  pull_request:
    paths:
      - 'src/**'
      - 'benchmarks/**'

permissions:
  contents: read
  actions: read

env:
  BENCHMARK_PROJECT: benchmarks/MyBenchmarks.csproj
  RESULTS_DIR: benchmarks/BenchmarkDotNet.Artifacts/results

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '8.0.x'

      - name: Download baseline results
        uses: actions/download-artifact@v4
        with:
          name: benchmark-baseline
          path: ./baseline-results
        continue-on-error: true
        id: download-baseline

      - name: Run benchmarks
        run: dotnet run -c Release --project ${{ env.BENCHMARK_PROJECT }} -- --exporters json

      - name: Compare with baseline
        if: steps.download-baseline.outcome == 'success'
        shell: bash
        run: |
          set -euo pipefail
          python3 scripts/compare-benchmarks.py \
            --baseline ./baseline-results \
            --current "${{ env.RESULTS_DIR }}" \
            --threshold 10 \
            --output benchmark-comparison.md

      - name: Upload current results as new baseline
        if: github.ref == 'refs/heads/main'
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-baseline
          path: ${{ env.RESULTS_DIR }}/
          retention-days: 90
          overwrite: true

      - name: Upload comparison report
        if: steps.download-baseline.outcome == 'success'
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-comparison-${{ github.sha }}
          path: benchmark-comparison.md
          retention-days: 30

```markdown

**Key design decisions:**

- `continue-on-error: true` on baseline download handles first-run (no baseline exists yet)
- Baseline is only updated from `main` branch merges to prevent PR branches from polluting the baseline
- `overwrite: true` replaces the previous baseline artifact

For converting these inline workflows into reusable `workflow_call` patterns, see [skill:dotnet-gha-patterns].

---

## Regression Detection Patterns

### Threshold-Based Comparison

Compare current benchmark results against baseline using percentage thresholds. A regression is flagged when the current
mean exceeds the baseline mean by more than the configured threshold:

```python

#!/usr/bin/env python3
"""compare-benchmarks.py -- Detect benchmark regressions from BenchmarkDotNet JSON exports."""

import json
import sys
from pathlib import Path

def load_benchmarks(results_dir: str) -> dict:
    """Load benchmark results from BenchmarkDotNet JSON export files."""
    benchmarks = {}
    for json_file in Path(results_dir).glob("*-report-full.json"):
        with open(json_file) as f:
            data = json.load(f)
        for bm in data.get("Benchmarks", []):
            name = bm["FullName"]
            benchmarks[name] = {
                "mean": bm["Statistics"]["Mean"],
                "median": bm["Statistics"]["Median"],
                "stddev": bm["Statistics"]["StandardDeviation"],
                "allocated": bm.get("Memory", {}).get("BytesAllocatedPerOperation", 0),
            }
    return benchmarks

def compare(baseline_dir: str, current_dir: str, threshold_pct: float) -> list:
    """Compare current results against baseline. Returns list of regressions."""
    baseline = load_benchmarks(baseline_dir)
    current = load_benchmarks(current_dir)
    regressions = []

    for name, curr in current.items():
        if name not in baseline:
            continue  # new benchmark, no comparison possible
        base = baseline[name]
        if base["mean"] == 0:
            continue  # avoid division by zero

        time_change_pct = ((curr["mean"] - base["mean"]) / base["mean"]) * 100
        alloc_change = curr["allocated"] - base["allocated"]

        if time_change_pct > threshold_pct:
            regressions.append({
                "name": name,
                "baseline_mean": base["mean"],
                "current_mean": curr["mean"],
                "change_pct": time_change_pct,
                "alloc_change": alloc_change,
            })

    return regressions

if __name__ == "__main__":
    import argparse

    parser = argparse.ArgumentParser(description="Compare BenchmarkDotNet results")
    parser.add_argument("--baseline", required=True, help="Path to baseline results directory")
    parser.add_argument("--current", required=True, help="Path to current results directory")
    parser.add_argument("--threshold", type=float, default=10.0,
                        help="Regression threshold percentage (default: 10)")
    parser.add_argument("--output", default="comparison.md", help="Output markdown file")
    args = parser.parse_args()

    regressions = compare(args.baseline, args.current, args.threshold)

    with open(args.output, "w") as f:
        if regressions:
            f.write("## Benchmark Regressions Detected\n\n")
            f.write("| Benchmark | Baseline (ns) | Current (ns) | Change | Alloc Delta |\n")
            f.write("|-----------|--------------|-------------|--------|-------------|\n")
            for r in regressions:
                f.write(f"| `{r['name']}` | {r['baseline_mean']:.2f} | "
                        f"{r['current_mean']:.2f} | +{r['change_pct']:.1f}% | "
                        f"{r['alloc_change']:+d} B |\n")
            f.write(f"\nThreshold: {args.threshold}%\n")
        else:
            f.write("## Benchmark Results\n\nNo regressions detected ")
            f.write(f"(threshold: {args.threshold}%).\n")

    if regressions:
        print(f"REGRESSION: {len(regressions)} benchmark(s) exceeded "
              f"{args.threshold}% threshold", file=sys.stderr)
        sys.exit(1)

```text

### Choosing Thresholds

| Environment                   | Suggested Threshold | Rationale                                                    |
| ----------------------------- | ------------------- | ------------------------------------------------------------ |
| Dedicated benchmark hardware  | 5%                  | Low noise floor; small regressions are signal                |
| GitHub Actions shared runners | 10-15%              | Shared runners introduce 5-10% variance from noisy neighbors |
| Self-hosted runners           | 5-10%               | More stable than shared, but still monitor variance          |

**Calibrate thresholds empirically:** Run the same benchmark suite 5-10 times on your CI environment without code
changes. The maximum observed variance sets your noise floor. Set the threshold above this noise floor (typically 2x the
observed variance).

### Allocation Regression Detection

Memory allocation regressions are more reliable signals than timing regressions because allocations are deterministic
(not affected by noisy neighbors):

```python

# Add to the compare function:
if alloc_change > 0:
    regressions.append({
        "name": name,
        "type": "allocation",
        "baseline_alloc": base["allocated"],
        "current_alloc": curr["allocated"],
        "alloc_change": alloc_change,
    })

```text

Use allocation changes as a **hard gate** (zero tolerance for new allocations in zero-alloc paths) and timing changes as
a **soft gate** (warning with threshold).

---

## Alerting Strategies

### PR Comment with Regression Summary

Post benchmark comparison results as a PR comment for reviewer visibility:

```yaml

- name: Comment PR with results
  if: steps.download-baseline.outcome == 'success' && github.event_name == 'pull_request'
  uses: actions/github-script@v7
  with:
    script: |
      const fs = require('fs');
      const body = fs.readFileSync('benchmark-comparison.md', 'utf8');
      await github.rest.issues.createComment({
        owner: context.repo.owner,
        repo: context.repo.repo,
        issue_number: context.issue.number,
        body: body
      });

```text

### Fail the Build on Regression

Exit with non-zero status from the comparison script to fail the GitHub Actions job. This prevents merging PRs that
introduce performance regressions:

```yaml

- name: Check for regressions
  if: steps.download-baseline.outcome == 'success'
  shell: bash
  run: |
    set -euo pipefail
    python3 scripts/compare-benchmarks.py \
      --baseline ./baseline-results \
      --current "${{ env.RESULTS_DIR }}" \
      --threshold 10
    # Script exits non-zero if regressions found -- fails the job

```text

For required status checks and branch protection integration with benchmark gates, see [skill:dotnet-gha-patterns].

### Trend Tracking

For long-term trend analysis beyond single-PR comparison, upload results to a persistent store and track metrics over
time:

| Approach                           | Tool                                          | Complexity                            |
| ---------------------------------- | --------------------------------------------- | ------------------------------------- |
| GitHub Actions artifacts           | Built-in, 90-day retention                    | Low -- artifact download/upload only  |
| GitHub Pages with benchmark-action | `benchmark-action/github-action-benchmark@v1` | Medium -- auto-generates trend charts |
| External time-series DB            | InfluxDB, Prometheus + Grafana                | High -- full observability stack      |

The simplest approach for most projects is the artifact-based baseline comparison shown in this skill. Graduate to trend
tracking when you need historical regression analysis across many releases.

---

## CI-Specific BenchmarkDotNet Configuration

### ShortRun for CI Speed

Full benchmark runs take 10-30+ minutes. Use `Job.ShortRun` in CI to reduce iteration counts while retaining regression
detection capability:

```csharp

using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;

public class CiConfig : ManualConfig
{
    public CiConfig()
    {
        AddJob(Job.ShortRun
            .WithWarmupCount(3)
            .WithIterationCount(5)
            .WithInvocationCount(1));

        AddExporter(BenchmarkDotNet.Exporters.Json.JsonExporter.Full);
    }
}

```json

Apply conditionally based on environment:

```csharp

var config = Environment.GetEnvironmentVariable("CI") is not null
    ? new CiConfig()
    : DefaultConfig.Instance;

BenchmarkRunner.Run<CriticalPathBenchmarks>(config);

```text

### Filtering Benchmarks for CI

Run only critical-path benchmarks in CI to reduce pipeline duration:

```bash

# Run only benchmarks in the "Critical" category
dotnet run -c Release --project benchmarks/MyBenchmarks.csproj -- \
  --filter *Critical* --exporters json

```bash

```csharp

[BenchmarkCategory("Critical")]
[MemoryDiagnoser]
[JsonExporterAttribute.Full]
public class CriticalPathBenchmarks
{
    [Benchmark]
    public void ProcessOrder() { /* ... */ }
}

[BenchmarkCategory("Extended")]
[MemoryDiagnoser]
public class ExtendedBenchmarks
{
    [Benchmark]
    public void RareCodePath() { /* ... */ }
}

```text

Run `Critical` benchmarks on every PR; run `Extended` benchmarks on a nightly schedule.

### Nightly Benchmark Schedule

```yaml

name: Nightly Benchmarks (Full Suite)

on:
  schedule:
    - cron: '0 3 * * *' # 3 AM UTC daily
  workflow_dispatch:

jobs:
  benchmark-full:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '8.0.x'

      - name: Run full benchmark suite
        run: dotnet run -c Release --project benchmarks/MyBenchmarks.csproj -- --exporters json
        # No --filter: runs all benchmarks including Extended category

      - name: Upload full results
        uses: actions/upload-artifact@v4
        with:
          name: benchmark-full-${{ github.run_number }}
          path: benchmarks/BenchmarkDotNet.Artifacts/results/
          retention-days: 90

```text

For scheduled workflow patterns and matrix builds across TFMs, see [skill:dotnet-gha-patterns].

---



## Code Navigation (Serena MCP)

**Primary approach:** Use Serena symbol operations for efficient code navigation:

1. **Find definitions**: `serena_find_symbol` instead of text search
2. **Understand structure**: `serena_get_symbols_overview` for file organization
3. **Track references**: `serena_find_referencing_symbols` for impact analysis
4. **Precise edits**: `serena_replace_symbol_body` for clean modifications

**When to use Serena vs traditional tools:**
- ✅ **Use Serena**: Navigation, refactoring, dependency analysis, precise edits
- ✅ **Use Read/Grep**: Reading full files, pattern matching, simple text operations
- ✅ **Fallback**: If Serena unavailable, traditional tools work fine

**Example workflow:**
```text
# Instead of:
Read: src/Services/OrderService.cs
Grep: "public void ProcessOrder"

# Use:
serena_find_symbol: "OrderService/ProcessOrder"
serena_get_symbols_overview: "src/Services/OrderService.cs"
```
## Agent Gotchas

1. **Use `Job.ShortRun` in CI, not `Job.Default`** -- default benchmark jobs run many iterations for statistical
   precision, taking 10-30+ minutes per benchmark class. CI pipelines need faster feedback with `ShortRun` (3 warmup, 5
   iteration).
2. **Set threshold above measured noise floor** -- shared CI runners introduce 5-10% timing variance from noisy
   neighbors. A 5% threshold on shared runners produces false positives. Calibrate by running the same code multiple
   times and measuring variance.
3. **Use allocation changes as hard gates** -- allocation counts are deterministic and unaffected by runner noise. A
   zero-to-nonzero allocation change is always a real regression, unlike timing variations.
4. **Only update baselines from main branch** -- if PR branches can update the baseline, a regression in one PR becomes
   the new baseline, masking it from subsequent comparisons.
5. **Always set `set -euo pipefail` in bash steps** -- without `pipefail`, a regression detection script that exits
   non-zero in a pipeline (e.g., `script | tee`) does not fail the GitHub Actions step.
6. **Handle missing baselines gracefully** -- the first CI run has no baseline to compare against. Use
   `continue-on-error: true` on the baseline download step and skip comparison when no baseline exists.
7. **Export JSON, not just Markdown** -- Markdown reports are human-readable but not machine-parseable for automated
   regression detection. Always include `[JsonExporterAttribute.Full]` or `JsonExporter.Full` in the config.