Braintrust — AI Evaluation and Observability

Overview

Braintrust measures the quality of LLM applications. An evaluation has three parts — data (test cases with input and expected), a task (the function under test) and scores (functions that grade each output from 0 to 1). Every run is stored as an experiment that can be compared with earlier runs. The same SDK logs production traces, and the bt CLI runs eval files, queries logs and manages datasets from a terminal. The platform is a hosted service (free Starter plan, self-hosting on Enterprise); the SDKs (braintrust), the scorer library (autoevals) and the CLI are open source. Official SDKs also exist for Go, Ruby, Java and C#; this skill covers TypeScript and Python.

Instructions

Install

npm install braintrust autoevals@latest openai   # TypeScript SDK, scorers, provider client
npm install --save-dev @braintrust/bt tsx        # bt CLI plus a TypeScript runner
pip install braintrust autoevals openai          # Python 3.10+

Keep @latest on autoevals: 0.3.0 declares a pnpm-only engines field, so a bare npm install autoevals silently resolves to the older 0.0.132 (with @latest npm prints an EBADENGINE warning and installs 0.3.0). npx bt --version prints bt 0.22.1 or later. bt needs a runner for TypeScript files and picks the first it finds of tsx, vite-node, ts-node, deno; for Python it uses the active virtualenv.

Authenticate

Create a key under Settings > API keys in the Braintrust app and export it. The SDKs and bt read it automatically; nothing has to be passed in code.

export BRAINTRUST_API_KEY="..."         # from your secret manager, never committed
export OPENAI_API_KEY="..."             # only if the task or an LLM scorer calls OpenAI
npx bt status                           # shows the active org and project

For interactive use bt login stores a profile instead (bt login --oauth opens the browser).

Write and run an eval (TypeScript)

Name eval files *.eval.ts so bt eval discovers them. The first argument of Eval is the project name.

// evals/support-bot.eval.ts
import { Eval } from "braintrust";
import { Factuality, Levenshtein } from "autoevals";
import OpenAI from "openai";

const openai = new OpenAI();

async function answerQuestion(question: string): Promise<string> {
  const res = await openai.chat.completions.create({
    model: "gpt-5-mini",
    messages: [
      { role: "system", content: "You are the support assistant for Parcelwise, a shipping-label app." },
      { role: "user", content: question },
    ],
  });
  return res.choices[0].message.content ?? "";
}

// Custom scorer: receives one object, returns a number in [0, 1] or { name, score }
const mentionsPrice = ({ output, expected }: { output: string; expected?: string }) => ({
  name: "mentions_price",
  score: /\$\d+/.test(output) === /\$\d+/.test(expected ?? "") ? 1 : 0,
});

Eval("support-bot", {
  experimentName: "gpt-5-mini-baseline",
  data: () => [
    { input: "How do I reset my password?", expected: "Go to Settings > Security > Reset password" },
    { input: "What does the Team plan cost?", expected: "The Team plan is $49 per month" },
  ],
  task: answerQuestion,
  scores: [Factuality, Levenshtein, mentionsPrice],
  metadata: { model: "gpt-5-mini", prompt_version: "v2" },
  trialCount: 3,        // run each case 3 times and average
  maxConcurrency: 5,
});

npx bt eval evals/support-bot.eval.ts        # run one file, upload the experiment
npx bt eval evals/                           # every *.eval.ts (or eval_*.py) under evals/; one language per run
npx bt eval --watch evals/support-bot.eval.ts
npx bt eval --first 20 evals/                # smoke run on the first 20 cases, marked non-final
npx bt eval --no-send-logs evals/            # run locally, upload nothing
npx bt eval --list evals/                    # list evaluators without running them

bt does not load .env files by itself: export the variables or pass --env-file .env. For TypeScript files bt eval auto-instruments supported LLM clients, so each model call inside the task appears as a span in the experiment.

Write and run an eval (Python)

Name files eval_*.py. Custom scorers take input, output, expected as arguments.

# evals/eval_support_bot.py
from autoevals import Factuality, Levenshtein
from braintrust import Eval, Score
from openai import OpenAI

client = OpenAI()

def answer_question(question: str) -> str:
    res = client.chat.completions.create(
        model="gpt-5-mini",
        messages=[{"role": "user", "content": question}],
    )
    return res.choices[0].message.content or ""

def mentions_price(input, output, expected):
    return Score(name="mentions_price", score=1 if ("$" in output) == ("$" in expected) else 0)

Eval(
    "support-bot",
    experiment_name="gpt-5-mini-baseline",
    data=lambda: [
        {"input": "How do I reset my password?", "expected": "Go to Settings > Security > Reset password"},
        {"input": "What does the Team plan cost?", "expected": "The Team plan is $49 per month"},
    ],
    task=answer_question,
    scores=[Factuality, Levenshtein, mentions_price],
    metadata={"model": "gpt-5-mini"},
    trial_count=3,
    max_concurrency=5,
)

Run it with npx bt eval evals/eval_support_bot.py; --num-workers 4 sets the number of Python worker threads.

Scorers

autoevals ships ready-made scorers; import them and put them in scores.

Kind	Scorers
Deterministic	ExactMatch, Levenshtein, NumericDiff, JSONDiff, ValidJSON, ListContains
Embedding	EmbeddingSimilarity
Model-graded	Factuality, ClosedQA, Battle, Humor, Possible, Security, Sql, Summary, Translation, Moderation
RAG	AnswerRelevancy, AnswerCorrectness, AnswerSimilarity, ContextPrecision, ContextRecall, ContextRelevancy, ContextEntityRecall, Faithfulness

For a rubric of your own, build an LLM judge: LLMClassifierFromTemplate({ name, promptTemplate, choiceScores, useCoT }) in TypeScript, LLMClassifier(name=..., prompt_template=..., choice_scores=..., model=...) in Python. The template can reference {{input}}, {{output}} and {{expected}}.

Datasets

Keep test cases in Braintrust instead of in code so the team can edit them and experiments can be compared row by row.

// scripts/seed-dataset.ts — run once with: npx tsx scripts/seed-dataset.ts
import { initDataset } from "braintrust";

async function main() {
  const dataset = initDataset("support-bot", { dataset: "billing-questions" }); // created if missing
  dataset.insert({
    input: "Can I get a refund after 30 days?",
    expected: "Refunds are available within 30 days of purchase",
    metadata: { category: "billing" },
  });
  await dataset.flush();
}
main();

In the eval file, replace the inline array with the dataset: data: initDataset("support-bot", { dataset: "billing-questions" }). Python equivalent: braintrust.init_dataset(project="support-bot", name="billing-questions"), then dataset.insert(input=..., expected=..., metadata=...) and dataset.flush().

Trace production calls

// src/tracing.ts — call once at startup, before any LLM call
import { initLogger, wrapOpenAI, wrapTraced } from "braintrust";
import OpenAI from "openai";

export const logger = initLogger({ projectName: "support-bot" });
export const openai = wrapOpenAI(new OpenAI());       // every call is logged with tokens and latency

export const retrieveArticles = wrapTraced(async function retrieveArticles(query: string) {
  return searchHelpCenter(query);                      // arguments and return value become a span
});

On Node.js the wrapper can be skipped by starting the process with node --import braintrust/hook.mjs dist/server.js, which patches supported provider SDKs at load time.

In Python call braintrust.init_logger(project="support-bot") and then braintrust.auto_instrument() before any provider client is created, and decorate your own functions with @traced (Example 2 shows the full file). wrap_openai(OpenAI()) is the manual alternative to auto-instrumentation.

Run evals in CI

# .github/workflows/evals.yml
on: pull_request
permissions:
  pull-requests: write
  contents: read
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npm ci
      - uses: braintrustdata/eval-action@v2
        with:
          api_key: ${{ secrets.BRAINTRUST_API_KEY }}
          runtime: node          # or python, go
          paths: evals

The action posts a score summary as a pull-request comment (the permissions block is required for that) and, with its default use_proxy: true, sends LLM calls through the Braintrust proxy, which caches repeated requests. On other CI systems run npx bt eval evals/ --no-input --json with BRAINTRUST_API_KEY set.

Examples

Example 1: Check a prompt change before merging

User request: "I changed the system prompt of our support bot. Tell me if answers got worse, and don't upload anything yet."

Run the existing eval file locally and read the scores as JSON:

npx bt eval --no-send-logs --jsonl evals/support-bot.eval.ts > scores.jsonl
jq -c '.scores | map_values(.score)' scores.jsonl

{"Factuality":0.8,"Levenshtein":0.75,"mentions_price":1.0}

To make the same check fail a pipeline, add a threshold: jq -e '.scores.Levenshtein.score >= 0.7' scores.jsonl exits 1 when the score is lower. Once the numbers look right, drop --no-send-logs; the run is uploaded as an experiment and the summary prints a link where it can be compared with earlier experiments.

Example 2: Add tracing to a Python RAG service

User request: "Our FastAPI app answers questions from our docs with OpenAI. I want every request in Braintrust, with the retrieval step visible."

# app/main.py
import braintrust
from braintrust import traced

braintrust.init_logger(project="docs-assistant")
braintrust.auto_instrument()

from fastapi import FastAPI
from openai import OpenAI

app = FastAPI()
client = OpenAI()

@traced
def retrieve(question: str) -> list[str]:
    return vector_store.search(question, limit=4)

@traced
def answer(question: str) -> str:
    context = "\n\n".join(retrieve(question))
    res = client.chat.completions.create(
        model="gpt-5-mini",
        messages=[
            {"role": "system", "content": f"Answer from this context only:\n{context}"},
            {"role": "user", "content": question},
        ],
    )
    return res.choices[0].message.content or ""

@app.post("/ask")
def ask(question: str) -> dict:
    return {"answer": answer(question)}

Each request to /ask now appears under Logs in the docs-assistant project as one trace with three spans: answer, a nested retrieve span holding the question and the returned chunks, and the OpenAI call with its prompt, completion, token counts and latency.

Guidelines

• bt eval exits non-zero only when an eval throws. A low score does not fail the build by itself — gate on the --jsonl output as in Example 1. braintrustdata/eval-action posts the scores on the pull request for review.
• A custom scorer in TypeScript takes a single object ({ input, output, expected, metadata }); in Python it takes those names as arguments. Return a number between 0 and 1, or an object with name and score.
• LLM-judge scorers make model calls of their own, so they cost tokens and are not deterministic. Use trialCount / trial_count for noisy tasks and prefer deterministic scorers where an exact check exists.
• autoevals before 0.3.0 sends OPENAI_API_KEY to the Braintrust endpoint when both keys are set. If a scorer fails with 401 Braintrust gateway error, upgrade autoevals or give it an explicit client: init({ client: new OpenAI({ apiKey: process.env.BRAINTRUST_API_KEY, baseURL: "https://gateway.braintrust.dev" }) }).
• Call initLogger / init_logger once at startup. In serverless functions and short scripts, flush before exit (await logger.flush()), or the last spans are lost.
• Traces contain prompts and completions. Do not log secrets or personal data you may not send to a third party; the SDKs support a masking function for redaction, and Enterprise plans can self-host the data plane.
• Keep keys in environment variables or CI secrets. The Python SDK also reads BRAINTRUST_API_KEY from a .env.braintrust file — add it to .gitignore.
• The older npx braintrust eval and Python braintrust eval commands still ship with the SDKs, but the documentation now uses bt eval; bt eval is not available on Windows, so use WSL there.
• Starter-plan limits apply to stored scores and ingested data (10k scores per month at the time of writing); run large sweeps with --first or --sample first.