208 件 ・ 関連度順
概要と使いどころ
Provides context about the Roo Code evals system structure in this monorepo. Use when tasks mention "evals", "evaluation", "eval runs", "eval exercises", or working with the evals infrastructure. Helps distinguish between the evals execution system (packages/evals, apps/web-evals) and the public website evals display page (apps/web-roo-code/src/app/evals).
日本語の概要は準備中です。原文の説明を表示しています。
OpenAI API (developers.openai.com) の Evals / Fine-tuning リファレンス。 Evals API, Datasets, graders（string check, text similarity, model, python, multi）, agent evals, trace grading, external models、 SFT, DPO, RFT（reinforcement fine-tuning）, vision fine-tuning, distillation, prompt optimizer, fine-tuning best practices。 セルフサービス fine-tuning は段階的廃止進行中。
Entry point for evals. Use when the user asks for help with evals, does not know where to begin, or asks for something no other skill in this plugin matches. Do NOT use when a more specific skill in this plugin already matches; load that skill directly.
日本語の概要は準備中です。原文の説明を表示しています。
DeepEval evaluation workflow for AI agents and LLM applications. TRIGGER when the user wants to evaluate or improve an AI agent, tool-using workflow, multi-turn chatbot, RAG pipeline, or LLM app; add evals; generate datasets or goldens; use deepeval generate; use deepeval test run; send results to Confident AI; monitor production; run online evals; inspect traces; or iterate on prompts, tools, retrieval, or agent behavior from eval failures. AI agents are the primary use case. Covers Python SDK, pytest eval suites, CLI generation, traced evals, Confident AI reporting, and agent-driven improvement loops. DO NOT TRIGGER for unrelated generic pytest, non-AI test setup, or non-DeepEval observability work unless the user asks to compare or migrate to DeepEval; for instrumenting an app with DeepEval tracing, @observe, or framework integrations (use the `deepeval-tracing` skill); or for raw OpenTelemetry / OTLP export without the deepeval package (use the `deepeval-otel` skill).
日本語の概要は準備中です。原文の説明を表示しています。
Designs coding-agent observability and evals. Use when measuring traces, replay, checkpoint lineage, tool grading, regression, or cost. Not for general LLM evals (ai-evals).
日本語の概要は準備中です。原文の説明を表示しています。
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.
日本語の概要は準備中です。原文の説明を表示しています。
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.
日本語の概要は準備中です。原文の説明を表示しています。
Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).
日本語の概要は準備中です。原文の説明を表示しています。
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.
日本語の概要は準備中です。原文の説明を表示しています。
Hamel Husain (@HamelHusain) 视角. 服务咨询派 + practitioner-as-figure 代表, 「evals are the new code」论断的提出者. ex-Airbnb / GitHub principal eng → 2017 起做 independent ML/AI consultant (parlance-labs) → 2023+ 转向 AI evals 专精. hamel.dev 长文 corpus (40+ 篇 1500+ words technical) + Maven「AI Evals for Engineers」课程 (3000+ paid 学员, 与 Shreya Shankar 共同主理). 在 monetize-agents 行业里代表「不规模化 / 不 productize 成 SaaS / 不融资」的第三条路 — independent consultant + course creator. 用途: 当用户面临「AI agent 上线不可靠 / 客户卡在 prompt 调不动 / 该不该雇团队 / 该不该做 SaaS / 怎样把 expertise productize 成课程而非公司」类问题时, 切换到这副镜片.
日本語の概要は準備中です。原文の説明を表示しています。
Design automated evaluation pipelines for LLM and agent systems — combining deterministic checks, statistical metrics, and LLM-as-judge scoring into repeatable, CI-integrated eval suites. Load when the user asks to set up automated evals, design an eval pipeline, integrate evals into CI/CD, create an eval suite, do eval-driven development, or says "automate my evals", "CI eval integration", "evaluation pipeline", "continuous evaluation", "monitoring eval quality", "set up regression testing for my agent". Sub-skill of eval-output orchestrator.
日本語の概要は準備中です。原文の説明を表示しています。
AIエージェントが継続して作業するループを、品質確認と評価、失敗時の停止・再実行の方針を組み合わせて設計し、進捗停滞や再試行に対処するスキル。
Bootstrap a new realtime eval folder inside this cookbook repo by choosing the right harness from examples/evals/realtime_evals, scaffolding prompt/tools/data files, generating a useful README, and validating it with smoke, full eval, and test runs. Use when a user wants to start a new crawl, walk, or run realtime eval in this repository.
日本語の概要は準備中です。原文の説明を表示しています。
Assertion-first AI eval framework aligned to Anthropic's 'Demystifying evals for AI agents' — typed deterministic asserts + a forced-structured LLM judge over an input→assert case schema, pass^k/pass@k, capability vs regression suites, subscription-billed. USE WHEN eval, evaluate, benchmark, regression test, assertion, assert, llm-rubric, judge, pass@k, pass^k, grade output, compare prompts/models, test agent. NOT FOR scientific-method framing (use Science), property/mutation testing of code (use Hardening), or live UI verification (use Interceptor).
日本語の概要は準備中です。原文の説明を表示しています。
Scaffolds evaluation suites for the Axiom AI SDK. Generates eval files, scorers, flag schemas, and config from natural-language descriptions. Use when creating evals, writing scorers, setting up flag schemas, or configuring axiom.config.ts.
日本語の概要は準備中です。原文の説明を表示しています。
do e2e tests, run e2e, validate feature, prove it works, PR proof, frame proof, pnpm evals. Launches iPolloWork on Daytona or local Electron and runs the coded eval flows via CDP. Launch + run mechanics; the proof loop itself is the fraimz skill.
日本語の概要は準備中です。原文の説明を表示しています。
Authoring and running behavioral evals for a shopping or merchant agent, covering the case shape, authoring rules, code graders and judges, the run pattern, and poisoned fixtures. Load when writing eval cases or rubrics or deciding how a suite runs.
日本語の概要は準備中です。原文の説明を表示しています。
Turn student course evaluations (free-text + numeric) into an actionable teaching-improvement plan — the teaching analogue of /respond-to-referees. Clusters comments into themes, separates signal from noise, classifies each theme Keep / Change / Investigate / Out-of-scope, and drafts concrete changes mapped to the syllabus and slide decks. Use when user says "respond to my evals", "what do these course evaluations tell me", "turn my teaching feedback into a plan", or after a semester's evals arrive.
日本語の概要は準備中です。原文の説明を表示しています。
Design LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle. Use when a failure mode requires interpretation (tone, faithfulness, relevance, completeness). Do NOT use when the failure mode can be checked with code (regex, schema validation, execution tests); use `write-code-eval`. To validate an existing judge, use `validate-evaluator`.
日本語の概要は準備中です。原文の説明を表示しています。
Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.
日本語の概要は準備中です。原文の説明を表示しています。
Write code evaluators for known failure modes with objective rules. Use when code can check the rule from a trace, with or without a reference answer. Use `write-judge-prompt` when the rule requires interpretation.
日本語の概要は準備中です。原文の説明を表示しています。
Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an eval dataset, when real user data is sparse, or when stress-testing specific failure hypotheses. Do NOT use when you already have 100+ representative real traces (use stratified sampling instead), or when the task is collecting production logs.
日本語の概要は準備中です。原文の説明を表示しています。
Build a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback. Use when you need to build an annotation tool, review traces, or collect human labels.
日本語の概要は準備中です。原文の説明を表示しています。
Calibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs. Do NOT use for code-based evaluators (those are deterministic; test with unit tests per `write-code-eval`).
日本語の概要は準備中です。原文の説明を表示しています。