「eval」の検索結果

Turn an eval study's numbers into on-brand, publish-ready figures using the Newsjack chart room (the eval design system), then validate them with Playwright. For producing the charts in a published eval/data study.

日本語の概要は準備中です。原文の説明を表示しています。

Design evaluation datasets and supporting documentation for AI systems and agents, covering the scoping interview, difficulty distribution, dataset contract, sample review, and metric and tooling selection. Use when building or reviewing an evaluation set for a conversational agent, assistant, or retrieval-grounded AI system.

日本語の概要は準備中です。原文の説明を表示しています。

Maintenance orchestrator for the brooks-lint plugin itself. Runs a sequential subagent pipeline — author → eval → QA → trigger-audit → release — to add or edit a skill, refresh the eval suite, keep the four manifests + all README translations + CHANGELOG + AGENTS/GEMINI in sync, audit trigger boundaries, and cut releases. Drives the five agents in .claude/agents/ (skill-author, eval-curator, consistency-qa, trigger-boundary-auditor, release-manager). Triggers when the maintainer asks to work ON brooks-lint itself: "add a new skill", "edit the brooks-debt guide", "update the eval suite", "fix the trigger descriptions", "make this change and validate it", "release brooks-lint", "bump and publish", and follow-ups: "re-run", "re-validate", "update that skill", "redo the audit", "do the X part again". Do NOT trigger for: USING the brooks-lint analysis skills on some target codebase (that's brooks-review / brooks-audit / brooks-debt / brooks-test / brooks-health / brooks-sweep); generic questions about brooks-lint that don't ask to change it; or maintenance of a different plugin.

日本語の概要は準備中です。原文の説明を表示しています。

Write code evaluators for known failure modes with objective rules. Use when code can check the rule from a trace, with or without a reference answer. Use `write-judge-prompt` when the rule requires interpretation.

日本語の概要は準備中です。原文の説明を表示しています。

Create diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation. Use when bootstrapping an eval dataset, when real user data is sparse, or when stress-testing specific failure hypotheses. Do NOT use when you already have 100+ representative real traces (use stratified sampling instead), or when the task is collecting production logs.

日本語の概要は準備中です。原文の説明を表示しています。

Run evaluations on Agent Skills to measure their effectiveness. USE FOR: "run skill evals", "evaluate my skill", "test skill quality", "check skill triggers", "skill compliance check", "measure skill performance", "run evals on [skill-name]", "grade skill execution". DO NOT USE FOR: writing skills (use skill-authoring), improving frontmatter (use sensei), or general testing unrelated to skills.

日本語の概要は準備中です。原文の説明を表示しています。

Use when evaluating the model and analyzing results for an Agricultural Systems (AgSy) manuscript so it survives expert systems review — independent model evaluation (observed vs. simulated, fit statistics), sensitivity and uncertainty analysis, and trade-off / scenario analysis across the system. Guides evaluation norms; it does not fabricate results or run the model.

日本語の概要は準備中です。原文の説明を表示しています。

Evaluate an AI workflow against representative cases and its permitted actions. Use for model, retrieval or agent evaluation; tests do not grant release authority.

日本語の概要は準備中です。原文の説明を表示しています。

Evaluate, select, and contract with vendors and SaaS tools. Use this skill when comparing alternatives, running an RFP, scoring vendors against criteria, negotiating contracts, planning a switch, or assessing a vendor's risk. Triggers on vendor evaluation, RFP, vendor selection, build vs buy, SaaS evaluation, vendor scorecard, vendor comparison, contract negotiation, vendor switch, procurement. Also triggers when a renewal is coming up or when a tool isn't meeting expectations.

日本語の概要は準備中です。原文の説明を表示しています。

Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.

日本語の概要は準備中です。原文の説明を表示しています。

Design RAG pipelines: chunking, retrieval evaluation, and architecture. Use when building a RAG system, selecting a chunking strategy, choosing a vector database, optimizing retrieval quality, or evaluating with RAGAS metrics.

日本語の概要は準備中です。原文の説明を表示しています。

This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality".

日本語の概要は準備中です。原文の説明を表示しています。

Use when the user wants to build an evaluation system for an LLM/agent application but doesn't know where to start — they have traces, prompts, RAG pipelines, or nothing at all. Also use when the user mentions evaluation, eval, benchmarking, testing LLM quality, measuring agent performance, assessing RAG accuracy, or wants to compare prompts/models. This skill is the entry router: it asks diagnostic questions then recommends which sub-skill (local workflow) to use next.

日本語の概要は準備中です。原文の説明を表示しています。

Design a source evaluation protocol using lateral reading and credibility checks for digital information. Use when students need to evaluate websites, online sources, or social media claims.

日本語の概要は準備中です。原文の説明を表示しています。

Post-run self-evaluation system that scores agent output on correctness, clarity, actionability, and conciseness. Use after /team runs, skill executions, or when explicitly asked to evaluate output quality.

日本語の概要は準備中です。原文の説明を表示しています。

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles

日本語の概要は準備中です。原文の説明を表示しています。

Enhancement-overlay SOP for adding sparse (BM25 / keyword) retrieval alongside dense (embedding) retrieval. Activate when a calling agent is building, reviewing, or debugging a retrieval pipeline whose corpus contains exact-match tokens — identifiers, error codes, SKUs, API/function names, proper nouns, citations, rare jargon — that pure dense embedding silently misses. Encodes the single decision rule (**hybrid is traffic-driven, not theoretical: add sparse only when the query share that depends on exact tokens is non-trivial**), the wiring of QueryFusionRetriever-style fusion (RRF vs alpha-weighted), and per-query-type alpha tuning. Frame the work as recovering lexical identity that dense pooling destroys, not as "add keyword search for completeness". Cross-links [[llamaindex]].

日本語の概要は準備中です。原文の説明を表示しています。

Builds an eval plan for an AI feature - error analysis on real traces, failure-mode ranking, code checks and binary LLM judges validated against human labels, CI tests, production monitoring, and a launch bar. Use when you ask how do we eval this, how do I test an AI feature, our LLM outputs are inconsistent, are the evals good enough, LLM-as-judge, hallucinations, or what accuracy is good enough to launch. Draws on 41 Lenny's Podcast guests incl. Hamel Husain, Shreya Shankar and Sander Schulhoff.

日本語の概要は準備中です。原文の説明を表示しています。

This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.

日本語の概要は準備中です。原文の説明を表示しています。

Build DSPy evaluation harnesses with rich-feedback metrics that are essential for GEPA optimization. Use when writing a metric function, calling dspy.Evaluate, splitting dev/val sets, debugging "why is my optimizer not improving?", or designing CI-ready DSPy eval suites.

日本語の概要は準備中です。原文の説明を表示しています。

Run a skill's evals and report results. Use when the user wants to evaluate a skill, run evals, benchmark a skill, validate triggers, optimize a description, or grade skill outputs.

日本語の概要は準備中です。原文の説明を表示しています。

Memory is the cornerstone of intelligent agents. Without it, every interaction starts from zero. This skill covers the architecture of agent memory: short-term (context window), long-term (vector stores), and the cognitive architectures that organize them. Key insight: Memory isn't just storage - it's retrieval. A million stored facts mean nothing if you can't find the right one. Chunking, embedding, and retrieval strategies determine whether your agent remembers or forgets. The field is fragmented with inconsistent terminology. We use the CoALA cognitive architecture framework: semantic memory (facts), episodic memory (experiences), and procedural memory (how-to knowledge). Use when "agent memory, long-term memory, memory systems, remember across sessions, memory retrieval, episodic memory, semantic memory, vector store, rag, langmem, memgpt, conversation history, memory, vector-store, rag, retrieval, embedding, episodic, semantic, procedural, langmem, memgpt, pinecone, qdrant, chromadb" mentioned.

日本語の概要は準備中です。原文の説明を表示しています。

Agent Evaluation workflow skill. Use this skill when the user needs Testing and benchmarking LLM agents including behavioral testing, and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.

日本語の概要は準備中です。原文の説明を表示しています。

Agent Evaluation workflow skill. Use this skill when the user needs Testing and benchmarking LLM agents including behavioral testing, and the operator should preserve the upstream workflow, copied support files, and provenance before merging or handing off.

日本語の概要は準備中です。原文の説明を表示しています。