Frontier Harness Research — Comparative Research Process
Purpose: Before implementing a feature, compare relevant frontier harnesses and the original/upstream implementation, then establish design decision rationale for GEODE application.
When to apply: The user requests a comparison, or an external contract or
architectural choice needs evidence. Ordinary fixes using an established
local pattern do not require a multi-system survey.
Research Sources
| # | System | Type | Core Pattern Areas | GEODE Skill Reference |
|---|
| 1 | Claude Code | CLI agent | Permission model, Hook, Memory, Skill, Context management, UI | current official documentation |
| 2 | Codex | Coding agent | Sandbox execution, PR workflow, code review, multi-file editing | current official documentation and source |
| 3 | OpenClaw | Chat agent | Gateway, Session Key, Binding, Lane Queue, Plugin, Failover, 4-tier automation | openclaw-patterns |
| 4 | autoresearch | Autonomous experiment loop | Constraint-based design, ratchet, Context Budget, program.md, Simplicity Selection | karpathy-patterns |
| 5 | Prime Agent | RLM-native coding/research harness | Persistent REPL, programmable context, recursive subagents, continual harness state, native-harness evaluation | official source and technical report |
| 6 | DeepSeek Harness | Plugin-composed agent harness | Plugin scope and disposal, session/context ownership, provider and tool composition | official source |
| 7 | Google AX | Task orchestration runtime | Task/Workspace/Model declarations, controller/runner boundary, sandbox and suspend/resume ownership | official source; distinguish orchestration from the hosted agent loop |
| 8 | Grok Build | Coding agent harness and TUI | Layered config, session/model selection, compaction, tools, hooks and lifecycle | official source; distinguish source mirror from product release |
| 9 | Original / upstream implementation | Native authority | Task and scorer semantics, prompts, assets, workspace assumptions, reference results, reproduction path | pinned upstream source and primary docs |
The cited source inventory
owns dated version pins and comparison entry points. A source's inclusion is
not evidence of GEODE adoption or equivalent task performance.
Version follow-up and responsibility alignment
When revisiting a selected source, record the retrieval date, latest stable and
prerelease tags where available, and the default-branch commit separately. A
missing stable release is not proof that prereleases or product builds are absent.
Pin file links to the inspected commit; compare the previously recorded revision
with the new one before changing a claim.
Trace only the affected producer → field/state → consumer → decision, including
config precedence, admission, persistence, invalidation and teardown when relevant.
Match each upstream responsibility to GEODE's existing owner and record
adopt/adapt/reject with the regression that would establish the intended behavior.
Keep documented intent, inspected implementation and executed evidence distinct;
do not import upstream fallbacks or synthetic results as verified task success.
Store the delta in the existing task artifact, not a parallel status ledger.
Research Process
Step 1: Topic Definition
Define the feature to implement in one line and extract related keywords.
Example:
Topic: "Model Failover automation"
Keywords: failover, fallback, retry, circuit breaker, model switching
Step 2: Relevant-Source Pattern Exploration
Explore only systems relevant to the decision. Read applicable local skills for
source routing, then verify external claims against current primary docs or
pinned source. Recalled knowledge supplies search terms, not verification;
report unavailable evidence without inventing a capability.
2a. Claude Code Pattern Exploration
| Exploration Area | Checkpoints |
|---|
| Permission Model | allowlist/denylist, auto-approve, fallback after denial |
| Hook System | pre/post tool hooks, settings.json-based automation |
| Memory | CLAUDE.md, project memory, auto-memory |
| Skill System | skill discovery, trigger keywords, 4-tier priority |
| Context Management | sliding window, compression, token management |
| UI Patterns | status line, progress indicators, error display |
| Safety | HITL tiers, bash safety, dangerous tool gates |
2b. Codex Pattern Exploration
| Exploration Area | Checkpoints |
|---|
| Sandbox Execution | Isolated environment, filesystem restrictions, network restrictions |
| TDD Loop | test-first, red-green-refactor, automated verification |
| PR Workflow | Branch creation, change summary, review request |
| Multi-file Editing | Dependency tracking, consistency maintenance, refactoring scope |
| Task Decomposition | Complex task breakdown, sequential/parallel determination |
2c. OpenClaw Pattern Exploration (see openclaw-patterns skill)
| Exploration Area | Checkpoints |
|---|
| Gateway + Agent dual system | Control plane vs execution plane separation |
| Session Key hierarchy | agent:{id}:{context} format session isolation |
| Binding routing | Most-Specific Wins, static rules, hot reload |
| Lane Queue | Session/Global/Subagent Lane concurrency control |
| Sub-agent Spawn+Announce | Isolated execution, automatic result injection |
| 4-tier automation | Heartbeat, Cron, Internal Hooks, Gateway Hooks |
| Plugin architecture | Channel/Tool/Skill/Hook — 4 extension points |
| Policy Chain | 6-layer tool access control |
| Failover | Auth Rotation, Thinking Fallback, Context Overflow, Model Failover |
| Operational patterns | Coalescing, Atomic Store, Run Log, Hot Reload, Stuck Detection |
2d. autoresearch Pattern Exploration (see karpathy-patterns skill)
| Exploration Area | Checkpoints |
|---|
| P1 Constraint-based design | Define "what cannot be done" first |
| P2 Single-file constraint | Minimize modification surface area |
| P3 Fixed time budget | Limit by wall clock, not steps |
| P4 Ratchet mechanism | Keep only improvements, auto-revert on degradation |
| P5 Git as State Machine | Commit=experiment, reset=discard |
| P6 Context Budget | Redirect + selective extraction |
| P7 program.md | Agent behavior change = instruction document modification |
| P10 Simplicity Selection | Code deletion improvement > code addition improvement |
2e. Prime Agent Pattern Exploration
| Exploration Area | Checkpoints |
|---|
| Recursive Language Model | Persistent REPL, programmatic context access, subagent calls |
| Continual Harness | Prompt, subagent, skill, and memory CRUD from trajectory evidence |
| Long-horizon operation | Goal, heartbeat, autonomous continuation, explicit budgets and gates |
| Evaluation discipline | Same-model harness comparison, native-harness baseline, token and score accounting |
| Failure analysis | Reward hacking, verifier boundary, harness/model co-adaptation |
2f. Original / Upstream Implementation Exploration
| Exploration Area | Checkpoints |
|---|
| Native authority | Pinned repository/version, official docs, canonical entry point |
| Benchmark invariants | Prompt, task data, assets, workspace, tools, scorer/verifier, stop conditions |
| Reference evidence | Official result, run configuration, seeds/repeats, metric denominator |
| Adapter parity | Same agent/model/config on original and adapted paths; deviations documented |
| Provenance | Original source and adapter code remain distinguishable and auditable |
Step 3: GAP Analysis
Compare exploration results against GEODE's current state to identify GAPs.
Output format:
| # | Pattern | Source | GEODE Status | GAP | Priority |
|---|---------|--------|-------------|-----|----------|
| 1 | Model Failover | OpenClaw | ⚠️ Definition only | No auto-switching logic | P1 |
| 2 | Circuit Breaker | Codex | ✗ None | No blocking on consecutive failures | P1 |
| 3 | Retry Budget | autoresearch P3 | ⚠️ Partial | No time-based limit | P2 |
Step 4: Design Decisions
Select items to implement from GAP analysis results and document design decision rationale.
Decision Criteria:
| Criterion | Application |
|---|
| Pattern appears in several systems | Compare the failure it solves and whether GEODE has that failure; prevalence alone does not require adoption |
| Pattern appears in one system | Evaluate the same consumer, failure, and verification evidence; source count alone does not reject it |
| Over-engineering risk | → Apply Karpathy P10, implement minimally |
| Conflicts with existing GEODE patterns | → Existing pattern takes priority, gradual transition |
| Benchmark integration or platform adapter | → Preserve original semantics and require parity evidence before equivalence claims |
Step 5: Plan Document Writing
Record the decision in the existing task plan or PR. Use a docs/plans/ document
when durable research detail warrants one; do not create a second plan merely
to satisfy this skill.
# Plan: [Feature Name]
## Frontier Research Summary
| System | Related Pattern | Adoption | Rationale |
|--------|----------------|----------|-----------|
| Claude Code | ... | Adopt/Adapt/Reject | ... |
| Codex | ... | Adopt/Adapt/Reject | ... |
| OpenClaw | ... | Adopt/Adapt/Reject | ... |
| autoresearch | ... | Adopt/Adapt/Reject | ... |
| Prime Agent | ... | Adopt/Adapt/Reject | ... |
| Original / upstream | ... | Preserve/Adapt/N/A | ... |
## Design Decisions
...
## Implementation Phases
...
Research Checklist
For the selected research scope, verify:
Notes
- Ground the affected decision before implementation. If later evidence changes it, revise the plan and verify the affected behavior before proceeding.
- Not every frontier system is relevant. Mark irrelevant systems "N/A"; the original/upstream source is mandatory whenever one exists.
- Always read skill files first if they exist. The
openclaw-patterns and karpathy-patterns skills already contain distilled patterns, preventing redundant exploration.
- Prevent over-research: Stop once the decision is supported or the missing evidence is identified. Respect the task's time and cost budget.