How to Implement and Evaluate a Low-Effort Model Routing Policy

TL;DR

• A useful Low-effort router needs four separate decisions: task class, model, execution boundary, and acceptance gate.
• Do not ask a model to choose the model for every request. Start with explicit task metadata such as routine, normal, or ambiguous; make the policy deterministic and inspectable.
• A practical starting policy is Sonnet 5 Low for narrow work, Opus 4.8 Low for normal work only after it passes your repo eval, and Fable 5 Low for ambiguity or expensive restart risk.
• The Claude Agent SDK exposes model, effort, maxTurns, maxBudgetUsd, cwd, tool configuration, and permission mode per run.
• Keep planning runs read-only. Run coding agents in disposable worktrees or sandboxes, remove Bash from the model’s tools, and execute allowlisted tests from the controller—not from model output.
• Compare the router against fixed-model baselines on the same tasks and commits. Measure accepted-task rate, cost per accepted result, escalation rate, planning-rubric results, and policy violations.
• Read total_cost_usd, num_turns, and modelUsage from the SDK result. modelUsage is the correct whole-tree view when subagents are involved.
• Start with local JSONL records. Add OpenTelemetry when the policy moves into shared or production use.

What You Will Learn Here

• How to define a small, deterministic routing policy
• How to run each route through the Claude Agent SDK at Low effort
• How to separate planning tools from coding tools
• How to isolate writable runs and keep tests outside model control
• How to build a representative evaluation matrix
• How to score code tasks and planning tasks differently
• How to calculate cost per accepted result
• How to decide whether the router is better than one approved default model

What This Router Is—and Is Not

This guide implements a routing layer for teams that already decided to test Low effort.

It is not:

• a claim that Low should replace Anthropic’s defaults
• a semantic classifier trained by another model
• an autonomous permission system
• a production scheduler
• a substitute for repository tests and human review

Anthropic currently recommends High as the default for Fable 5, Xhigh as the starting point for Opus 4.8 coding and agentic work, and Low primarily for simple or cost-sensitive work. A Low router is therefore an optimization you adopt after evaluation.

The target architecture is deliberately small:

Task request + explicit metadata
              |
              v
      [Deterministic router]
        |       |       |
        |       |       +--> ambiguous -> Fable 5 Low
        |       +----------> normal ----> Opus 4.8 Low
        +------------------> routine ---> Sonnet 5 Low
              |
              v
    [Tool, turn, and cost policy]
              |
              v
 [Read-only plan or isolated writer]
              |
              v
 [Controller-owned acceptance checks]
              |
       +------+------+
       |             |
     accept       escalate once
                       |
                       v
              human review / stop

The model router chooses a starting configuration. The acceptance system decides whether the work is usable.

How this differs from the other routing guides

The High and Ultracode comparison addresses model selection at the default and upper effort levels. This article assumes a team has already chosen to experiment with Low and asks a narrower question: can an explicit Low policy beat one approved default after failed runs, review, and escalation are included?

The Spec Kit model strategy routes workflow phases. This policy routes task risk inside implementation and agent runs. Cursor-native fast-loop routing is another axis again; reuse the evaluation contract, not the model mapping.

Why Explicit Metadata Beats a Model Classifier First

A model-based classifier adds:

• another paid request
• another source of nondeterminism
• a new prompt that needs evaluation
• the possibility that a weak classifier routes difficult work to a weak lane

For a small policy, let the caller provide task metadata:

// src/schema.ts
import { z } from "zod";

export const PlanningRubricSchema = z.object({
  requiredRequirements: z.array(z.string()).min(1),
  requiredDecisions: z.array(z.string()).min(1),
  requiredDependencies: z.array(z.string()),
  requiredScopeBoundaries: z.array(z.string()),
  mustIncludeRollback: z.boolean(),
  mustIncludeVerification: z.boolean(),
  requiresActionableSteps: z.boolean(),
  forbiddenAssumptions: z.array(z.string()),
});

export const TaskSchema = z
  .object({
    id: z.string().regex(/^[a-z0-9-]+$/),
    kind: z.enum(["plan", "code"]),
    risk: z.enum(["routine", "normal", "ambiguous"]),
    prompt: z.string().min(1),
    baseSha: z.string().regex(/^[0-9a-f]{7,40}$/),
    allowedPaths: z.array(z.string()),
    // Additional task-specific restrictions. Mandatory oracle paths live
    // in the controller-owned CHECKS registry and cannot be removed here.
    forbiddenPaths: z.array(z.string()).default([]),
    checkId: z.enum(["docs-links", "auth-tests"]).optional(),
    planningRubric: PlanningRubricSchema.optional(),
  })
  .superRefine((task, ctx) => {
    if (task.kind === "code" && !task.checkId) {
      ctx.addIssue({ code: "custom", message: "code tasks require checkId" });
    }
    if (task.kind === "plan" && !task.planningRubric) {
      ctx.addIssue({
        code: "custom",
        message: "plan tasks require planningRubric",
      });
    }
  });

export type RoutedTask = z.infer<typeof TaskSchema>;
export type PlanningRubric = z.infer<typeof PlanningRubricSchema>;

This makes routing visible in code review and easy to override:

{
  "id": "auth-cookie-fix",
  "kind": "code",
  "risk": "normal",
  "prompt": "Fix SameSite handling for the session cookie.",
  "baseSha": "8de1c9a",
  "allowedPaths": ["src/auth/**"],
  "forbiddenPaths": ["tests/**", "fixtures/**"],
  "checkId": "auth-tests"
}

The metadata can come from:

• a ticket template
• a command-line flag
• repository ownership rules
• a reviewed task manifest
• an upstream planning system

Automate classification later, after the explicit policy has enough accepted-run data to train or evaluate the classifier.

Step 1: Define the Policy

Pin the SDK used by the experiment and commit the lockfile. Version 0.3.211 bundled Claude Code 2.1.211 when this article was validated:

npm install @anthropic-ai/claude-agent-sdk@0.3.211 zod minimatch
npm install --save-dev tsx typescript @types/node

Define policy output separately from task metadata:

// src/policy.ts
export const POLICY_VERSION = "low-router-2026-07-16.1";

export type RouteDecision = {
  model: "claude-sonnet-5" | "claude-opus-4-8" | "claude-fable-5";
  effort: "low" | "high" | "xhigh";
  maxTurns: number;
  maxBudgetUsd: number;
  reason: string;
};

const LOW_POLICY: Record<string, RouteDecision> = {
  routine: {
    model: "claude-sonnet-5",
    effort: "low",
    maxTurns: 10,
    maxBudgetUsd: 1.5,
    reason: "Narrow work with deterministic acceptance checks",
  },
  normal: {
    model: "claude-opus-4-8",
    effort: "low",
    maxTurns: 16,
    maxBudgetUsd: 3,
    reason: "Measured middle route after repository evaluation",
  },
  ambiguous: {
    model: "claude-fable-5",
    effort: "low",
    maxTurns: 24,
    maxBudgetUsd: 6,
    reason: "Scoping or ambiguity is part of the task",
  },
};

export function routeTask(risk: string): RouteDecision {
  const decision = LOW_POLICY[risk];

  if (!decision) {
    throw new Error(`No approved Low-effort route for risk=${risk}`);
  }

  return decision;
}

The turn and dollar ceilings are example starting values, not benchmark-derived defaults. Tune them from your own run distribution. Their purpose is to make budget policy explicit before an agent starts.

The model mapping is also an experimental hypothesis. Anthropic recommends High for most Fable work and Xhigh for Opus coding. Treat Fable Low and Opus Low as candidate routes that must beat those vendor-guided baselines for their assigned task classes.

Do not silently fall through to the cheapest route. Unknown metadata should fail closed.

Step 2: Give Plans and Code Different Boundaries

Planning should be read-only. Coding needs writes, but not unrestricted shell access.

// src/boundary.ts
import type { Options } from "@anthropic-ai/claude-agent-sdk";

type TaskKind = "plan" | "code";

export function boundaryFor(kind: TaskKind): Pick<
  Options,
  "tools" | "allowedTools" | "permissionMode" | "disallowedTools"
> {
  if (kind === "plan") {
    return {
      tools: ["Read", "Glob", "Grep", "ExitPlanMode"],
      allowedTools: ["ExitPlanMode"],
      permissionMode: "plan",
      disallowedTools: ["Edit", "Write", "Bash"],
    };
  }

  return {
    tools: ["Read", "Edit", "Write", "Glob", "Grep"],
    allowedTools: [],
    permissionMode: "acceptEdits",
    disallowedTools: ["Bash"],
  };
}

The coding route intentionally removes Bash. The controller will run a fixed test command after the model stops.

This separation prevents the model from:

• inventing a weaker test command
• changing tests and then grading itself against the changed target
• reading secrets through arbitrary shell commands
• installing packages or reaching the network
• hiding a failing command behind a prose summary

Filesystem tools are still powerful. Run writable tasks in an isolated worktree, container, or VM and verify changed paths afterward.

Step 3: Run One Routed Task

The Agent SDK emits a final result message with:

• subtype
• total_cost_usd
• num_turns
• duration_ms
• cumulative usage
• per-model modelUsage

Use modelUsage for whole-tree accounting when a model invokes subagents; the top-level usage field can undercount nested work.

// src/run-task.ts
import {
  query,
  type SDKResultMessage,
} from "@anthropic-ai/claude-agent-sdk";

import { boundaryFor } from "./boundary.js";
import { routeTask, type RouteDecision } from "./policy.js";

const REVIEWED_SYSTEM_PROMPT = `
Work only on the requested task and inside the supplied workspace.
Do not weaken tests, fixtures, acceptance scripts, or success criteria.
Do not access the network or files outside the workspace.
Report incomplete work and failed assumptions explicitly.
`;

export type RunTaskInput = {
  id: string;
  kind: "plan" | "code";
  risk: "routine" | "normal" | "ambiguous";
  prompt: string;
  cwd: string;
  configDir: string;
  route?: RouteDecision;
};

export async function runTask(input: RunTaskInput) {
  const route = input.route ?? routeTask(input.risk);
  const boundary = boundaryFor(input.kind);
  let result: SDKResultMessage | undefined;
  let cliVersion: string | undefined;
  let thrown: unknown;

  try {
    for await (const message of query({
      prompt: input.prompt,
      options: {
        cwd: input.cwd,
        model: route.model,
        effort: route.effort,
        maxTurns: route.maxTurns,
        maxBudgetUsd: route.maxBudgetUsd,
        tools: boundary.tools,
        allowedTools: boundary.allowedTools,
        permissionMode: boundary.permissionMode,
        disallowedTools: boundary.disallowedTools,
        settingSources: [],
        strictMcpConfig: true,
        skills: [],
        systemPrompt: {
          type: "preset",
          preset: "claude_code",
          append: REVIEWED_SYSTEM_PROMPT,
        },
        env: {
          ...process.env,
          CLAUDE_CONFIG_DIR: input.configDir,
          CLAUDE_CODE_DISABLE_AUTO_MEMORY: "1",
          ENABLE_CLAUDEAI_MCP_SERVERS: "false",
        },
        persistSession: false,
      },
    })) {
      if (message.type === "system" && message.subtype === "init") {
        cliVersion = message.claude_code_version;
      }
      if (message.type === "result") {
        result = message;
      }
    }
  } catch (error) {
    // A single-shot query throws after yielding an error result.
    thrown = error;
  }

  if (!result) {
    throw thrown ?? new Error(`Agent ended without a result: ${input.id}`);
  }

  const refused = result.stop_reason === "refusal";

  return {
    taskId: input.id,
    route,
    sdkVersion: "0.3.211",
    cliVersion,
    subtype: result.subtype,
    loopCompleted: result.subtype === "success" && !refused,
    stopReason: result.stop_reason,
    resultText: result.subtype === "success" ? result.result : undefined,
    errors: result.subtype === "success" ? [] : result.errors,
    totalCostUsd: result.total_cost_usd,
    numTurns: result.num_turns,
    durationMs: result.duration_ms,
    modelUsage: result.modelUsage,
    permissionDenials: result.permission_denials,
  };
}

Three implementation details matter:

1. maxBudgetUsd uses a client-side estimate. Reconcile production spend against provider billing.
2. Controlled evals use an empty per-run CLAUDE_CONFIG_DIR and load no filesystem settings, skills, MCP servers, or auto-memory. Managed organization policy still applies; use a controlled host and hold that policy constant across conditions.
3. An SDK success subtype means the loop completed. A refusal is still rejected, and neither completion nor non-refusal means the code or plan passed acceptance.

Step 4: Create an Isolated Workspace

Every repeated coding run needs a clean linked checkout at the same base commit. Git worktrees still share repository history and administrative metadata; they isolate checked-out files, not the whole repository.

One small local approach is a detached Git worktree:

// src/workspace.ts
import { rmSync } from "node:fs";
import { mkdtemp } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { spawnSync } from "node:child_process";

function git(args: string[], cwd: string) {
  const result = spawnSync("git", args, {
    cwd,
    encoding: "utf8",
    shell: false,
  });
  if (result.error || result.status !== 0) {
    throw result.error ?? new Error(result.stderr);
  }
  return result.stdout.trim();
}

export async function createWorkspace(repo: string, requestedSha: string) {
  const baseSha = git(["rev-parse", requestedSha], repo);
  const cwd = await mkdtemp(join(tmpdir(), "low-router-"));
  const configDir = await mkdtemp(join(tmpdir(), "low-router-config-"));

  git(["worktree", "add", "--detach", cwd, baseSha], repo);

  const actualSha = git(["rev-parse", "HEAD"], cwd);
  if (actualSha !== baseSha) {
    throw new Error(`Workspace SHA mismatch: ${actualSha} != ${baseSha}`);
  }

  return {
    cwd,
    configDir,
    baseSha,
    cleanup() {
      git(["worktree", "remove", "--force", cwd], repo);
      rmSync(configDir, { recursive: true, force: true });
    },
  };
}

The task manifest is the single source of truth for baseSha; the controller resolves it to a full SHA and verifies the worktree before invoking the model.

For untrusted repositories or multi-tenant use, a worktree is not a security sandbox. Use an ephemeral container or VM with:

• no network by default
• short-lived credentials
• CPU, memory, disk, time, turn, and token ceilings
• a read-only base image
• an allowlisted writable workspace
• guaranteed cleanup

Step 5: Keep Acceptance Outside the Model

The task manifest selects a controller-owned check ID, not an executable command.

// src/checks.ts
export const CHECKS = {
  "docs-links": {
    executable: "npm",
    args: ["run", "check:links"],
    timeoutMs: 120_000,
    immutablePaths: ["tests/**", "fixtures/**", "scripts/check-links.*"],
    env: { CI: "1" },
    cwdPolicy: "workspace-root",
  },
  "auth-tests": {
    executable: "npm",
    args: ["test", "--", "tests/auth"],
    timeoutMs: 180_000,
    immutablePaths: ["tests/**", "test/**", "fixtures/**", "snapshots/**"],
    env: { CI: "1" },
    cwdPolicy: "workspace-root",
  },
} as const;

export type CheckId = keyof typeof CHECKS;

The immutable registry is code-reviewed with the router. It defines executable, arguments, timeout, environment, working-directory policy, and mandatory immutable paths. A manifest can add restrictions but cannot remove these.

Enforce changed paths before running the check:

// src/accept-code.ts
import { minimatch } from "minimatch";

import { CHECKS } from "./checks.js";
import type { SandboxRunner } from "./sandbox.js";
import type { RoutedTask } from "./schema.js";

import { spawnSync } from "node:child_process";

function gitPaths(cwd: string, args: string[]) {
  const result = spawnSync("git", args, {
    cwd,
    encoding: "utf8",
    shell: false,
  });
  if (result.error || result.status !== 0) {
    throw result.error ?? new Error(result.stderr);
  }
  return result.stdout
    .split("\0")
    .filter(Boolean);
}

export function changedPaths(cwd: string) {
  return [
    ...new Set([
      ...gitPaths(cwd, ["diff", "--name-only", "-z", "HEAD", "--"]),
      ...gitPaths(cwd, ["ls-files", "-z", "--others", "--"]),
    ]),
  ].sort();
}

export function untrackedPaths(cwd: string) {
  return gitPaths(cwd, ["ls-files", "-z", "--others", "--"]).sort();
}

export async function acceptCodeTask(
  task: RoutedTask,
  cwd: string,
  loopCompleted: boolean,
  stopReason: string | null,
  runInSandbox: SandboxRunner,
) {
  const reasons: string[] = [];
  const check =
    task.kind === "code" && task.checkId ? CHECKS[task.checkId] : undefined;
  const immutablePaths = [
    ...(check?.immutablePaths ?? []),
    ...task.forbiddenPaths,
  ];
  const beforeCheck = changedPaths(cwd);

  if (!loopCompleted) reasons.push(`agent stopped: ${stopReason ?? "error"}`);

  const outsideScope = beforeCheck.filter(
    (path) => !task.allowedPaths.some((glob) => minimatch(path, glob)),
  );
  const forbidden = beforeCheck.filter((path) =>
    immutablePaths.some((glob) => minimatch(path, glob)),
  );

  if (outsideScope.length) {
    reasons.push(`outside allowed paths: ${outsideScope.join(", ")}`);
  }
  if (forbidden.length) {
    reasons.push(`modified immutable oracle files: ${forbidden.join(", ")}`);
  }

  if (reasons.length || !check) {
    return {
      accepted: false,
      reasons,
      paths: beforeCheck,
      testExitCode: undefined,
      testStdout: undefined,
      testStderr: undefined,
      sandboxAuditViolations: [],
    };
  }

  const test = await runInSandbox({
    workspace: cwd,
    executable: check.executable,
    args: [...check.args],
    timeoutMs: check.timeoutMs,
    env: { ...check.env },
    cwdPolicy: check.cwdPolicy,
    network: "none",
    credentials: "none",
  });

  if (test.exitCode !== 0) {
    reasons.push(test.error ?? `check exited ${test.exitCode}`);
  }
  reasons.push(...test.auditViolations);

  // A compliant sandbox receives a disposable copy or a read-only mount.
  // Scan again anyway; a post-check change is always a policy violation.
  const afterCheck = changedPaths(cwd);
  if (JSON.stringify(afterCheck) !== JSON.stringify(beforeCheck)) {
    reasons.push("controller check changed the agent workspace");
  }

  return {
    accepted: reasons.length === 0,
    reasons,
    paths: afterCheck,
    testExitCode: test.exitCode,
    testStdout: test.stdout,
    testStderr: test.stderr,
    sandboxAuditViolations: test.auditViolations,
  };
}

The sandbox adapter is an intentional infrastructure boundary:

// src/sandbox.ts
export type SandboxRunner = (request: {
  workspace: string;
  executable: string;
  args: string[];
  timeoutMs: number;
  env: Record<string, string>;
  cwdPolicy: "workspace-root";
  network: "none";
  credentials: "none";
}) => Promise<{
  exitCode: number;
  stdout: string;
  stderr: string;
  error?: string;
  auditViolations: string[];
}>;

Do not provide an unsafe host-process fallback. Wire this interface to your container, VM, or existing sandbox service. It must copy or read-only mount the agent output, disable network, expose no controller credentials, enforce resource and time limits, and discard the check environment afterward.

Keep the unconfigured state fail-closed:

// src/configured-sandbox.ts (Docker reference adapter)
import { randomUUID } from "node:crypto";
import { spawnSync } from "node:child_process";
import type { SandboxRunner } from "./sandbox.js";

const docker = (args: string[], timeout?: number) =>
  spawnSync("docker", args, {
    encoding: "utf8",
    shell: false,
    timeout,
  });

export const runCheckInSandbox: SandboxRunner = async (request) => {
  if (request.cwdPolicy !== "workspace-root") {
    throw new Error(`Unsupported cwd policy: ${request.cwdPolicy}`);
  }
  const image = process.env.LOW_ROUTER_SANDBOX_IMAGE;
  if (!image?.includes("@sha256:")) {
    throw new Error("LOW_ROUTER_SANDBOX_IMAGE must be pinned by digest");
  }
  const imageCheck = docker(["image", "inspect", image]);
  if (imageCheck.error || imageCheck.status !== 0) {
    throw new Error("Pinned sandbox image must already exist locally");
  }

  const name = `low-router-${randomUUID()}`;
  const envArgs = Object.entries(request.env).flatMap(([key, value]) => [
    "--env",
    `${key}=${value}`,
  ]);
  const create = docker([
    "create",
    "--name",
    name,
    "--network",
    "none",
    "--cap-drop",
    "ALL",
    "--security-opt",
    "no-new-privileges",
    "--memory",
    "2g",
    "--cpus",
    "2",
    "--pids-limit",
    "256",
    "--workdir",
    "/workspace",
    ...envArgs,
    image,
    request.executable,
    ...request.args,
  ]);
  if (create.error || create.status !== 0) {
    throw create.error ?? new Error(create.stderr);
  }

  const containerId = create.stdout.trim();
  try {
    const networkMode = docker([
      "inspect",
      "--format",
      "{{.HostConfig.NetworkMode}}",
      containerId,
    ]);
    const auditViolations =
      networkMode.stdout.trim() === "none" ? [] : ["sandbox network enabled"];

    const copy = docker(["cp", `${request.workspace}/.`, `${containerId}:/workspace`]);
    if (copy.error || copy.status !== 0) {
      throw copy.error ?? new Error(copy.stderr);
    }

    const result = docker(["start", "--attach", containerId], request.timeoutMs);
    return {
      exitCode: result.status ?? 124,
      stdout: result.stdout,
      stderr: result.stderr,
      error: result.error?.message,
      auditViolations,
    };
  } finally {
    docker(["rm", "--force", containerId]);
  }
};

The pinned image must contain an empty /workspace, dependencies for the base commit, and no credentials. The controller copies the agent output into the stopped container, runs the fixed executable without a shell, gives the container no host mounts or inherited environment, disables networking, limits resources, and destroys it afterward.

Use this adapter only where the Docker daemon is an approved isolation boundary. Replace it with your organization’s VM or sandbox service when stronger isolation is required.

For code tasks, deterministic acceptance now requires:

agent loop completed without refusal
+ changed files stay inside allowed paths
+ tests, fixtures, and acceptance scripts are unchanged
+ controller-owned check exits 0
= deterministically accepted code run

The model cannot weaken the visible tests without failing the path gate. An immutable hidden acceptance suite outside the writable worktree is stronger when the task itself requires adding or changing tests.

Capture raw output, exit code, changed paths, and the final diff as artifacts. Human or policy review may still reject a deterministically accepted patch.

Preserve the result before deleting the worktree:

// src/artifacts.ts
import { createHash } from "node:crypto";
import {
  copyFile,
  lstat,
  mkdir,
  readFile,
  writeFile,
} from "node:fs/promises";
import { dirname, isAbsolute, join, relative, resolve } from "node:path";
import { spawnSync } from "node:child_process";

import { changedPaths, untrackedPaths } from "./accept-code.js";

const sha256 = (data: Buffer | string) =>
  createHash("sha256").update(data).digest("hex");

export async function preserveArtifacts(
  cwd: string,
  outputDir: string,
  runId: string,
) {
  const artifactDir = resolve(outputDir, "artifacts", runId);
  await mkdir(artifactDir, { recursive: true });

  const diff = spawnSync("git", ["diff", "--binary", "HEAD", "--"], {
    cwd,
    encoding: "utf8",
    shell: false,
  });
  if (diff.error || diff.status !== 0) {
    throw diff.error ?? new Error(diff.stderr);
  }

  const patchPath = join(artifactDir, "tracked.patch");
  await writeFile(patchPath, diff.stdout);

  const untracked: Array<{ path: string; sha256: string; mode: number }> = [];
  for (const path of untrackedPaths(cwd)) {
    const source = resolve(cwd, path);
    const destination = resolve(artifactDir, "untracked", path);
    const relativeSource = relative(resolve(cwd), source);
    if (relativeSource.startsWith("..") || isAbsolute(relativeSource)) {
      throw new Error(`Unsafe untracked path: ${path}`);
    }

    const stat = await lstat(source);
    if (!stat.isFile()) throw new Error(`Unsupported untracked type: ${path}`);

    await mkdir(dirname(destination), { recursive: true });
    await copyFile(source, destination);
    untracked.push({
      path,
      sha256: sha256(await readFile(source)),
      mode: stat.mode,
    });
  }

  const manifest = {
    changedPaths: changedPaths(cwd),
    patchPath,
    patchSha256: sha256(diff.stdout),
    untracked,
  };
  const manifestPath = join(artifactDir, "manifest.json");
  const manifestText = JSON.stringify(manifest, null, 2);
  await writeFile(manifestPath, manifestText);
  return {
    manifestPath,
    manifestSha256: sha256(manifestText),
  };
}

The binary patch preserves tracked edits, deletions, renames, and mode changes. Untracked regular files are copied separately with modes and SHA-256 hashes; unsupported types fail the run instead of being silently discarded.

Planning acceptance

The validated manifest carries:

type PlanningRubric = {
  requiredRequirements: string[];
  requiredDecisions: string[];
  requiredDependencies: string[];
  requiredScopeBoundaries: string[];
  mustIncludeRollback: boolean;
  mustIncludeVerification: boolean;
  requiresActionableSteps: boolean;
  forbiddenAssumptions: string[];
};

Plans need a different oracle. Use a fixed rubric:

Prefer blinded human review for the first evaluation. If you later add an automated judge, evaluate the judge against human labels before trusting it.

Make the review machine-checkable without asking the reviewer for a free-form verdict:

// src/accept-plan.ts
import { z } from "zod";
import type { RoutedTask } from "./schema.js";

export const PlanReviewSchema = z.object({
  runId: z.string().uuid(),
  coveredRequirements: z.array(z.string()),
  coveredDecisions: z.array(z.string()),
  coveredDependencies: z.array(z.string()),
  coveredScopeBoundaries: z.array(z.string()),
  includesRollback: z.boolean(),
  includesVerification: z.boolean(),
  actionableSteps: z.boolean(),
  assessedForbiddenAssumptions: z.array(z.string()),
  invalidAssumptions: z.array(z.string()),
  humanReviewMinutes: z.number().nonnegative(),
});

export type PlanReview = z.infer<typeof PlanReviewSchema>;

export function acceptPlan(task: RoutedTask, review: PlanReview) {
  if (task.kind !== "plan" || !task.planningRubric) {
    throw new Error("planning rubric required");
  }

  const rubric = task.planningRubric;
  const missingRequirements = rubric.requiredRequirements.filter(
    (item) => !review.coveredRequirements.includes(item),
  );
  const missingDecisions = rubric.requiredDecisions.filter(
    (item) => !review.coveredDecisions.includes(item),
  );
  const missingDependencies = rubric.requiredDependencies.filter(
    (item) => !review.coveredDependencies.includes(item),
  );
  const missingScope = rubric.requiredScopeBoundaries.filter(
    (item) => !review.coveredScopeBoundaries.includes(item),
  );
  const unassessedAssumptions = rubric.forbiddenAssumptions.filter(
    (item) => !review.assessedForbiddenAssumptions.includes(item),
  );
  const reasons = [
    ...missingRequirements.map((item) => `missing requirement: ${item}`),
    ...missingDecisions.map((item) => `missing decision: ${item}`),
    ...missingDependencies.map((item) => `missing dependency: ${item}`),
    ...missingScope.map((item) => `missing scope boundary: ${item}`),
    ...unassessedAssumptions.map(
      (item) => `forbidden assumption not assessed: ${item}`,
    ),
    ...review.invalidAssumptions.map((item) => `invalid assumption: ${item}`),
  ];

  if (rubric.mustIncludeRollback && !review.includesRollback) {
    reasons.push("rollback missing");
  }
  if (rubric.mustIncludeVerification && !review.includesVerification) {
    reasons.push("verification missing");
  }
  if (rubric.requiresActionableSteps && !review.actionableSteps) {
    reasons.push("actionable steps missing");
  }

  return { accepted: reasons.length === 0, reasons };
}

Store the blinded review as a separate append-only record linked to the run ID:

// src/review-plan.ts
import { createHash } from "node:crypto";
import { appendFile, readFile } from "node:fs/promises";
import { join, resolve } from "node:path";

import {
  acceptPlan,
  PlanReviewSchema,
} from "./accept-plan.js";
import type { EvalRecord } from "./record.js";
import { TaskSchema } from "./schema.js";

const [runsArg, taskFileArg, reviewFileArg, outputDirArg] =
  process.argv.slice(2);
if (!runsArg || !taskFileArg || !reviewFileArg || !outputDirArg) {
  throw new Error(
    "usage: review-plan <runs.jsonl> <task.json> <review.json> <output-dir>",
  );
}

const sha256 = (value: string) =>
  createHash("sha256").update(value).digest("hex");
const taskText = await readFile(resolve(taskFileArg), "utf8");
const task = TaskSchema.parse(JSON.parse(taskText));
const review = PlanReviewSchema.parse(
  JSON.parse(await readFile(resolve(reviewFileArg), "utf8")),
);
const runs = (await readFile(resolve(runsArg), "utf8"))
  .split("\n")
  .filter(Boolean)
  .map((line) => JSON.parse(line) as EvalRecord);
const run = runs.find((candidate) => candidate.runId === review.runId);

if (
  !run ||
  task.kind !== "plan" ||
  run.taskId !== task.id ||
  run.accepted !== null
) {
  throw new Error("Review does not target one pending plan run");
}
if (
  run.taskManifestSha256 !== sha256(taskText) ||
  run.planningRubricSha256 !== sha256(JSON.stringify(task.planningRubric))
) {
  throw new Error("Task manifest or planning rubric changed after the run");
}
if (!run.planArtifactPath || !run.planArtifactSha256) {
  throw new Error("Pending run has no bound plan artifact");
}
const planText = await readFile(run.planArtifactPath, "utf8");
if (sha256(planText) !== run.planArtifactSha256) {
  throw new Error("Plan artifact hash mismatch");
}

const acceptance = acceptPlan(task, review);
const reviewPath = join(resolve(outputDirArg), "plan-reviews.jsonl");
let existingReviews: Array<{ runId: string }> = [];
try {
  existingReviews = (await readFile(reviewPath, "utf8"))
    .split("\n")
    .filter(Boolean)
    .map((line) => JSON.parse(line) as { runId: string });
} catch (error) {
  if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
}
if (existingReviews.some((candidate) => candidate.runId === review.runId)) {
  throw new Error("Plan run already has a terminal review");
}

await appendFile(
  reviewPath,
  `${JSON.stringify({
    runId: review.runId,
    taskId: run.taskId,
    experimentId: run.experimentId,
    condition: run.condition,
    reviewVersion: "plan-review-2026-07-16.1",
    accepted: acceptance.accepted,
    acceptanceReasons: acceptance.reasons,
    humanReviewMinutes: review.humanReviewMinutes,
  })}\n`,
);

Join plan-reviews.jsonl to runs.jsonl by runId. Exclude pending plan rows from accepted-rate and cost-per-accepted-result denominators until a terminal review exists. Do not rewrite the original agent-run record.

Step 6: Build the Evaluation Matrix

The router must beat a simpler baseline, not merely produce plausible outputs.

This is an offline scenario evaluation: pinned tasks and explicit acceptance criteria decide whether the policy can be promoted. After rollout, add production trace grading to detect tool misuse, drift, and new task shapes. See Trace Grading vs. Scenario Testing for that wider evaluation loop.

Compare at least:

1. Sonnet 5 Low for every task
2. Opus 4.8 Low for every task
3. Fable 5 Low for every task
4. The routed policy
5. Your approved High or Xhigh default

Represent fixed baselines with the same run configuration shape:

// src/conditions.ts
import { routeTask, type RouteDecision } from "./policy.js";
import type { RoutedTask } from "./schema.js";

type EvalCondition = {
  name: string;
  decide(task: RoutedTask): RouteDecision;
};

const conditions: EvalCondition[] = [
  { name: "routed-low", decide: (task) => routeTask(task.risk) },
  {
    name: "sonnet-low",
    decide: () => ({
      model: "claude-sonnet-5",
      effort: "low",
      maxTurns: 16,
      maxBudgetUsd: 3,
      reason: "Fixed Low baseline",
    }),
  },
  {
    name: "opus-low",
    decide: () => ({
      model: "claude-opus-4-8",
      effort: "low",
      maxTurns: 16,
      maxBudgetUsd: 3,
      reason: "Fixed Low baseline",
    }),
  },
  {
    name: "fable-low",
    decide: () => ({
      model: "claude-fable-5",
      effort: "low",
      maxTurns: 24,
      maxBudgetUsd: 6,
      reason: "Fixed Low baseline",
    }),
  },
  {
    name: "approved-opus-high",
    decide: () => ({
      model: "claude-opus-4-8",
      effort: "high",
      maxTurns: 24,
      maxBudgetUsd: 6,
      reason: "Existing approved baseline",
    }),
  },
  {
    name: "approved-opus-xhigh",
    decide: () => ({
      model: "claude-opus-4-8",
      effort: "xhigh",
      maxTurns: 30,
      maxBudgetUsd: 10,
      reason: "Vendor-guided coding baseline; keep only if team-approved",
    }),
  },
  {
    name: "routed-escalation-routine",
    decide: () => ({
      model: "claude-opus-4-8",
      effort: "low",
      maxTurns: 16,
      maxBudgetUsd: 3,
      reason: "Linked escalation from routed routine work",
    }),
  },
  {
    name: "routed-escalation-normal",
    decide: () => ({
      model: "claude-opus-4-8",
      effort: "high",
      maxTurns: 24,
      maxBudgetUsd: 6,
      reason: "Linked escalation from routed normal work",
    }),
  },
];

export function decisionFor(name: string, task: RoutedTask) {
  const condition = conditions.find((candidate) => candidate.name === name);
  if (!condition) throw new Error(`Unknown eval condition: ${name}`);
  return condition.decide(task);
}

Pass the selected condition into runTask. The High baseline should be the configuration your team already approves, not a new recommendation from this article.

The same effort label is calibrated per model and does not imply an identical hidden reasoning budget. Compare accepted outcomes and total economics, not effort names or token prices alone.

Use a manifest with representative work:

[
  {
    "id": "docs-link-update",
    "kind": "code",
    "risk": "routine",
    "baseSha": "8de1c9a",
    "prompt": "Update the deprecated documentation links.",
    "allowedPaths": ["docs/**"],
    "forbiddenPaths": ["tests/**", "fixtures/**"],
    "checkId": "docs-links"
  },
  {
    "id": "auth-refactor-plan",
    "kind": "plan",
    "risk": "normal",
    "baseSha": "8de1c9a",
    "prompt": "Plan the session-store migration without editing files.",
    "allowedPaths": [],
    "planningRubric": {
      "requiredRequirements": ["no forced logout", "preserve session TTL"],
      "requiredDecisions": ["dual-write", "cutover", "rollback"],
      "requiredDependencies": ["database migration", "session TTL"],
      "requiredScopeBoundaries": ["auth service", "session store"],
      "mustIncludeRollback": true,
      "mustIncludeVerification": true,
      "requiresActionableSteps": true,
      "forbiddenAssumptions": ["zero active sessions"]
    }
  }
]

A reasonable first dataset contains 20–50 representative tasks sampled from real failures and accepted work. Include routine coding, normal coding, plans with known dependencies, and ambiguous scoping or migration tasks.

Run at least five repeats per condition as a pilot. Then use observed variance and the confidence required by the decision to select the final sample size.

Randomize execution order so provider load, cache warmth, or time-of-day effects do not consistently favor one condition.

Step 7: Record Every Attempt

Write one append-only JSONL record per run:

// src/record.ts
export type EvalRecord = {
  runId: string;
  rootRunId: string;
  parentRunId?: string;
  condition: string;
  attemptKind: "initial" | "escalation";
  experimentId: string;
  taskId: string;
  repeat: number;
  baseSha: string;
  policyVersion: string;
  requestedRisk: "routine" | "normal" | "ambiguous";
  selectedModel: string;
  selectedEffort: "low" | "high" | "xhigh";
  selectedMaxTurns: number;
  selectedMaxBudgetUsd: number;
  promptVersion: string;
  checkRegistryVersion: string;
  taskManifestSha256: string;
  planningRubricSha256?: string;
  resultSubtype: string;
  stopReason: string | null;
  accepted: boolean | null;
  acceptanceReasons: string[];
  totalCostUsd: number | null;
  numTurns: number | null;
  durationMs: number | null;
  controllerDurationMs: number;
  modelUsage: unknown;
  permissionDenials: unknown[];
  sandboxAuditViolations: string[];
  sdkVersion: string;
  cliVersion?: string;
  changedPaths: string[];
  testExitCode?: number;
  artifactManifestPath?: string;
  artifactManifestSha256?: string;
  planArtifactPath?: string;
  planArtifactSha256?: string;
  followUpKind: "model_escalation" | "human_review" | null;
  policyViolations: string[];
};

Define one bounded escalation policy:

// src/escalation.ts
import type { RoutedTask } from "./schema.js";
import type { RouteDecision } from "./policy.js";

export type FollowUp =
  | { kind: "model_escalation"; route: RouteDecision }
  | { kind: "human_review" }
  | null;

export function followUpFor(
  condition: string,
  task: RoutedTask,
  accepted: boolean | null,
): FollowUp {
  if (accepted === null) return { kind: "human_review" };
  if (accepted || condition !== "routed-low") return null;

  if (task.risk === "routine") {
    return {
      kind: "model_escalation",
      route: {
        model: "claude-opus-4-8",
        effort: "low",
        maxTurns: 16,
        maxBudgetUsd: 3,
        reason: "One measured escalation from routine to normal",
      },
    };
  }

  if (task.risk === "normal") {
    return {
      kind: "model_escalation",
      route: {
        model: "claude-opus-4-8",
        effort: "high",
        maxTurns: 24,
        maxBudgetUsd: 6,
        reason: "One effort escalation after Opus Low rejection",
      },
    };
  }

  return { kind: "human_review" };
}

Now provide the missing controller entrypoint:

// src/run-one.ts
import { createHash, randomUUID } from "node:crypto";
import { appendFile, mkdir, readFile, writeFile } from "node:fs/promises";
import { join, resolve } from "node:path";

import {
  acceptCodeTask,
  changedPaths as inspectChangedPaths,
} from "./accept-code.js";
import { preserveArtifacts } from "./artifacts.js";
import { decisionFor } from "./conditions.js";
import { runCheckInSandbox } from "./configured-sandbox.js";
import { followUpFor } from "./escalation.js";
import { POLICY_VERSION } from "./policy.js";
import type { EvalRecord } from "./record.js";
import { runTask } from "./run-task.js";
import { TaskSchema } from "./schema.js";
import { createWorkspace } from "./workspace.js";

const [repoArg, taskFileArg, outputDirArg] = process.argv.slice(2);
if (!repoArg || !taskFileArg || !outputDirArg) {
  throw new Error("usage: run-one <repo> <task.json> <output-dir>");
}

const repo = resolve(repoArg);
const outputDir = resolve(outputDirArg);
const taskText = await readFile(resolve(taskFileArg), "utf8");
const task = TaskSchema.parse(JSON.parse(taskText));
const sha256 = (value: string) =>
  createHash("sha256").update(value).digest("hex");
const taskManifestSha256 = sha256(taskText);
const planningRubricSha256 = task.planningRubric
  ? sha256(JSON.stringify(task.planningRubric))
  : undefined;
const condition = process.env.CONDITION ?? "routed-low";
const route = decisionFor(condition, task);
const experimentId = process.env.EXPERIMENT_ID ?? "local-pilot";
const repeat = Number(process.env.REPEAT ?? "0");
if (!Number.isInteger(repeat) || repeat < 0) {
  throw new Error("REPEAT must be a non-negative integer");
}
const parentRunId = process.env.PARENT_RUN_ID || undefined;
const uuidPattern =
  /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;

await mkdir(outputDir, { recursive: true });

let parent: EvalRecord | undefined;
if (parentRunId) {
  if (!uuidPattern.test(parentRunId)) throw new Error("Invalid parent UUID");
  const existing = (await readFile(join(outputDir, "runs.jsonl"), "utf8"))
    .split("\n")
    .filter(Boolean)
    .map((line) => JSON.parse(line) as EvalRecord);
  parent = existing.find((row) => row.runId === parentRunId);
  if (!parent || parent.followUpKind !== "model_escalation") {
    throw new Error("Parent is not eligible for model escalation");
  }
  if (!uuidPattern.test(parent.rootRunId)) {
    throw new Error("Parent has invalid root UUID");
  }
  if (existing.some((row) => row.parentRunId === parentRunId)) {
    throw new Error("Parent already has an escalation attempt");
  }
  const expectedCondition =
    parent.requestedRisk === "routine"
      ? "routed-escalation-routine"
      : "routed-escalation-normal";
  if (
    condition !== expectedCondition ||
    parent.experimentId !== experimentId ||
    parent.taskId !== task.id ||
    parent.repeat !== repeat ||
    parent.requestedRisk !== task.risk ||
    parent.taskManifestSha256 !== taskManifestSha256 ||
    parent.policyVersion !== POLICY_VERSION
  ) {
    throw new Error("Escalation metadata does not match parent");
  }
}

const workspace = await createWorkspace(repo, task.baseSha);
if (parent && parent.baseSha !== workspace.baseSha) {
  throw new Error("Escalation base SHA does not match parent");
}
const runId = randomUUID();
const rootRunId = parent?.rootRunId ?? runId;
const attemptKind = parentRunId ? "escalation" : "initial";
const controllerStartedAt = Date.now();
const PROMPT_VERSION = "low-router-prompt-2026-07-16.1";
const CHECK_REGISTRY_VERSION = "low-router-checks-2026-07-16.1";

let recordWritten = false;
let agent: Awaited<ReturnType<typeof runTask>> | undefined;
let changedPaths: string[] = [];
let artifactManifestPath: string | undefined;
let artifactManifestSha256: string | undefined;
let planArtifactPath: string | undefined;
let planArtifactSha256: string | undefined;
let sandboxAuditViolations: string[] = [];

try {
  try {
    agent = await runTask({
      id: task.id,
      kind: task.kind,
      risk: task.risk,
      prompt: task.prompt,
      cwd: workspace.cwd,
      configDir: workspace.configDir,
      route,
    });
  } catch (error) {
    if (task.kind === "code") {
      try {
        changedPaths = inspectChangedPaths(workspace.cwd);
        const artifacts = await preserveArtifacts(
          workspace.cwd,
          outputDir,
          runId,
        );
        artifactManifestPath = artifacts.manifestPath;
        artifactManifestSha256 = artifacts.manifestSha256;
      } catch {
        // Preserve the original SDK/process error in the attempt record.
      }
    }

    const record: EvalRecord = {
      runId,
      rootRunId,
      parentRunId,
      condition,
      attemptKind,
      experimentId,
      taskId: task.id,
      repeat,
      baseSha: workspace.baseSha,
      policyVersion: POLICY_VERSION,
      requestedRisk: task.risk,
      selectedModel: route.model,
      selectedEffort: route.effort,
      selectedMaxTurns: route.maxTurns,
      selectedMaxBudgetUsd: route.maxBudgetUsd,
      promptVersion: PROMPT_VERSION,
      checkRegistryVersion: CHECK_REGISTRY_VERSION,
      taskManifestSha256,
      planningRubricSha256,
      resultSubtype: "controller_error",
      stopReason: null,
      accepted: false,
      acceptanceReasons: [
        error instanceof Error ? error.message : String(error),
      ],
      totalCostUsd: null,
      numTurns: null,
      durationMs: null,
      controllerDurationMs: Date.now() - controllerStartedAt,
      modelUsage: {},
      permissionDenials: [],
      sandboxAuditViolations: [],
      sdkVersion: "0.3.211",
      changedPaths,
      artifactManifestPath,
      artifactManifestSha256,
      followUpKind: "human_review",
      policyViolations: ["controller_error"],
    };
    await appendFile(
      join(outputDir, "runs.jsonl"),
      `${JSON.stringify(record)}\n`,
    );
    recordWritten = true;
    throw error;
  }

  let accepted: boolean | null;
  let reasons: string[];
  let testExitCode: number | undefined;

  if (task.kind === "code") {
    changedPaths = inspectChangedPaths(workspace.cwd);
    const artifacts = await preserveArtifacts(
      workspace.cwd,
      outputDir,
      runId,
    );
    artifactManifestPath = artifacts.manifestPath;
    artifactManifestSha256 = artifacts.manifestSha256;

    const check = await acceptCodeTask(
      task,
      workspace.cwd,
      agent.loopCompleted,
      agent.stopReason,
      runCheckInSandbox,
    );
    accepted = check.accepted;
    reasons = check.reasons;
    changedPaths = check.paths;
    testExitCode = check.testExitCode;
    sandboxAuditViolations = check.sandboxAuditViolations;
    await writeFile(
      join(outputDir, `${runId}.test.log`),
      `${check.testStdout ?? ""}\n${check.testStderr ?? ""}`,
    );
  } else if (!agent.loopCompleted) {
    accepted = false;
    reasons = [`agent stopped: ${agent.stopReason ?? agent.subtype}`];
  } else {
    accepted = null;
    reasons = ["pending blinded planning-rubric review"];
    await writeFile(
      (planArtifactPath = join(outputDir, `${runId}.plan.md`)),
      agent.resultText ?? "",
    );
    planArtifactSha256 = sha256(agent.resultText ?? "");
  }

  const followUp = followUpFor(condition, task, accepted);

  const record: EvalRecord = {
    runId,
    rootRunId,
    parentRunId,
    condition,
    attemptKind,
    experimentId,
    taskId: task.id,
    repeat,
    baseSha: workspace.baseSha,
    policyVersion: POLICY_VERSION,
    requestedRisk: task.risk,
    selectedModel: route.model,
    selectedEffort: route.effort,
    selectedMaxTurns: route.maxTurns,
    selectedMaxBudgetUsd: route.maxBudgetUsd,
    promptVersion: PROMPT_VERSION,
    checkRegistryVersion: CHECK_REGISTRY_VERSION,
    taskManifestSha256,
    planningRubricSha256,
    resultSubtype: agent.subtype,
    stopReason: agent.stopReason,
    accepted,
    acceptanceReasons: reasons,
    totalCostUsd: agent.totalCostUsd,
    numTurns: agent.numTurns,
    durationMs: agent.durationMs,
    controllerDurationMs: Date.now() - controllerStartedAt,
    modelUsage: agent.modelUsage,
    permissionDenials: agent.permissionDenials,
    sandboxAuditViolations,
    sdkVersion: agent.sdkVersion,
    cliVersion: agent.cliVersion,
    changedPaths,
    testExitCode,
    artifactManifestPath,
    artifactManifestSha256,
    planArtifactPath,
    planArtifactSha256,
    followUpKind: followUp?.kind ?? null,
    policyViolations: [
      ...reasons.filter(
        (reason) =>
          reason.startsWith("outside allowed paths") ||
          reason.startsWith("modified immutable oracle files") ||
          reason === "controller check changed the agent workspace",
      ),
      ...sandboxAuditViolations,
      ...agent.permissionDenials.map(
        (denial) => `permission denied: ${JSON.stringify(denial)}`,
      ),
    ],
  };

  await appendFile(
    join(outputDir, "runs.jsonl"),
    `${JSON.stringify(record)}\n`,
  );
  recordWritten = true;
  console.log(JSON.stringify({ condition, record, followUp }, null, 2));
} catch (error) {
  if (!recordWritten) {
    const record: EvalRecord = {
      runId,
      rootRunId,
      parentRunId,
      condition,
      attemptKind,
      experimentId,
      taskId: task.id,
      repeat,
      baseSha: workspace.baseSha,
      policyVersion: POLICY_VERSION,
      requestedRisk: task.risk,
      selectedModel: route.model,
      selectedEffort: route.effort,
      selectedMaxTurns: route.maxTurns,
      selectedMaxBudgetUsd: route.maxBudgetUsd,
      promptVersion: PROMPT_VERSION,
      checkRegistryVersion: CHECK_REGISTRY_VERSION,
      taskManifestSha256,
      planningRubricSha256,
      resultSubtype: "controller_error",
      stopReason: agent?.stopReason ?? null,
      accepted: false,
      acceptanceReasons: [
        error instanceof Error ? error.message : String(error),
      ],
      totalCostUsd: agent?.totalCostUsd ?? null,
      numTurns: agent?.numTurns ?? null,
      durationMs: agent?.durationMs ?? null,
      controllerDurationMs: Date.now() - controllerStartedAt,
      modelUsage: agent?.modelUsage ?? {},
      permissionDenials: agent?.permissionDenials ?? [],
      sandboxAuditViolations,
      sdkVersion: agent?.sdkVersion ?? "0.3.211",
      cliVersion: agent?.cliVersion,
      changedPaths,
      artifactManifestPath,
      artifactManifestSha256,
      planArtifactPath,
      planArtifactSha256,
      followUpKind: "human_review",
      policyViolations: ["controller_error"],
    };
    await appendFile(
      join(outputDir, "runs.jsonl"),
      `${JSON.stringify(record)}\n`,
    );
  }
  throw error;
} finally {
  workspace.cleanup();
}

Invoke the pinned local executable; do not use npx in the eval, because it may download or select a different package:

./node_modules/.bin/tsx src/run-one.ts \
  /path/to/repository \
  evals/auth-cookie-fix.json \
  results/low-router-pilot

Select baselines with CONDITION and label repeats explicitly:

EXPERIMENT_ID=low-router-01 \
CONDITION=approved-opus-high \
REPEAT=0 \
./node_modules/.bin/tsx src/run-one.ts \
  /path/to/repository \
  evals/auth-cookie-fix.json \
  results/low-router-pilot

The controller records a proposed escalation but does not execute it in the dirty workspace. Start it in a fresh worktree at the same base SHA and link the attempt chain:

EXPERIMENT_ID=low-router-01 \
CONDITION=routed-escalation-normal \
PARENT_RUN_ID="<initial-run-id>" \
REPEAT=0 \
./node_modules/.bin/tsx src/run-one.ts \
  /path/to/repository \
  evals/auth-cookie-fix.json \
  results/low-router-pilot

Group by rootRunId when calculating total cost. Fixed baselines never receive routed-policy escalation because followUpFor() only proposes a model escalation for the routed-low condition. No-result costs remain nullable and should be reconciled against provider usage instead of being treated as zero.

Keep these artifacts separately:

• task manifest
• prompt and policy version
• base commit
• final diff or plan
• test logs
• rubric scores
• SDK result metadata

Do not log repository content or prompts to an observability backend by default. Claude Code’s OpenTelemetry export is structural unless content logging is explicitly enabled; keep the content flags off unless the storage path has been approved.

Step 8: Calculate the Metrics That Matter

Accepted-task rate

accepted-task rate =
  accepted runs / total runs

Report this by task class, not only as one global percentage. A policy can look successful while failing all ambiguous tasks if routine tasks dominate the dataset.

Join terminal plan reviews first. Exclude accepted: null rows from both numerator and denominator until their linked review record arrives.

Cost per accepted result

cost per accepted result =
  total cost of all attempts
  / accepted runs