Codex Skill Lab

Codex Skill Lab (csl) is a local workbench for developing and evaluating Codex skills as reproducible experiments. It keeps skill versions, visible case inputs, hidden grading material, execution traces, grades, and comparison reports in one lab directory.

Use it to answer questions such as:

Does a new SKILL.md version improve task success?
Is a result stable across repeated trials?
Which skill sections influenced the agent's behavior?
Did the agent produce the required files without seeing the hidden rubric?
Does a new evaluation campaign regress against a previous one?

Important

Codex Skill Lab no longer offers an unconfined local backend. The default auto mode selects Bubblewrap on Linux/WSL, then Docker, and fails closed when neither secure backend is available.

Note

The Codex skill definition for this project lives at skills/codex-skill-lab/SKILL.md. Codex reads this file to learn the csl workflow, commands, and isolation rules. Keep it in sync with codebase changes.

Features

Scaffold skills or import them from Git repositories.
Create immutable, hashed skill snapshots and compare versions.
Build cases with a strict public/ and hidden/ boundary.
Run codex exec with a temporary workspace and CODEX_HOME.
Capture JSONL traces, summaries, final messages, process results, and artifact hashes.
Combine deterministic assertions, JavaScript graders, and LLM graders.
Run repeated evaluation campaigns with pass rates, dispersion, confidence intervals, and flakiness detection.
Compare individual runs or full evaluation campaigns.
Attribute trace events to skill sections and perform causal section-ablation experiments.
Capture token usage from Codex traces with per-run, per-campaign, and comparison-level statistics including cost estimation with prompt-cache awareness.
Use JSON output and meaningful exit codes in scripts and CI.

Requirements

Node.js 18 or newer.
An installed and authenticated codex CLI, or a compatible command supplied with --codex.
Git when importing a skill with --from-git.
Bubblewrap (bwrap) on Linux/WSL, or a running Docker daemon with the project runner image.

The project has no third-party runtime dependencies.

Installation

Install the published CLI from npm:

npm install --global codex-skill-lab
csl --help

To install from source instead:

git clone https://github.com/Su1furicAcid/codex-skill-lab.git
cd codex-skill-lab
npm install
npm link
csl --help

If Bubblewrap is unavailable, build the Docker runner before running cases:

csl docker build
csl preflight --all --isolation docker

Use --image, --codex-version, --docker, or --pull to customize the build. Source checkouts can also run npm run docker:build -- [options].

npm link makes csl available globally. If you prefer not to link it, replace csl in every example with:

node bin/csl.mjs

Five-Minute Workflow

The following sequence creates a skill, freezes version v1, creates a case, runs it, and grades the evidence.

# 1. Create the lab directory and .csl-lab marker.
csl init

# 2. Create and edit the development copy.
csl skill new my-skill
$EDITOR lab/skills/my-skill/dev/SKILL.md

# 3. Freeze the exact version to test.
csl skill snapshot --name my-skill --version v1

# 4. Create and customize a test case.
csl case new first-case
$EDITOR lab/cases/first-case/public/prompt.md
$EDITOR lab/cases/first-case/hidden/assertions.json
$EDITOR lab/cases/first-case/hidden/grader.mjs
$EDITOR lab/cases/first-case/hidden/grader-prompt.md

# 5. Validate inputs before spending a model call.
csl preflight --case first-case --skill my-skill@v1

# 6. Run the case. The command prints the created run directory/run ID.
csl run first-case --skill my-skill@v1

# 7. Inspect available IDs, then grade and render a report.
csl run list
csl grade <run-id> --graders assertions,js,llm
csl report <run-id>

The tested Codex process receives:

the case prompt;
a workspace populated from cases/<case-id>/public/ under input/;
a temporary CODEX_HOME containing selected skill snapshots plus copied Codex configuration/authentication files.

It does not receive hidden/ grader files. An enforced output schema is the exception: it is passed to codex exec --output-schema, so schemas must never contain secret rubric information.

Core Concepts

Lab

csl init creates a lab named lab by default and writes .csl-lab in the project directory. The marker lets later commands resolve a custom lab name automatically.

csl init --lab my-workbench

All commands also accept --root DIR and --lab NAME. Resolution order is an explicit --lab, the .csl-lab marker, then lab.

Skills and snapshots

The dev directory is the editable skill. Runs should normally target a named snapshot so the tested content is reproducible.

csl skill new my-skill
csl skill snapshot --name my-skill --version v1
csl skill list

# After editing dev and creating v2:
csl skill diff my-skill@v1 my-skill@v2

Snapshot metadata records a content hash and, when available, Git provenance. Diff reports are written under lab/diffs/ in JSON and Markdown.

Import one skill from a local or remote Git repository:

csl skill new imported-skill \
  --from-git https://github.com/example/skills.git \
  --git-ref v1.0.0

If a repository contains multiple SKILL.md files, import all discovered skills:

csl skill new --from-git ../skills-repo --all-skills

Use --group-name NAME with --all-skills to create a composite group. Running one group member automatically installs the other members at the same version when available, falling back to their dev versions.

Cases and visibility

A case defines the task, model settings, timeout, artifact contract, and grading strategy.

cases/<case-id>/
  case.json
  public/                 # visible to the tested process
    prompt.md
    ...input files
  hidden/                 # available only after the run, during grading
    assertions.json
    grader.mjs
    grader-prompt.md
    output.schema.json

Create a scaffold or import existing material:

csl case new fix-parser \
  --require-artifact result.json \
  --allow-artifact result.json

csl case import imported-case \
  --prompt task.md \
  --public input.c \
  --hidden rubric.md \
  --schema schema.json \
  --enforce-output-schema

Repeat --public, --hidden, --require-artifact, or --allow-artifact, or pass comma-separated values. Imported files use their basenames.

requiredArtifacts must exist after the run. When allowedArtifacts is non-empty, any non-input file outside the required/allowed lists fails the artifact contract.

To share a case without leaking grading data:

csl case export-public imported-case
csl case export-public imported-case --out ./handoff --force

The export contains only public/ plus a redacted, hashed case-public.json manifest.

Running and Isolation

csl run <case-id> --skill <name>@<version> [options]

Useful options:

Option	Purpose
`--codex COMMAND`	Use another Codex binary or a compatible wrapper.
`--timeout DURATION`	Override the case timeout; accepts `10s`, `5min`, `1hour`, or milliseconds.
`--model MODEL`	Override the model for this run.
`--reasoning-effort low\|medium\|high`	Override model reasoning effort.
`--isolation auto\|bwrap\|docker`	Select the execution isolation backend.
`--no-preserve-subagent-traces`	Use ephemeral execution; do not persist subagent rollouts.

If the command contains arguments or spaces, pass one shell-style string. It is parsed without invoking a shell:

csl run first-case --skill my-skill@v1 \
  --codex 'node "/path with spaces/fake-codex.mjs"'

Isolation backends

Backend	Availability	Guarantee
`auto`	Default	On Linux/WSL, selects `bwrap` first and Docker second. On other systems, selects Docker. Fails if no secure backend is ready.
`bwrap`	Linux/WSL with Bubblewrap and kernel support	Mount-level filesystem isolation. Only the workspace, temporary `CODEX_HOME`, required runtime paths, and explicit evidence/schema paths are exposed.
`docker`	Any platform with a reachable Docker daemon and built runner image	Read-only container root with dropped capabilities. Only the current workspace, temporary `CODEX_HOME`, and explicit runtime/evidence paths are bind-mounted.

Every explicit backend fails closed when its probe fails. Docker never mounts the lab root, historical runs, or the Docker socket.

csl preflight --case first-case --skill my-skill@v1 --isolation bwrap
csl run first-case --skill my-skill@v1 --isolation bwrap

The Docker runner is built from docker/Dockerfile with Node.js 22, a pinned Codex CLI, and common task tools. Override docker.image for cases that require additional system dependencies. Docker uses bridge networking because Codex must reach the model service; these backends do not provide network isolation.

Existing labs configured with "isolationBackend": "local" must be changed manually. Commands return a migration error rather than silently weakening isolation.

Sandbox path auto-discovery

At run time, csl automatically discovers tool-specific filesystem paths and environment variables that must be propagated into the sandbox. The discovery is entirely general — no tool-specific knowledge is hardcoded.

PATH parsing: Non-standard directories in the host PATH (e.g. ~/.opam/default/bin, ~/.cargo/bin, /opt/conda/bin) are identified. Standard system prefixes (/usr, /bin, /lib, /etc, /proc, /sys, /dev) are skipped because they are already mounted. /tmp is excluded because bwrap creates an empty tmpfs.
Tool root derivation: For each non-standard PATH entry, the minimal tool root is derived by walking up until a dot-directory (.opam, .cargo, .local, .nvm), HOME, /opt, or / is reached. The tool root (not just bin/) is mounted so that lib/, share/, and other sibling directories are available.
Read-only mount at original path: Discovered tool roots are mounted at their original path inside the sandbox (bwrap --ro-bind X X / Docker --mount source=X,target=X,readonly). No path remapping is needed — the host PATH works directly inside the sandbox.
Environment variable scanning: After discovering tool paths, csl scans all host environment variables whose values reference those paths (including colon-separated values). Matching variables (e.g. OPAM_SWITCH_PREFIX, CAML_LD_LIBRARY_PATH) are propagated into the sandbox. Variables whose names match secret patterns (SECRET, TOKEN, PASSWORD, CREDENTIAL, API_KEY, PRIVATE_KEY) are never auto-whitelisted.
Manual overrides: Use config.json sandboxPaths to mount additional host paths (files or directories) read-only at their original path. Use config.json sandboxEnv to pass additional env var names to Docker. These cover cases that auto-discovery cannot handle, such as configuration files under $HOME (e.g. ~/.why3.conf) or tool binaries referenced inside wrapper scripts.

Grading

New cases enable three graders:

Type	Source	Best for
`assertions`	`hidden/assertions.json`	Fast, deterministic checks over process results, messages, artifacts, traces, and security evidence.
`js`	`hidden/grader.mjs`	Custom deterministic logic that inspects run and case files.
`llm`	`hidden/grader-prompt.md`	Semantic judgment that is difficult to express deterministically.

Select a subset at grading or evaluation time:

csl grade <run-id> --graders assertions,js
csl eval --skills my-skill@v1 --cases first-case --graders assertions,js

The legacy --grader prompt alias maps to llm. Prefer deterministic graders for hard pass/fail requirements and use the LLM grader for semantic quality.

Grader definitions live in case.json:

{
  "grading": {
    "graders": [
      { "id": "assertions", "type": "assertions", "file": "hidden/assertions.json", "enabled": true, "weight": 1, "gate": true },
      { "id": "js", "type": "js", "file": "hidden/grader.mjs", "enabled": true, "weight": 1, "gate": true },
      { "id": "llm", "type": "llm", "prompt": "hidden/grader-prompt.md", "enabled": true, "weight": 1, "gate": true }
    ],
    "aggregation": { "mode": "all", "passThreshold": 1 }
  }
}

Aggregation modes are all, any, and weighted threshold. A failed grader with gate: true always fails the aggregate.

Supported assertion types are:

process-exit
final-message
output-schema
artifact
trace-error-count
trace-command
trace-file-write
security-boundary

Every grade also runs framework security checks, including temporary CODEX_HOME contents and detected hidden-boundary indicators. Results use schema version 2 and include the aggregate decision, component scores, metrics, grader summaries, security findings, and a compatibility checks view.

Evaluation Campaigns

Evaluate one or more snapshots against one or more cases:

csl eval \
  --skills my-skill@v1,my-skill@v2 \
  --cases first-case,second-case \
  --repeat 3 \
  --graders assertions,js,llm \
  --id parser-campaign

Repeat precedence is CLI --repeat, case-level repeat, config.json defaultRepeat, then 1. Each trial receives its own run directory and repeat index.

Campaign summaries include pass rate, mean, median, sample standard deviation, min/max, deterministic bootstrap 95% confidence intervals, and flakiness. Compare campaigns to detect regressions:

csl eval compare baseline-campaign parser-campaign

The comparison writes JSON and Markdown into the newer evaluation directory and exits non-zero when it detects a removed row or a regression in pass status, score, or trace errors.

Trace Analysis

Each run writes raw codex exec --json output to trace/events.jsonl and a parsed trace/summary.json.

Subagent trace preservation is enabled by default. Runs use Codex session persistence long enough to copy each newly created spawned-subagent rollout into trace/subagents/<thread-id>.jsonl, with parent/thread metadata in trace/subagents/manifest.json; zstd-compressed rollouts are normalized to JSONL. Codex can prefix a child rollout with inherited parent history, so manifest schema v2 records rolloutEvents, inheritedEvents, subagentHistoryStartLine, and whether that boundary is known. Consumers should use the boundary instead of attributing every raw rollout line to the child. Files that cannot be decoded are retained under trace/subagents/unclassified/ and reported in manifest warnings. Temporary session files and databases created by the run are then removed from the isolated CODEX_HOME without touching baseline files. The parent trace's collab_tool_call items are also summarized under summary.json.subagents. Set preserveSubagentTraces to false, or pass --no-preserve-subagent-traces, to use ephemeral execution and retain only collaboration events exposed by the parent JSON stream.

Attribute events to sections in the tested SKILL.md:

csl trace attribution <run-id> --model <judge-model>

Skill reads are attributed structurally. Other supported events are sent to an LLM judge, which produces section candidates, confidence, and reasoning. The command writes trace/attribution.json and trace/attribution.md.

Test causal influence by removing skill sections one at a time and rerunning:

csl trace intervene <run-id>
csl trace intervene <run-id> --section "Validation" --repeat 3

Only level-two-or-deeper sections are candidates. Each base and ablated trial is regraded, and the result is classified as outcome-impact, behavioral-change, or no-change. Evidence is stored under trace/interventions/, with summaries in trace/intervention.json and trace/intervention.md.

Both commands can invoke additional model runs; intervention can be expensive because it reruns and regrades the case for every selected section.

Token Usage and Cost Estimation

Each run's trace/summary.json includes a usage object extracted from turn.completed events in the Codex JSON stream. The usage data is cumulative across the session and includes:

Field	Description
`inputTokens`	Total input tokens (includes cached and cache-write tokens)
`cachedInputTokens`	Cache hits — tokens served from prompt cache
`cacheWriteInputTokens`	Cache writes — tokens written to prompt cache (GPT-5.6+ only)
`outputTokens`	Total output tokens
`reasoningOutputTokens`	Reasoning tokens (subset of output)

Cost estimation uses built-in OpenAI pricing for all models from GPT-4o through GPT-5.6. The cost model accounts for prompt caching:

Cache hits are billed at the cached input rate (typically 10% of standard input).
Cache writes on GPT-5.6+ models are billed at 1.25x the input rate; on earlier models they are free (already included in standard input billing).
Fresh input (non-cached, non-cache-write) is billed at the standard input rate.

Configure custom pricing in lab/config.json:

{
  "pricing": {
    "custom-model": { "input": 1.0, "cached": 0.1, "cacheWrite": null, "output": 5.0 }
  }
}

Set cacheWrite to null for models without cache-write fees, or to a number for models that charge for cache writes.

Token usage appears in:

trace/summary.json — per-run usage object
report.md — Token Usage section with breakdown and cost
Eval summary.json — totals.usage and per-row usage fields
Eval summary.md — token columns and summary
Eval comparison — token deltas per row and at campaign level
Run comparison — usageDelta in compare JSON
csl run list — total token count per run

Configuration

csl init creates lab/config.json:

{
  "codexCommand": "codex",
  "defaultModel": null,
  "defaultReasoningEffort": null,
  "runSandbox": "workspace-write",
  "skillInstallMode": "copy",
  "copyAuthFiles": ["auth.json"],
  "outputSchemaRequired": false,
  "preserveSubagentTraces": true,
  "isolationBackend": "auto",
  "docker": {
    "command": "docker",
    "image": "codex-skill-lab-runner:0.1.0",
    "network": "bridge",
    "env": ["OPENAI_API_KEY", "OPENAI_BASE_URL", "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY", "PATH"]
  },
  "sandboxPaths": [],
  "sandboxEnv": [],
  "defaultRepeat": 1,
  "profiles": {},
  "pricing": {}
}

Field	Meaning
`codexCommand`	Default executable or wrapper command.
`defaultModel`	Fallback model when no more specific setting is present.
`defaultReasoningEffort`	Fallback reasoning effort.
`runSandbox`	Value forwarded to `codex exec --sandbox`.
`skillInstallMode`	Skill installation mode; current runs copy snapshots into temporary `CODEX_HOME`.
`copyAuthFiles`	Safe, single-segment files copied from the current Codex home. `config.toml` is copied automatically when present.
`outputSchemaRequired`	Pass a configured case schema to every run.
`preserveSubagentTraces`	Persist spawned subagent rollouts under `trace/subagents/`; disable for ephemeral runs.
`isolationBackend`	`auto`, `bwrap`, or `docker`; `auto` prefers bwrap and never falls back to an unsafe process.
`docker`	Docker CLI, runner image, network, and the names of host environment variables allowed into the container. `PATH` is included by default; other tool-specific variables (e.g. `OPAM_SWITCH_PREFIX`) are auto-discovered at runtime. Values are never stored in run metadata.
`sandboxPaths`	Additional host directories to mount read-only at their original path inside the sandbox (both bwrap and Docker). Non-standard PATH entries (e.g. `~/.opam/default/bin`, `~/.cargo/bin`) are auto-discovered and their tool roots are mounted automatically.
`sandboxEnv`	Additional host environment variable names to pass through to the Docker sandbox. Variables referencing auto-discovered paths are detected automatically; use this for any extras.
`defaultRepeat`	Default evaluation repeat count.
`profiles`	Per-command model/reasoning defaults.
`pricing`	Custom model pricing overrides for cost estimation. Keys are model names, values are `{ input, cached, cacheWrite, output }` in USD per 1M tokens.

Example profiles:

{
  "defaultModel": "gpt-5",
  "profiles": {
    "run":   { "model": "gpt-5", "reasoningEffort": "high" },
    "eval":  { "model": "gpt-5", "reasoningEffort": "medium" },
    "trace": { "model": "gpt-5", "reasoningEffort": "medium" },
    "grade": { "model": "gpt-5" }
  }
}

For run, model and reasoning priority is CLI option, case.json, profiles.run, global default, then the Codex default. An LLM grader may set case.json.graderModel; otherwise it uses profiles.grade.model and then defaultModel.

Command Reference

Run csl --help for the authoritative syntax.

Command	Description
`csl init [--root DIR] [--lab NAME]`	Initialize a lab and marker.
`csl skill new <name>`	Scaffold an editable skill under `dev`.
`csl skill new <name> --from-git SOURCE`	Import one skill from Git.
`csl skill new --from-git SOURCE --all-skills`	Discover and import every skill in a repository.
`csl skill snapshot --name NAME [--version ID]`	Create a hashed snapshot; the version defaults to a timestamp.
`csl skill list [--json]`	List skill versions and provenance.
`csl skill diff A@VERSION B@VERSION [--json]`	Compare two snapshots and write reports.
`csl case new ID`	Scaffold a case and three graders.
`csl case import ID [options]`	Build a case from existing files.
`csl case list [--json]`	List cases and artifact contracts.
`csl case export-public ID [--out DIR] [--force]`	Export only shareable case data.
`csl preflight --case ID --skill NAME@VERSION`	Validate selected inputs and isolation availability.
`csl preflight --all`	Validate every case and non-`dev` snapshot.
`csl run ID --skill NAME@VERSION`	Execute one case and preserve evidence.
`csl grade RUN_ID [--graders LIST]`	Grade an existing run.
`csl report RUN_ID`	Write a human-readable `report.md`.
`csl compare RUN_A RUN_B [--json]`	Compare scores, security, traces, and artifacts.
`csl eval --skills LIST --cases LIST`	Run and grade a campaign.
`csl eval compare EVAL_A EVAL_B [--json]`	Compare campaigns and fail on regression.
`csl trace attribution RUN_ID`	Attribute trace events to skill sections.
`csl trace intervene RUN_ID`	Run section-ablation experiments.
`csl docker build [options]`	Build the isolated Docker runner image.
`csl run list [--json]` / `csl eval list [--json]`	List preserved evidence.
`csl run archive ID` / `csl eval archive ID`	Move evidence under `lab/archive/`.
`csl run rm ID --yes` / `csl eval rm ID --yes`	Permanently delete evidence.

Identifiers such as case IDs, run IDs, skill names, and versions must be single path segments. Absolute paths and .. traversal are rejected. Copied directory trees also reject symlinks that escape their source.

Evidence Layout

lab/
  config.json
  skills/<skill-name>/
    dev/
    <version-id>/
  cases/<case-id>/
    case.json
    public/
    hidden/
  runs/<run-id>/
    run.json
    agent-workspace/
    codex-home/
    trace/
      events.jsonl
      summary.json
      subagents/
        manifest.json
        <thread-id>.jsonl
      attribution.json
      intervention.json
      interventions/
    outputs/
      last-message.txt
      process.json
      security-boundary.json
      artifact-manifest.json
    grade/result.json
    report.md
  evals/<eval-id>/
    summary.json
    summary.md
    compare-<eval-id>.json
    compare-<eval-id>.md
  diffs/
  exports/
  archive/

Treat run evidence as immutable. Archive evidence before pruning it; permanent removal intentionally requires --yes.

Automation and CI

List, preflight, compare, and trace commands support --json where shown in csl --help. Commands propagate failure through their exit status: failed runs preserve evidence and return the process status, failed grades/preflights return non-zero, and campaign comparison returns non-zero on regression.

A typical CI sequence is:

csl preflight --all --json
csl eval --skills my-skill@v1 --cases smoke --repeat 3 --graders assertions,js --id ci-eval
csl eval compare accepted-baseline ci-eval --json

Use case export-public before handing case inputs to a subagent, external reviewer, or another CI stage that must not receive hidden graders.

Development

npm test
csl docker build
npm run test:docker
npm run validate:plugin

The unit test suite uses an injected test process runner, so npm test does not require a system sandbox. npm run test:docker executes the real Docker boundary test when the daemon and runner image are available. Plugin validation uses the validator configured in package.json.

Publishing

Publishing is driven by GitHub Releases. Add an npm automation token as the repository secret NPM_TOKEN, update package.json to the intended version, commit and push the change, then publish a GitHub Release whose tag is exactly v<version> (for example, v0.1.0). The workflow verifies the tag, runs the tests and package dry run, and publishes the public package with npm provenance.

For a manual release from an authenticated checkout, run the following. Manual releases disable CI provenance because npm can only generate it inside a supported CI provider.

npm test
npm pack --dry-run
npm publish --access public --provenance=false

Security Notes

Keep secrets and expected answers in hidden/, never in the prompt, skill, or runtime-visible output schema.
Keep auto or select bwrap/docker explicitly; no unsafe local fallback exists.
Never mount the lab root, historical evidence, or Docker socket into custom runner images.
Review grade/result.json security findings even when task-specific graders pass.
Avoid placing extra credentials in the source Codex home. Docker receives only the environment variable names listed in docker.env.
Do not edit trace/events.jsonl, manifests, or grade files after a run if reproducibility matters.

License

MIT. See LICENSE.

Navigation Menu

Repository files navigation

Codex Skill Lab

Features

Requirements

Installation

Five-Minute Workflow

Core Concepts

Lab

Skills and snapshots

Cases and visibility

Running and Isolation

Isolation backends

Sandbox path auto-discovery

Grading

Evaluation Campaigns

Trace Analysis

Token Usage and Cost Estimation

Configuration

Command Reference

Evidence Layout

Automation and CI

Development

Publishing

Security Notes

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 18 Commits
.codex-plugin		.codex-plugin
.github/workflows		.github/workflows
bin		bin
docker		docker
lab		lab
scripts		scripts
skills/codex-skill-lab		skills/codex-skill-lab
src		src
support		support
tests		tests
.csl-lab		.csl-lab
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
README_zh.md		README_zh.md
package.json		package.json

Folders and files

Latest commit

History

Repository files navigation

Codex Skill Lab

Features

Requirements

Installation

Five-Minute Workflow

Core Concepts

Lab

Skills and snapshots

Cases and visibility

Running and Isolation

Isolation backends

Sandbox path auto-discovery

Grading

Evaluation Campaigns

Trace Analysis

Token Usage and Cost Estimation

Configuration

Command Reference

Evidence Layout

Automation and CI

Development

Publishing

Security Notes

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages