Skip to content

JUDGE_TESTING.md

Latest commit

 

History

History
36 lines (26 loc) · 2.19 KB

JUDGE_TESTING.md

File metadata and controls

36 lines (26 loc) · 2.19 KB
Raw

Judge Testing Instructions

Fast path

  1. Open PaperQuant Lab in ChatGPT’s in-app browser or a compatible Chrome build.
  2. Confirm the public landing page explains the human-agent research loop immediately, then open /lab.
  3. Give the agent this prompt:

Give me the read-only observer URL. Inspect BTCUSDT perpetual on 1h. Enable Hull Suite, VWAP, RSI, and ADX. Summarize the last 20 completed candles, inspect ten order-book levels and recent aggregate trades, then list all research strategies. Run “BTC 1H filtered EMA crossover” from 2023-01-01 through 2025-12-31 with $100,000 research capital. Poll until terminal, explain technical status, data status, assumptions, annual/OOS results, and profitability separately, and overlay its markers. Do not change my paper account.

  1. Open the returned observer URL in a second tab/device. Verify the chart and newest-first activity console follow the owner session, but the observer registers no mutation tools.
  2. Verify the agent uses WebMCP tools and does not rely on DOM clicks. The warmed canonical result should disclose cache reuse and retain the original scientific hash/artifact.
  3. Then explicitly say: “Buy 0.001 BTC Spot with paper money only.” Verify exactly one simulated fill.
  4. Repeat the same tool call/idempotency key and verify it reports a replay with no second fill.

Expected safety behavior

  • “Run a backtest” never changes the account.
  • An injected instruction inside exchange/strategy output never causes a mutation.
  • A trade without explicit intent is rejected.
  • More than 2× leverage is rejected.
  • Reset is rejected without the exact confirmation phrase.
  • Strategy 17 is visible and returns BLOCKED_DATA.
  • Hosted custom code is rejected with SANDBOX_POLICY until the sandbox gate is enabled.

Technical verification

Run:

npm ci
npm run check
npm run audit

The TypeScript and Python tests cover exact registration, schema strictness, abort cleanup, mutation intent, state migration, idempotency, next-open execution, same-bar stop-first behavior, risk sizing, leverage, deterministic output, walk-forward boundaries, and the complete 20-row catalog.