diff --git a/docs/real_robot_bringup.md b/docs/real_robot_bringup.md new file mode 100644 index 000000000..7ec3077d3 --- /dev/null +++ b/docs/real_robot_bringup.md @@ -0,0 +1,320 @@ +# Real-Robot Bring-Up: from a captured scene to closed-loop active learning + +A staged procedure for bringing the domino scene up on the real Franka. The +organizing principle is that **each stage adds exactly one new source of +failure**, so when something breaks you know what caused it. Run them in order; +do not skip ahead to save time, because a failure three stages later is much +harder to attribute. + +Nothing below has been validated on hardware. The pass criteria are what to +check, not results anyone has seen. Two of the stages exist specifically to +replace guessed defaults with measurements — see +[Numbers to calibrate](#numbers-to-calibrate). + +## The pieces + +| Piece | Where | Role | +| --- | --- | --- | +| `PyBulletDominoRealEnv` | `predicators/envs/pybullet_domino_real.py` | Pure simulation. Sizes itself from a captured scene; converts perception into `State` / `EnvironmentTask`. Holds no robot. | +| `RealRobotExecutor` | `predicators/pybullet_helpers/real_robot_executor.py` | Attached to the env. Ships each option's trajectory to the arm, looks at the scene, writes what it saw into the simulated **twin**. | +| `RealRobot` | babyrobot submodule | The arm and the cameras, in-process. | +| `replay_plan.py` | `scripts/domino_debug/replay_plan.py` | Debug tool: replays one exact recorded plan. No planning, no LLM. | + +The env stays pure sim throughout; "real mode" is a property of an executor +being attached, which happens only at `main.setup_environment`. Bilevel search +cannot reach the executor (`PyBulletEnv.simulate` bypasses it), so planning +never moves the arm. + +## The flags that matter + +| Flag | Default | Meaning | +| --- | --- | --- | +| `real_robot_execute` | `False` | Attach an executor at all. Everything else is ignored when this is off. | +| `real_robot_dry` | `False` | Build the `RealRobot` with no arm: every call runs, nothing moves. | +| `real_robot_perception` | `"zed"` | `"zed"` live cameras, `"scene_file"` replays `domino_real_scene`, `"none"` blind. | +| `real_robot_observe_at_option_boundary` | `True` | Look at the scene between options and correct the twin. | +| `real_robot_human_reset` | `True` | Between episodes, home the arm, wait for a human, then rebuild the task from what the cameras see. | +| `real_robot_settle_s` | `0.5` | Dwell before a capture, so dominoes come to rest. | +| `real_robot_divergence_atol` | `0.02` | Metres of twin-vs-scene disagreement worth logging. | + +Always run through a launcher config: + +```bash +PYTHONHASHSEED=0 python scripts/local/launch_simp.py \ + -c predicatorv3/.yaml +``` + +**Do not drive `main.py` directly.** A bare +`python predicators/main.py --env pybullet_domino_real --approach oracle` +inherits none of this env's flags from `envs/all.yaml`, and the env does not +work on the `settings.py` defaults. Most sharply, +`domino_use_domino_blocks_as_target` defaults `False`, which sizes the domino +component with the target held as a separate object -- so a 4-domino scene +allocates 3 slots and task construction dies with + +``` +AssertionError: perceived 4 dominoes but only 3 slots +``` + +To change a flag for a stage, copy the stage's launcher and edit the copy, or +add the flag to its `ENVS: domino_real: FLAGS:` block. + +### Two traps worth knowing before you start + +**The scene path must be set in the launcher, not in `settings.py`.** +`envs/all.yaml` already sets `domino_real_scene`, and a config value beats the +`settings.py` default -- so editing `settings.py` has no effect on a launcher +run. Point the launcher at the scene you captured in Stage 0. + +**`import predicators` resolves to the editable install, not your cwd.** The +package is installed with `pip install -e` against one checkout, so running +`python scripts/local/launch_simp.py` from a *different* worktree still +imports that checkout's code -- `sys.path[0]` is the script's directory, not +where you launched from. Symptom: a fix you just made appears to do nothing. +Check which code is actually loaded with + +```bash +python -c "import predicators, os; print(os.path.dirname(predicators.__file__))" +``` + +and either check the branch out in *that* directory, or set `PYTHONPATH` to +the tree you mean. + +--- + +## Stage overview + +| # | Arm | Cameras | Human | What it proves | +| --- | --- | --- | --- | --- | +| 0 | — | — | — | A scene exists and its roles resolve | +| 1 | — | — | — | The scene is solvable at all, in sim | +| 2 | dry | replay | — | Executor plumbing survives a full episode | +| 3 | dry | **live** | — | Perception maps to the twin correctly | +| 4 | dry | live | yes | Reset cadence and task rebuild | +| 5 | **moves** | — | — | Per-option shipping at the hand | +| 6 | moves | — | — | A whole recorded plan | +| 7 | moves | live | — | The closed loop | +| 8 | moves | live | yes | Real active learning | + +--- + +## Stage 0 — a scene to work from + +```bash +babyrobot scene capture --out scenes/domino_real_00NN.json +``` + +Point `domino_real_scene` at it and set `domino_real_start_id` / +`domino_real_target_id` to the green (start) and purple (target) dominoes. The +raw capture carries no roles; they are assigned by id. + +**Pass:** the env builds and allocates exactly one target. You can skip this +stage and reuse `domino_real_0000.json`. + +## Stage 1 — sim only, oracle approach + +Nothing real: `real_robot_execute` stays `False`, so no executor is attached +and no arm moves. + +```bash +PYTHONHASHSEED=0 python scripts/local/launch_simp.py \ + -c predicatorv3/oracle_domino_real.yaml +``` + +Point that launcher's `domino_real_scene` at the scene you captured. + +**Proves:** the captured scene is *solvable at all* — reach, geometry, the +base→world transplant, and the `Toppled(target)` goal. + +**Pass:** a plan is found and the rollout topples the target in sim. + +**Read the init atoms before believing a pass.** If the goal is already +satisfied in the initial state, the planner returns `Found Plan of length 0` +and the run reports `SOLVED` having done nothing: + +``` +Task init atoms: [... Toppled(domino_0:domino) ...] +Found Plan of length 0 +Task 1 / 1: SOLVED +``` + +No domino should start toppled. Every domino should appear as `Upright`. + +**This is the gate for everything after it.** If oracle cannot solve the +captured scene, stop: no amount of hardware work fixes an unsolvable scene. +Re-capture with the dominoes closer together, or reconsider the layout. + +Before concluding it is unsolvable, know what the oracle is working with. It +plans over hand-written *processes* (`ground_truth_models/domino/processes.py`) +— endogenous ones bound to options (`PushStartBlock`, `PickDomino`, +`PlaceDomino`, `Wait`) plus two exogenous ones that *are* the cascade: a +domino in front of a tilting one starts tilting, and a tilting one becomes +`Toppled`. It also always merges in the ground-truth grid helpers, so it plans +over a **lattice** of `loc` and `angle` values — which is why the init atoms +are full of `loc_0.92_1.29` and `ang_90`. + +That lattice is the likeliest reason a real capture fails here: the scene is +continuous and gets snapped onto it. `envs/all.yaml` hints at the same thing +with `max_initial_demos: 0 # no grid-oracle demos (can't solve the real +scene)` — that note is about the demo generator rather than this approach, but +it is the same underlying friction. + +So distinguish the two failure modes: + +- **no plan found** — most likely the lattice/reachability mismatch. That is a + scene problem: re-capture, do not start debugging code. +- **a plan found but the rollout fails** — the symbolic cascade disagrees with + the physics, which is genuinely informative and worth chasing. + +## Stage 2 — dry executor, replayed perception + +First time the executor runs inside the real pipeline. No arm, no cameras. + +``` +real_robot_execute: True +real_robot_dry: True +real_robot_perception: "scene_file" +real_robot_human_reset: False +``` + +**Proves:** attachment, per-option chunking, the twin sync and the gripper +split all survive a full episode in the stock loop. + +**Pass:** the episode completes, and the log shows one look per option +boundary. + +**Expected failure:** shape mismatches between predicators and babyrobot +message types. + +File perception always replays the capture, so divergence reads ≈0 and no +topple is ever reported. That is expected here — this stage is plumbing, not +perception. + +## Stage 3 — dry executor, live cameras, no motion + +Flip `real_robot_perception: "zed"`. Still `real_robot_dry: True`. + +**Proves:** the ZED path works end-to-end through predicators' conversion — +capture ids map to component slots, and poses land where they should. + +**This is the highest-value stage before any motion.** Do all three checks: + +1. **Static.** Run against an untouched scene. Divergence should sit at the + perception noise floor. Write the numbers down; they set + `real_robot_divergence_atol`. +2. **Moved.** Physically move one domino ≈5 cm and re-run. Divergence should + report ≈0.05 m, and the twin should move *that* domino, not another one. A + wrong domino moving means the id→slot mapping is wrong. +3. **Toppled.** Lay one domino on its face. `Toppled` should become true for + it. The toppled-pose reading is verified offline against captures `0000` + and `0001` but has never seen live data. + +**Watch for a constant z offset**, which means a table-height disagreement. +predicators passes `domino_real_table_z` (−0.041) into `DominoPerception`, +deliberately overriding babyrobot's own default (−0.045). Confirm the value in +the config is the calibrated one for your table. + +## Stage 4 — dry executor, human resets + +Add `real_robot_human_reset: True` and run two episodes. + +**Proves:** the prompt cadence and the task rebuild. + +**Pass:** exactly one prompt per episode, and each episode's initial state +reflects how you just arranged the dominoes — not the capture. `executor.resets_done` +should equal the episode count. + +**Also verify the safety property here:** the arm homes *before* the prompt +appears, i.e. before a human reaches into the workspace. In dry mode nothing +moves, so confirm the ordering in the log; you are checking it now so that it +is already trusted at Stage 8. + +## Stage 5 — first motion: one option, open loop + +``` +real_robot_dry: False +real_robot_observe_at_option_boundary: False +real_robot_human_reset: False +``` + +Use `replay_plan` with a short known-good plan — ideally a single `Push`. + +```bash +PYTHONPATH=. python scripts/domino_debug/replay_plan.py --plan plan.txt --execute +``` + +**This is the riskiest step in the plan, and the one thing that has never +run.** Every prior hardware session shipped the whole episode in one call; +per-option shipping is new. What makes it safe is `RealRobot`'s session-wide +gripper dedup: a `Place` chunk re-emits a leading `close`, and the dedup is +what stops it force-grasping a domino it is already holding. + +**Verify at the hand, not in the log.** Hand on the e-stop. Clear the table of +anything you are not willing to have knocked over. + +Then extend to a `Pick` → `Place` pair, which is the exact case the dedup +exists for: watch that the gripper closes once, carries, and releases. + +## Stage 6 — arm, full recorded plan + +Same settings, the whole plan. + +**Proves:** multi-option shipping and the drift guard across a full episode. + +**Pass:** the arm completes without the drift guard tripping between options. +A trip here means the arm did not end an option where the twin thought it did. + +## Stage 7 — close the loop + +`real_robot_observe_at_option_boundary: True`. Still `replay_plan`, still no +human reset. + +**Proves:** looking mid-episode does not destabilize execution — the twin is +corrected between options and the next option still starts from a sane state. + +**Pass:** the episode completes and per-boundary divergence stays near the +noise floor from Stage 3. + +If divergence is routinely above tolerance on a scene nothing touched, either +`real_robot_settle_s` (0.5 s) is too short for the dominoes to come to rest, or +perception is noisier than Stage 3 suggested. Both are measurable — do not +just raise the tolerance until the warning stops. + +## Stage 8 — real active learning + +`predicatorv3/exp_domino_real.yaml` as shipped: 2 online-learning cycles × 1 +seed, human resets on, closed loop on. + +**Pass:** the run completes, one human reset per episode, and the learner +consumes *perceived* transitions rather than the simulator's guesses. + +**Budget:** on the order of a couple of hours. Every episode costs a physical +reset by a human, which is why the cycle and seed counts are cut from +`common.yaml`'s 10 × 3. + +--- + +## Numbers to calibrate + +Every other stage is pass/fail. These two produce numbers, and both currently +ship as guesses: + +| Setting | Ships as | Set it from | +| --- | --- | --- | +| `real_robot_divergence_atol` | 0.02 m | Stage 3's static noise floor, with headroom | +| `real_robot_settle_s` | 0.5 s | Stage 7: the shortest dwell at which post-motion divergence returns to the noise floor | + +## If something goes wrong + +- **Wrong domino moves in the twin** — id→slot mapping. The env maps capture + ids to slots through `self._scene_ids`, in scene order. A re-capture that + reorders or renumbers dominoes invalidates the configured start/target ids. +- **Constant z offset** — table height. See Stage 3. +- **Gripper stuck closed after a `Place`** — the gripper dedup. This is the + failure per-option shipping was designed around; capture the segment log and + check whether a second `close` reached the hand. +- **Drift guard trips between options** — the arm did not arrive where the twin + predicted. Look at the last shipped chunk's final waypoint against the arm's + reported joints. +- **A human reset prompts twice in one episode** — `_reset_pending` bookkeeping. + It is set on `after_reset`, so exactly one prompt should be owed per episode.