W&B Weave traces not landing on scaleai.wandb.io from Modal runs

[shehabyasser-scale]

Summary

The harness build configs set wandb.project: vero-{gaia,tau3,swe-atlas-qna,officeqa} with log_traces: true, and the live secrets.env files carry a valid WANDB_API_KEY (local-...) plus WANDB_BASE_URL=scaleai.wandb.io. Yet no traces are landing:

• count_weave_traces / infer_trace_schema against shehab-yasser/vero-gaia, vero-tau3, vero-swe-atlas-qna all return 403 Project not found.
• The W&B projects API lists 0 projects for the entity.
• Locally, a run's session/artifacts/wandb/state.json shows a run_id was minted (e.g. vero-e79e8ce8593c4504) but evaluation_ids: [], next_step: 0, request_log_files: {}: zero steps/traces recorded.

Impact

Optimizer runs on Modal produce no server-side observability. The harbor summary tables become the only signal, which is what hid the swe-atlas-qna flat-zero (see the companion swe-atlas-qna provisioning issue and taxonomy fix).

To investigate

• Whether Modal egress can reach scaleai.wandb.io (the run mints a local run_id but never creates the project server-side).
• Whether the harbor log_traces path actually flushes Weave (state.json records 0 request_log_files).

This is not a missing-credentials issue: the creds are present in secrets.env.

[shehabyasser-scale]

Root-caused and fixed in #57. Two corrections to this issue as filed:

1. It is not egress, and it is not Weave.

There is no Weave in this codebase — weave is not a dependency and is not importable in the built environment. wandb.log_traces drives SidecarWandbSink._log_trace, which uploads a wandb.Artifact of type evaluation_trace. count_weave_traces would therefore return nothing even on a perfectly healthy run. The right server-side check is whether the project exists and carries runs and artifacts.

2. The cause is a missing URL scheme, reproduced locally at $0.

$ WANDB_BASE_URL=scaleai.wandb.io  python -c "import wandb; wandb.init(project='vero-egress-probe')"
pydantic_core.ValidationError: 1 validation error for Settings
base_url
  Input should be a valid URL, relative URL without a base
    [type=url_parsing, input_value='scaleai.wandb.io', input_type=str]

W&B parses base_url as a URL. That exception comes out of wandb.init(), harbor/deployment.py:281 catches it and continues with wandb_sink = None, and the whole run reports nothing.

It explains the state.json fingerprint exactly: SidecarWandbSink.__init__ writes state at runtime/wandb.py:225 and opens the run at :230, so a run_id with evaluation_ids: [] / next_step: 0 / request_log_files: {} is precisely what a failed init leaves behind — including for a session that completed 11 evaluations.

Prepending https:// and re-running the same script: run created, metric logged, artifact uploaded, and the project auto-created server-side. query_wandb_entity_projects(entity="shehab-yasser") went from [] to [vero-egress-probe]. Egress to scaleai.wandb.io was never blocked.

Why it stayed hidden for so long: the only signal was a logger.warning on the sidecar container's stderr, which no run artifact captures. Grepping every artifact of the failing run for "wandb" returns nothing. #57 therefore also writes the reason to session/artifacts/wandb/init-error.json.

Modal-sandbox egress specifically is still unproven — a live run on the fix branch is in flight and I will post the project URL here.

[shehabyasser-scale]

Confirmed end-to-end from Modal, not just locally: a run started 08:58:16 on the #57 branch created shehab-yasser/vero-swe-atlas-qna at 05:59:21Z (the entity had zero projects before today) and is streaming 59 steps across 47 metric keys, including the per-scope inference counters and budget ledger. Egress was never the problem; it was the missing URL scheme. Details on #57.