Skip to content

feat: progress tracking, error classification, and CLI UI (rebased on n-ary search)#308

Closed
tonyxiao wants to merge 34 commits intotx/nary-search-paginationfrom
tx/progress-on-nary
Closed

feat: progress tracking, error classification, and CLI UI (rebased on n-ary search)#308
tonyxiao wants to merge 34 commits intotx/nary-search-paginationfrom
tx/progress-on-nary

Conversation

@tonyxiao
Copy link
Copy Markdown
Collaborator

@tonyxiao tonyxiao commented Apr 17, 2026

Summary

Merges the progress tracking / error classification / CLI UI work from PR #296 onto the n-ary search pagination branch (PR #307), reconciling both feature sets cleanly.

Key reconciliation decisions:

  • Simplified status model: TraceStreamStatus.status narrowed from ['start', 'running', 'complete', 'range_complete'] to ['started', 'complete']. Range completions use a separate range_complete field with lifecycle status: 'started', making lifecycle and ranges orthogonal.
  • Errors orthogonal to lifecycle: Non-global errors emit stream_status: complete after the error trace. Global errors (auth 401/403) return early without complete emission.
  • Merged progress tracking: Both PR 307's completedRanges / mergeRanges utility and PR 296's change-driven emitIfStatusChanged / CatalogMessage / TraceGlobalProgress coexist cleanly.
  • Error classification: globalPermanent vs streamPermanent — only global permanent errors park the Temporal workflow.

What's new (from PR #296):

  • sync-progress-state.ts — pure reducer for processing SyncOutput into EofPayload
  • sync-ui.tsx — Ink/React CLI progress display
  • --progress CLI flag for interactive terminal output
  • CatalogMessage emitted at sync start so UI knows all streams upfront
  • TraceGlobalProgress with cumulative stats across runs
  • renderSyncProgress replaces verbose formatEof in HTTP API

Test plan

  • pnpm build passes cleanly
  • @stripe/sync-protocol — 50/50 tests pass
  • @stripe/sync-source-stripe — 102/102 tests pass
  • @stripe/sync-engine — 231/231 tests pass (Docker-dependent sync.test.ts excluded)
  • @stripe/sync-service — 26/26 tests pass (2 skipped, Docker-dependent)
  • E2E tests with Docker + Stripe keys

Made with Cursor

tonyxiao and others added 30 commits April 16, 2026 22:54
@tonyxiao @claude
refactor: overhaul progress tracking with cumulative stats and simpli…
73546d5 
…fied stream status

Simplify stream lifecycle to just started/complete — errors are now orthogonal
(a stream can be complete with errors). Replace interval-based progress emission
with event-driven emission on status transitions. Add cumulative tracking across
runs (record count, request count, elapsed time) persisted via engine state.
Rename TraceProgress → TraceGlobalProgress and rows_per_second → records_per_second
for consistency. Emit catalog as first message so the UI knows all streams upfront.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
feat: add CLI progress display and extract sync progress reducer
00db06c 
Extract sync display state as a pure reducer over SyncOutput messages
that accumulates into an EofPayload shape. The renderer is a stateless
function of (EofPayload, catalog) -> string[].

- Add --progress flag to sync command (auto-enabled on TTY)
- Add --base-url flag for QA/non-prod Stripe API endpoints
- Extract sync-progress-state.ts: createSyncDisplayState() reducer +
  renderSyncProgress() renderer with emoji status groups
- Deduplicate: app.ts formatEof now delegates to renderSyncProgress
- Suppress raw source stream_status traces in trackProgress (engine
  re-emits enriched versions)
- Add scripts/test-all-accounts.sh for multi-account testing

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
fix: overhaul error handling — terminal stream_status, stream vs glob…
8138df9 
…al errors, system_error reclassification

Source always emits stream_status:complete on error (errors are orthogonal to
lifecycle). Adds 'Unrecognized request URL' to skippable patterns for treasury.
Stream-scoped permanent errors no longer park the entire workflow — only global
permanent errors (bad API key, invalid config) do. Reclassifies system_error
using isRetryableHttpError: 429/5xx/network→transient, everything else→permanent.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
chore: regenerate OpenAPI specs, fix formatting, update lockfile
4167862 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao
chore: trigger CI
636e54a 
Committed-By-Agent: claude
@tonyxiao @claude
fix: remove hacky label option from withHttpRetry
0a4db1b 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: add sync lifecycle architecture design
db9fb19 
Introduces the protocol design for sync runs, covering:
- start/end messages replacing EOF-based lifecycle
- Engine-managed time ranges with binary search subdivision
- Segment-level progress tracking with merged synced_ranges
- Error levels as discriminated union (global/stream/segment/transient)
- ProgressPayload as the unified shape for run and request stats
- Source state simplified to pure cursors, engine owns range tracking
- Frozen upper bounds via sync_run_id continuations

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: fix stale error model references in sync-lifecycle
39a3f6d 
Update remaining failure_type references to error_level discriminated
union. Fix field names and remove errors from segment example (errors
live on ProgressPayload, not inline on segments).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: simplify sync-lifecycle — remove segments from protocol, 3 erro…
063ed10 
…r levels

- Segments are source-internal; engine only tracks completed_ranges
- completed_ranges derived from source_state messages with time_range
- Error levels reduced to global/stream/transient (no segment level)
- SyncError is discriminated union on error_level
- completed_ranges optional on StreamProgress
- Rates (records_per_second, states_per_second) nested under rates field
- source_state messages carry time_range for engine range tracking

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: add Stripe source sync lifecycle
719b0fc 
Describes how the Stripe source manages pagination within engine-assigned
time ranges: initialization, density probing, sub-range splitting, cursor
tracking, resumption, subdivision, and completion signaling.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: rewrite Stripe source sync with n-ary search algorithm
57821c1 
- Remove density probing — source starts with full range and subdivides
- Rename ranges → remaining (just the work left to do)
- Describe subdivision: split unpaginated portion of a range into N parts
- Full walkthrough example showing subdivision across requests
- Cursor tracking: null = not started, string = resume point, removed = done

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: update sync-lifecycle with derived field, nested run_progress, …
d495000 
…flat rates

- EngineState contains run_progress: ProgressPayload (not flattened)
- completed_ranges optional on StreamProgress
- Rates flat on ProgressPayload (records_per_second, states_per_second)
- Remove old sync-lifecycle-source-stripe.md from docs/ root (moved to docs/engine/)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: add concurrency model to Stripe source sync lifecycle
2931580 
- max_concurrent_streams (configurable, default 5, capped at catalog size)
- max_requests_per_second (inferred: live=20, test=10)
- max_segments_per_stream (derived: rps / effective_streams)
- Examples showing budget distribution across different scenarios
- Single-stream syncs get the full rate limit budget

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: treat events as regular stream, not incremental mode
4ac53d7 
Events endpoint uses the same time_range + remaining model as all other
streams. Live event polling is experimental and opt-in, stored in
source.global separately from backfill cursor logic.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: keep 'remaining' for state field, 'segments' for config
02e6f1e 
remaining describes the work left to do in source state.
max_segments_per_stream is the config limit on fan-out.
Different concepts, different names.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: fix wording — 'more segments' not 'more remaining'
26954d3 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: subdivision between requests, range_complete as stream_status
1917611 
- Subdivision happens between requests, not mid-request
- Source subdivides if a range didn't complete in previous request
- range_complete is a stream_status subtype (Stripe polymorphism pattern)
- Full example shows actual messages emitted across 3 requests
- stream_status uses 'start' not 'started'

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: fix example to use 100 records per page (Stripe API limit)
9535f36 
Each Stripe API page returns max 100 records. Example now shows
page-by-page pagination with state checkpoints after each page.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: add engine log messages to sync lifecycle
60cb06d 
Engine emits log messages (info/warn/error) alongside progress.
Tolerant processing — anomalies are logged as warnings, not rejected.
Defines the full set of engine log messages by level.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: engine logs warn/error only, source logs rps and rate limits
2c78fdd 
Engine: no redundant info logs (progress stream covers that). Only
warn (anomalies) and error (failures).

Source: emits info logs for real-time rps, warn logs for rate limits
and retries.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: remove fields and system_columns from ConfiguredStream
3676ce7 
fields is redundant with json_schema (use schema to express field
projection). system_columns is a destination concern, not protocol.

Also made time_range.gte optional (omit for "from the beginning").

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: flatten ConfiguredStream, remove nested stream object
b4a883d 
ConfiguredStream.stream.name → ConfiguredStream.name. No more nesting
of Stream inside ConfiguredStream.

TODO: move metadata (api_version, account_id, live_mode) out of
per-stream into source_config or destination injection.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: rename state → starting_state/ending_state, clarify reducer note
b4d3281 
- StartPayload.state → starting_state
- EndPayload.state → ending_state
- Round-trip is self-documenting: end.ending_state → start.starting_state
- Progress: client generally doesn't need a reducer, but can diff for deltas

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: add TODO for destination write_stats on StreamProgress
cd07ff1 
Future: destination can report inserted/updated/deleted counts per
stream (e.g. Postgres upsert). Extension point noted in StreamProgress.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: add inserted/updated/deleted to StreamProgress
5e6b470 
All three are first-class fields. For now inserted = record_count,
updated = 0, deleted = 0. Destination can report real values when
it supports upsert/delete tracking.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: change_count + insert/update/delete breakdown on StreamProgress
6da7ece 
change_count = total records processed (always known by engine).
insert_count + update_count + delete_count = change_count when
destination reports the breakdown. All zero until then.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: make change_count and insert/update/delete mutually exclusive
82b7c66 
Either change_count (when breakdown unavailable) or insert_count +
update_count + delete_count (when destination reports). All optional.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: rename change_count back to record_count
805f28f 
It's a count of record messages, not an interpretation of what they mean.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: make record_count required, breakdown optional
c7baa0a 
record_count is always known (engine counts record messages).
insert/update/delete are optional enrichment from the destination.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao @claude
docs: clarify insert/update/delete as future destination feature
7a5c136 
Reserved for when destinations report per-operation counts.
Not implemented yet.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
tonyxiao and others added 4 commits April 17, 2026 07:27
@tonyxiao @codex
docs: tighten sync lifecycle protocol
b4c3cf0 
Committed-By-Agent: codex
Co-authored-by: codex <noreply@openai.com>
@tonyxiao @claude
docs: sync Stripe source lifecycle with latest formatting
3ba817d 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Committed-By-Agent: claude
@tonyxiao
feat: merge progress tracking, error classification, and CLI UI from PR
7cf9081 
#296

Merge the "massive progress fix" (PR #296) onto the n-ary search pagination
branch (PR #307), reconciling the two feature sets:

Protocol:
- Simplify TraceStreamStatus.status to ['started', 'complete']
- Errors are orthogonal to lifecycle — a stream can be complete with errors
- range_complete is now a separate field on TraceStreamStatus, not a status value
- Add CatalogMessage to SyncOutput, TraceGlobalProgress with cumulative stats

Engine:
- Progress tracker emits catalog upfront, stream_status + global_progress
  pairs on transitions, and tracks completed_ranges via mergeRanges
- New sync-progress-state.ts reducer and sync-ui.tsx Ink/React CLI component
- CLI gains --progress flag for interactive terminal progress display

Service:
- Error classification distinguishes globalPermanent vs streamPermanent
- Only global permanent errors (bad API key, invalid config) park the workflow
- Stream-scoped permanent errors skip the stream on resume

Source:
- Remove label from HttpRetryOptions
- Use isRetryableHttpError for failure type classification
- Emit complete status after non-global errors (errors orthogonal to lifecycle)
- Add 'Unrecognized request URL' to skippable error patterns

Made-with: Cursor
Committed-By-Agent: cursor
Made-with: Cursor
Committed-By-Agent: cursor
@tonyxiao
docs: keep lifecycle rollout on eof semantics
8d8f4b3 
Keep the lifecycle docs aligned with the current rollout by preserving `eof` as
the backward-compatible terminal message and moving the explicit `start` / `end`
envelope migration into its own follow-up plan.

Constraint: Preserve current eof-based sync APIs while documenting continuation via has_more
Rejected: Keep start/end in the same lifecycle spec | mixed current behavior with a future migration
Confidence: high
Scope-risk: narrow
Not-tested: Runtime behavior; docs-only changes
Made-with: Cursor
Committed-By-Agent: cursor
@tonyxiao tonyxiao force-pushed the tx/nary-search-pagination branch from 44c421b to 58dfc8a Compare April 20, 2026 02:06
@tonyxiao tonyxiao closed this Apr 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tonyxiao