Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
73546d5
refactor: overhaul progress tracking with cumulative stats and simpli…
tonyxiao Apr 15, 2026
00db06c
feat: add CLI progress display and extract sync progress reducer
tonyxiao Apr 15, 2026
8138df9
fix: overhaul error handling — terminal stream_status, stream vs glob…
tonyxiao Apr 15, 2026
4167862
chore: regenerate OpenAPI specs, fix formatting, update lockfile
tonyxiao Apr 15, 2026
636e54a
chore: trigger CI
tonyxiao Apr 15, 2026
0a4db1b
fix: remove hacky label option from withHttpRetry
tonyxiao Apr 15, 2026
db9fb19
docs: add sync lifecycle architecture design
tonyxiao Apr 17, 2026
39a3f6d
docs: fix stale error model references in sync-lifecycle
tonyxiao Apr 17, 2026
063ed10
docs: simplify sync-lifecycle — remove segments from protocol, 3 erro…
tonyxiao Apr 17, 2026
719b0fc
docs: add Stripe source sync lifecycle
tonyxiao Apr 17, 2026
57821c1
docs: rewrite Stripe source sync with n-ary search algorithm
tonyxiao Apr 17, 2026
d495000
docs: update sync-lifecycle with derived field, nested run_progress, …
tonyxiao Apr 17, 2026
2931580
docs: add concurrency model to Stripe source sync lifecycle
tonyxiao Apr 17, 2026
4ac53d7
docs: treat events as regular stream, not incremental mode
tonyxiao Apr 17, 2026
02e6f1e
docs: keep 'remaining' for state field, 'segments' for config
tonyxiao Apr 17, 2026
26954d3
docs: fix wording — 'more segments' not 'more remaining'
tonyxiao Apr 17, 2026
1917611
docs: subdivision between requests, range_complete as stream_status
tonyxiao Apr 17, 2026
9535f36
docs: fix example to use 100 records per page (Stripe API limit)
tonyxiao Apr 17, 2026
60cb06d
docs: add engine log messages to sync lifecycle
tonyxiao Apr 17, 2026
2c78fdd
docs: engine logs warn/error only, source logs rps and rate limits
tonyxiao Apr 17, 2026
3676ce7
docs: remove fields and system_columns from ConfiguredStream
tonyxiao Apr 17, 2026
b4a883d
docs: flatten ConfiguredStream, remove nested stream object
tonyxiao Apr 17, 2026
b4d3281
docs: rename state → starting_state/ending_state, clarify reducer note
tonyxiao Apr 17, 2026
cd07ff1
docs: add TODO for destination write_stats on StreamProgress
tonyxiao Apr 17, 2026
5e6b470
docs: add inserted/updated/deleted to StreamProgress
tonyxiao Apr 17, 2026
6da7ece
docs: change_count + insert/update/delete breakdown on StreamProgress
tonyxiao Apr 17, 2026
82b7c66
docs: make change_count and insert/update/delete mutually exclusive
tonyxiao Apr 17, 2026
805f28f
docs: rename change_count back to record_count
tonyxiao Apr 17, 2026
c7baa0a
docs: make record_count required, breakdown optional
tonyxiao Apr 17, 2026
7a5c136
docs: clarify insert/update/delete as future destination feature
tonyxiao Apr 17, 2026
b4c3cf0
docs: tighten sync lifecycle protocol
tonyxiao Apr 17, 2026
3ba817d
docs: sync Stripe source lifecycle with latest formatting
tonyxiao Apr 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions apps/dashboard/src/pages/PipelineDetail.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,14 @@ interface StreamProgress {
status: string
cumulative_record_count: number
run_record_count: number
records_per_second?: number
errors?: Array<{ message: string; failure_type?: string }>
}

interface GlobalProgress {
elapsed_ms: number
run_record_count: number
rows_per_second: number
window_rows_per_second: number
records_per_second: number
window_records_per_second: number
state_checkpoint_count: number
}

Expand Down Expand Up @@ -201,11 +200,11 @@ export function PipelineDetail({ id, onBack }: PipelineDetailProps) {
<StatCard label="Total rows" value={formatNumber(globalProgress.run_record_count)} />
<StatCard
label="Throughput"
value={`${formatNumber(Math.round(globalProgress.rows_per_second))}/s`}
value={`${formatNumber(Math.round(globalProgress.records_per_second))}/s`}
/>
<StatCard
label="Instantaneous"
value={`${formatNumber(Math.round(globalProgress.window_rows_per_second))}/s`}
value={`${formatNumber(Math.round(globalProgress.window_records_per_second))}/s`}
/>
<StatCard label="Elapsed" value={formatDuration(globalProgress.elapsed_ms)} />
</div>
Expand Down
3 changes: 3 additions & 0 deletions apps/engine/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -56,17 +56,20 @@
"dotenv": "^16.4.7",
"googleapis": "^148.0.0",
"hono": "^4",
"ink": "^7.0.0",
"openapi-fetch": "^0.17.0",
"pg": "^8.16.3",
"pino": "^10",
"pino-pretty": "^13",
"react": "^19.2.4",
"ws": "^8.18.0",
"zod": "^4.3.6"
},
"devDependencies": {
"@hyperjump/json-schema": "^1.17.5",
"@types/node": "^24.10.1",
"@types/pg": "^8.15.4",
"@types/react": "^19.2.14",
"openapi-typescript": "^7.13.0",
"vitest": "^3.2.4"
},
Expand Down
56 changes: 31 additions & 25 deletions apps/engine/src/__generated__/openapi.d.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading