feat(eval): read-only batch-evaluation TUI (list + get) - #1938
Conversation
3ec32c9 to
30585a8
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## refactor #1938 +/- ##
============================================
+ Coverage 96.71% 96.74% +0.02%
============================================
Files 296 300 +4
Lines 16209 16331 +122
============================================
+ Hits 15677 15799 +122
Misses 532 532 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Wire batch-evaluation into the eval TUI, mirroring evaluator/online-eval:
- BatchEvaluationPicker (PaginatedTablePicker over listBatchEvaluations) +
BatchEvaluationListScreen; enter on a row opens the job.
- BatchEvaluationGetJsonScreen — raw JSON only (no metadata hub); unwraps
getBatchEvaluation's { detail } (metadata + merged CloudWatch results).
- BatchEvaluationScreen menu (RouterScreen); the menu route is mandatory or the
eval menu entry falls through to the HelpScreen catch-all and exits the app.
- index.tsx now opens the TUI on a bare invocation (renderTui +
withTuiOnEmptyFlagsAndArgs) instead of printing help.
- Root.tsx: menu + list + bare-get-redirect + get/:id routes.
Screen tests cover menu, picker (rows/options/redirect/select/empty), and
detail (full JSON, CloudWatch-failure-degrades, retry). 752 pass.
Align the list column with every other picker (harness, runtime, online-eval, evaluator, memory, …): show updatedAt as 'updated UTC' rather than createdAt as 'created UTC'.
30585a8 to
b8e7f9b
Compare
aidandaly24
left a comment
There was a problem hiding this comment.
One comment though it can be a follow up
| const opts = coreOptsFromCtx(ctx); | ||
| return useQuery({ | ||
| queryKey: ["batch-evaluation", opts.region, id], | ||
| queryFn: async () => (await core.eval.getBatchEvaluation(id!, opts)).detail, |
There was a problem hiding this comment.
I may be missing an intended TUI-specific behavior, but should we preserve resultsError here instead of dropping it? A CloudWatch access or read failure currently renders normal job JSON without results, so the user cannot distinguish incomplete data from a job that simply has no results yet. The headless command warns for this reason. Would it make sense to return both values from the query and show a persistent warning above the JSON while still preserving the metadata?
There was a problem hiding this comment.
I'll update now to fix it.
The batch-evaluation get screen dropped resultsError, so a CloudWatch access/read failure rendered identical JSON to a job with no results yet. Keep resultsError from the query and render a persistent warning above the JSON (the TUI equivalent of the CLI's stderr warning). Adds an optional warning prop to JsonDetail.
What
Adds the read-only TUI for
agentcore eval batch-evaluation, wiring it into the eval TUI alongsideevaluatorandonline-eval:list— a paginated, searchable table of batch evaluations (BatchEvaluationPickeroverPaginatedTablePicker). Enter on a row opens that job.get— a raw-JSON detail screen (JsonDetail). No metadata hub: the value is the full response (job metadata + per-sessionresults), which the JSON shows cleanly. UnwrapsgetBatchEvaluation's{ detail }(metadata + merged CloudWatch results).BatchEvaluationScreen(RouterScreen). This route is mandatory: the eval menu already listsbatch-evaluation, and without a route the entry falls through to theHelpScreencatch-all and exits the app.A bare
agentcore eval batch-evaluationnow opens the TUI (renderTui+withTuiOnEmptyFlagsAndArgs) instead of printing help.Demo
Files
src/components/BatchEvaluationPicker.tsx— list body (columns/toRow/loadPage), mirrorsOnlineEvalPicker.src/handlers/eval/batch-evaluation/list/screen.tsx—BatchEvaluationListScreen; enter → get.src/handlers/eval/batch-evaluation/get/screen.tsx—BatchEvaluationGetJsonScreen+useBatchEvaluationDetail.src/handlers/eval/batch-evaluation/screen.tsx—BatchEvaluationScreenmenu.index.tsxre-wired torenderTui;Root.tsxgets the menu + list + bare-get-redirect +get/:idroutes.Testing
batch-evaluation.screen.test.tsx— menu (get/list), picker (rows, Core-options, bare-get redirect, selection → JSON, empty), detail (full JSON incl. merged results, CloudWatch-failure-degrades, retry).