Skip to content

feat(eval): read-only batch-evaluation TUI (list + get) - #1938

Open
jariy17 wants to merge 3 commits into
refactorfrom
feat/eval-batch-evaluation-tui
Open

feat(eval): read-only batch-evaluation TUI (list + get)#1938
jariy17 wants to merge 3 commits into
refactorfrom
feat/eval-batch-evaluation-tui

Conversation

@jariy17

@jariy17 jariy17 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

What

Adds the read-only TUI for agentcore eval batch-evaluation, wiring it into the eval TUI alongside evaluator and online-eval:

  • list — a paginated, searchable table of batch evaluations (BatchEvaluationPicker over PaginatedTablePicker). Enter on a row opens that job.
  • get — a raw-JSON detail screen (JsonDetail). No metadata hub: the value is the full response (job metadata + per-session results), which the JSON shows cleanly. Unwraps getBatchEvaluation's { detail } (metadata + merged CloudWatch results).
  • menuBatchEvaluationScreen (RouterScreen). This route is mandatory: the eval menu already lists batch-evaluation, and without a route the entry falls through to the HelpScreen catch-all and exits the app.

A bare agentcore eval batch-evaluation now opens the TUI (renderTui + withTuiOnEmptyFlagsAndArgs) instead of printing help.

Demo

batch-evaluation-tui-demo

Files

  • src/components/BatchEvaluationPicker.tsx — list body (columns/toRow/loadPage), mirrors OnlineEvalPicker.
  • src/handlers/eval/batch-evaluation/list/screen.tsxBatchEvaluationListScreen; enter → get.
  • src/handlers/eval/batch-evaluation/get/screen.tsxBatchEvaluationGetJsonScreen + useBatchEvaluationDetail.
  • src/handlers/eval/batch-evaluation/screen.tsxBatchEvaluationScreen menu.
  • index.tsx re-wired to renderTui; Root.tsx gets the menu + list + bare-get-redirect + get/:id routes.

Testing

  • batch-evaluation.screen.test.tsx — menu (get/list), picker (rows, Core-options, bare-get redirect, selection → JSON, empty), detail (full JSON incl. merged results, CloudWatch-failure-degrades, retry).
  • Full suite: 752 pass, typecheck + lint clean.

@jariy17
jariy17 marked this pull request as ready for review August 6, 2026 21:29
@jariy17
jariy17 force-pushed the feat/eval-batch-evaluation-tui branch from 3ec32c9 to 30585a8 Compare August 7, 2026 19:08
@codecov-commenter

codecov-commenter commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.74%. Comparing base (8cfce32) to head (6958442).
⚠️ Report is 2 commits behind head on refactor.

Additional details and impacted files
@@             Coverage Diff              @@
##           refactor    #1938      +/-   ##
============================================
+ Coverage     96.71%   96.74%   +0.02%     
============================================
  Files           296      300       +4     
  Lines         16209    16331     +122     
============================================
+ Hits          15677    15799     +122     
  Misses          532      532              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jariy17
jariy17 requested review from AdnaneKhan, aidandaly24 and nborges-aws and removed request for AdnaneKhan August 10, 2026 17:40
Base automatically changed from feat/eval-batch-evaluation-readonly to refactor August 10, 2026 18:23
jariy17 added 2 commits August 10, 2026 14:23
Wire batch-evaluation into the eval TUI, mirroring evaluator/online-eval:
- BatchEvaluationPicker (PaginatedTablePicker over listBatchEvaluations) +
  BatchEvaluationListScreen; enter on a row opens the job.
- BatchEvaluationGetJsonScreen — raw JSON only (no metadata hub); unwraps
  getBatchEvaluation's { detail } (metadata + merged CloudWatch results).
- BatchEvaluationScreen menu (RouterScreen); the menu route is mandatory or the
  eval menu entry falls through to the HelpScreen catch-all and exits the app.
- index.tsx now opens the TUI on a bare invocation (renderTui +
  withTuiOnEmptyFlagsAndArgs) instead of printing help.
- Root.tsx: menu + list + bare-get-redirect + get/:id routes.

Screen tests cover menu, picker (rows/options/redirect/select/empty), and
detail (full JSON, CloudWatch-failure-degrades, retry). 752 pass.
Align the list column with every other picker (harness, runtime, online-eval,
evaluator, memory, …): show updatedAt as 'updated UTC' rather than createdAt as
'created UTC'.
@jariy17
jariy17 force-pushed the feat/eval-batch-evaluation-tui branch from 30585a8 to b8e7f9b Compare August 10, 2026 18:23
aidandaly24
aidandaly24 previously approved these changes Aug 10, 2026

@aidandaly24 aidandaly24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment though it can be a follow up

const opts = coreOptsFromCtx(ctx);
return useQuery({
queryKey: ["batch-evaluation", opts.region, id],
queryFn: async () => (await core.eval.getBatchEvaluation(id!, opts)).detail,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may be missing an intended TUI-specific behavior, but should we preserve resultsError here instead of dropping it? A CloudWatch access or read failure currently renders normal job JSON without results, so the user cannot distinguish incomplete data from a job that simply has no results yet. The headless command warns for this reason. Would it make sense to return both values from the query and show a persistent warning above the JSON while still preserving the metadata?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll update now to fix it.

The batch-evaluation get screen dropped resultsError, so a CloudWatch
access/read failure rendered identical JSON to a job with no results yet.
Keep resultsError from the query and render a persistent warning above the
JSON (the TUI equivalent of the CLI's stderr warning).

Adds an optional warning prop to JsonDetail.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants