fix: refresh LLMAsJudgeEvaluator allowed evaluator models#5987
Open
jam-jee wants to merge 2 commits into
Open
Conversation
The hardcoded _ALLOWED_EVALUATOR_MODELS list in sagemaker/train/constants.py had gone stale: several entries were EOL on Bedrock and current judge models were missing, so LLMAsJudgeEvaluator rejected every currently usable model at client-side validation. Re-sync the allowlist (model IDs and per-region availability) with the Bedrock Console judge-model selector, cross-checked against the Bedrock user guide supported-evaluator-models list. Adds Claude Sonnet 4 / 4.5, Claude Haiku 4.5, Claude Opus 4.5, and Nova 2 Lite / Micro / Premier, and widens the region lists for existing models to match the source of truth. Update the docstring examples and unit tests that pinned the previous default judge model (Claude 3.5 Sonnet v1), which is now only available in ap-northeast-1.
nargokul
previously approved these changes
Jul 6, 2026
rsareddy0329
approved these changes
Jul 6, 2026
Remove anthropic.claude-3-5-haiku-20241022-v1:0 from the LLM-as-Judge evaluator allowlist and update the unit and integration tests that referenced it as the judge model.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Addresses internal ticket P467477489.
LLMAsJudgeEvaluatorvalidates the user-suppliedevaluator_modelagainst a hardcoded dict,_ALLOWED_EVALUATOR_MODELS, insagemaker/train/constants.py. That list had gone stale — several entries are now EOL on Bedrock, and every currently usable judge model (Claude Sonnet 4 / 4.5, Haiku 4.5, Opus 4.5, Nova 2 Lite / Micro / Premier) was missing. As a result the client-side check rejected valid models, and models that passed the check failed downstream at Bedrock with end-of-life errors. LLM-as-Judge evaluation was effectively unusable without monkey-patching the SDK.Description of changes
constants.py— re-sync_ALLOWED_EVALUATOR_MODELS(model IDs + per-region availability) with the Bedrock Console judge-model selector, cross-checked against the Bedrock user guide supported-evaluator-models list:us-west-2→us-west-2+us-east-1; Nova Prous-east-1→ all four regions).anthropic.claude-3-5-haiku-20241022-v1:0from the allowlist.ap-northeast-1to match the source of truth.llm_as_judge_evaluator.py— updated docstring examples that pinned Claude 3.5 Sonnet v1 (nowap-northeast-1-only) alongsideus-west-2ARNs.test_llm_as_judge_evaluator.py— updated the default judge model and thevalid_modelslist to reflect the new allowlist.tests/integ/train/— updated the two LLM-as-Judge integ configs that used Claude 3.5 Haiku as the judge model to Claude Sonnet 4.Testing
sagemaker-trainevaluate unit test suite passes locally: 557 passed, 7 skipped.Merge Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.