chore(weave): add structured fields when ref objects are not found#7200
Open
domphan-wandb wants to merge 2 commits into
Open
chore(weave): add structured fields when ref objects are not found#7200domphan-wandb wants to merge 2 commits into
domphan-wandb wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
domphan-wandb
force-pushed
the
dom/improve-error-handling-ref-object
branch
from
6e112fc to
b4e5e74
Compare
|
Preview this PR with FeatureBee: https://beta.wandb.ai/?betaVersion=55868991ab6c679826058d68a2589184ea338247 |
Comment on lines
+5807
to
+5808
| f"Ref read contains {len(ref_digests)} digests, but found {len(found_digests)} objects. Diff digests: {ref_digests - found_digests}", | ||
| missing_digests, |
Collaborator
There was a problem hiding this comment.
why duplicate ref_digests - found_digests?
andrewtruong
approved these changes
Jun 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR surfaces the missing refs in a structured field,
missing_digests.This PR allows for better error handling on the application when a batched read is made.See accompanying FE PR for how the error gets surfaced on an eval table: https://github.com/wandb/core/pull/45716
When a run dies before the Weave SDK flushes all of its objects, refs_read_batch references objects the server doesn't have and fails the whole batch with a generic 404:
{ "reason": "Ref read contains 4 digests, but found 3 objects. Diff digests: {'aq4j…'}" }The model evals client, reads every compared run's eval + model objects in one batch) can't tell which object is missing without scraping that human-readable string. This adds the missing digests as a structured field so clients can attribute the failure programmatically (we want to map a digest back to the run that referenced it) instead of parsing the message.
What changed
RefObjectsNotFoundError(NotFoundError)carrying missing_object_digests: list[str].Response shape
Status code (404) and reason are unchanged; the only addition is the new field:
{ "reason": "Ref read contains 4 digests, but found 3 objects. Diff digests: {'aq4j…'}", "missing_object_digests": ["aq4j…"] }Compatibility
Additive:
Testing
How was this PR tested?