feat: add vision toggle for agents#11504
Closed
JumpLink wants to merge 5 commits intodanny-avila:mainfrom
Closed
Conversation
Add `vision` boolean field to modelSpecs configuration to explicitly declare model vision support. This enables proper filtering of image artifacts for non-vision models and UI gating for image upload options. - Add vision field to TModelSpec type/schema - Extend validateVisionModel() to check modelSpecs first - Pass modelSpecs from API to agents package - Update UI components to use vision capability check
- Removed direct calls to validateVisionModel in AttachFileMenu and DragDropModal components. - Introduced useVisionModel hook to encapsulate vision model validation logic. - Updated imports to reflect the new hook usage, improving code modularity and readability.
- Remove modelSpecs parameter from createRun() function - Remove modelSpecs conversion logic (handled by agent-level vision toggle) - Remove modelSpecs from createRun() call in client.js - This keeps PR 11501 focused on modelSpecs vision for UI gating only
- Add vision to AgentCapabilities enum and default capabilities - Add vision?: boolean field to Agent type and validation schema - Add vision toggle UI component for agents with hover card and info description - Include vision in agent create/update payload - Pass vision from agent to AgentInputs in run API Depends on PR danny-avila#11501 (modelSpecs vision) for validateVisionModel function
JumpLink
force-pushed
the
feat/agent-vision-toggle
branch
from
1a0d2ff to
3ec5bea
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
visionboolean toggle to agent configuration, allowing users to control vision capability per agent. This works together with the agents package PR to filter image artifacts when vision is disabled.Related to: #11418 (partially addresses)
Related PRs:
Changes
visiontoAgentCapabilitiesenum and default capabilitiesvision?: booleanfield toAgenttype and validation schemavisionin agent create/update payloadvisionfrom agent toAgentInputsin run APImodelSpecsparameter fromcreateRun()API (agents use direct vision toggle)Benefits
end_after_toolsandhide_sequential_outputsDependencies
validateVisionModel()functionTesting
visionis passed to agents package correctly