refactor(server): move speculative init to speculative.cpp#24952
Closed
wadealexc wants to merge 1 commit into
Closed
refactor(server): move speculative init to speculative.cpp#24952wadealexc wants to merge 1 commit into
wadealexc wants to merge 1 commit into
Conversation
- unifies draft/spec mtp parameter initialization, model, and context load - changes server_context_impl model_dft and ctx_dft to use raw pointers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR unifies draft/mtp parameter initialization, model, and context loading. I wrote this primarily considering the perspective of
server_context_impl::load_model, after reviewing the main/text model initialization path and comparing against draft/mtp model/context init paths.Draft/MTP model/context are now initialized almost identically to the main model - a constructor that uses a pimpl and exposes model/context via accessor methods.
server_context_implhas been changed so thatmodel_dftandctx_dftare raw pointers, like the main model.I also added a helper for
common_paramsinitialization that captures the behavior spread across a few different places inload_modeland provides a single method that should handle all initialization cases correctly. (seecommon_base_params_to_speculative)Additional information
A few things I want to point out:
params_dft.n_outputs_max = params_base.n_parallel. Every other use ofparams_dftinload_modeldoes (both fit and spec_mtp init), meaning those uses inheritserver_n_outputs_max(params_base). This seemed unintended to me (possibly reserving more VRAM than necessary?), so I fixed it incommon_base_params_to_speculative.load_progress_callbackto capture the prior spec_mtp branch behavior. There might be a cleaner way to accomplish this.common_speculative_initfails, we now reset the model as well as the context (master only resets the context).I'm happy to make any adjustments as needed.
Requirements