slimAttn: conditioning, not invertibility, and the V-cache direction - #28
Open
Kayvan-Zahiri wants to merge 1 commit into
Open
slimAttn: conditioning, not invertibility, and the V-cache direction#28Kayvan-Zahiri wants to merge 1 commit into
Kayvan-Zahiri wants to merge 1 commit into
Conversation
The text says reconstructing V from K needs W_K invertible, and that a determinant of exactly 0 is unlikely. Both hold, but invertibility is not what separates a usable inverse from an unusable one. In large-v3's decoder cross-attention cond(W_K) has a median of 1.4e7 and reaches 5.2e9, with a determinant nowhere near zero, and reconstructing V from K in every layer loses the model at fp32: relative logit error 0.54. Adds the mirror direction, caching V and reconstructing K through W_VK = inv(W_V) W_K, which halves the cache the same way. cond(W_V) peaks at 4.5e6 there, three orders lower, and caching V takes the error from 0.54 to 0.0059. Notes the V-projection bias explicitly. On Whisper, K = V W_VK without subtracting b_V is wrong by more than 100%, so the equation carries the subtraction and points at the appendix's bias removal. Which direction wins is per layer, and the mix moves with model size: V-cache is better in 3 of 4 layers in tiny, 9 of 12 in small, 21 of 24 in medium, 31 of 32 in large-v3. That is the argument for deciding per layer rather than per model. States the limit of the criterion: cond * eps is 619 for the K-cache and 0.53 for the V-cache at fp32 while the per-layer choice measures 5.9e-3, so conditioning orders the choice but overstates the error. Called K-cache and V-cache rather than numbered: Option 1 and Option 2 already mean multiplication order in eq 7 and, separately, K-cache vs X-cache in table 5.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The paper says V-from-K needs
W_Kinvertible, and that a zero determinant isunlikely. True, but that isn't what decides whether the inverse is usable.
large-v3's cross-attention has
cond(W_K)median 1.4e7, max 5.2e9. Thedeterminant is nowhere near zero and reconstructing V from K in every layer
still loses the model at fp32, relative logit error 0.54.
So this adds the other direction: cache V, get K back through
W_VK = inv(W_V) W_K. Therecond(W_V)tops out at 4.5e6 and the error drops to0.0059. Which direction is better varies by layer, and the split moves with model
size: 3 of 4 in tiny, 9 of 12 in small, 21 of 24 in medium, 31 of 32 in large-v3.
The V bias has to come off first. On Whisper, skipping it is wrong by over 100%,
so the equation carries the subtraction.
I avoided numbering these. Option 1 and 2 already mean multiplication order in
eq 7, and K-cache vs X-cache in table 5. Went with K-cache and V-cache. Renumber
if you want.
Numbers from
slimAttn_whisper.py. Builds clean with tectonic, 18 pages, no newwarnings or overfull boxes.