Repository navigation
Generalize gradient flow probe to any experiment config - #128
Merged
Merged
Conversation
- formatting: align on key lists with Levenshtein.opcodes so punctuation-only
tokens no longer shift every later pair (corrupted case/punct metrics)
- audio: word-bounded spelling fixes ("overall rights" no longer becomes
"overalrights"); torchcodec audio no longer round-trips through PCM_16;
shared to_mono downmix (fixes channels-first stereo written as n channels)
- swift_sdk: reuse prepare_wav_bytes instead of a parallel WAV writer
- metrics: per-utterance WER via word_errors; scipy.stats.bootstrap for
paired_bootstrap_delta (same CI for the same seed)
- speaker_metrics: rapidfuzz.process.cdist for the cpWER cost matrix
- common: drop unreachable subsequence fallback in entity_in_text
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014D4gdQCgNfW9ZWGdgMuEVg
- generate/generate_streaming: fold kwargs into a config copy via GenerationConfig.update (also stops v5's deprecation warnings) - save_pretrained: set base_model_name_or_path on the PEFT config before saving instead of rewriting adapter_config.json afterwards - asr_config: AutoConfig.for_model - asr_layers: nn.Module.get_submodule for the layer-stack lookup - projectors: torch.nn.RMSNorm instead of transformers' LlamaRMSNorm Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014D4gdQCgNfW9ZWGdgMuEVg
…/deploy - production.yaml: hub_strategy end; PushToHubCallback already pushes every save, so Trainer was uploading the same weights a second time - train: Dataset.repeat for upsampling/epoch expansion; select_columns; token preflight via huggingface_hub.get_token (accepts `hf auth login` and training.hub_token, which is now actually passed to the pushes) - remote_scripts: shlex.quote/shlex.join for tokens, keys and Hydra overrides so `$`, quotes, spaces or brackets survive bash on the pod - plan: count projector params on the meta device - check_gradient_flow: read a real experiment (--experiment, default stage_1; embedded.yaml no longer exists), build the batch with the training DataCollator, get_total_norm, transformers.set_seed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014D4gdQCgNfW9ZWGdgMuEVg
nn.RMSNorm multiplies by the weight before the final bf16 cast; LlamaRMSNorm casts first. In bf16 about a quarter of outputs differed by one ULP, so existing checkpoints would no longer decode bit-identically. Not worth it for dropping one import. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014D4gdQCgNfW9ZWGdgMuEVg
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactors the gradient flow debugging script to work with any experiment configuration instead of being hardcoded to
embedded.yaml. The script now accepts an--experimentflag and uses the same config composition and model building logic as the training pipeline.Key Changes
scripts/debug/check_gradient_flow.py
embedded.yamlreading withload_experiment_config()that composesconfig.yaml+ experiment overrides using Hydra, matching the trainer's approachbuild_asr_config()fromscripts/train.pyinstead of manually constructingASRConfig, ensuring the probe uses identical freeze settings and configuration as trainingDataCollatorfromscripts/train_collator.py, eliminating divergence between probe and actual training data flowencoder_learning_rateandencoder_weight_decayto the training knobs dictionarytorch.nn.utils.get_total_norm()instead of manual L2 norm calculationdisable_chat_template_thinking()before collation to match training setup--experimentparameter (defaults tostage_1) via Typertests/test_check_gradient_flow.py (new)
Supporting Changes
get_token()import for Hub token preflight checksgenerate()to fold generation flags into a deepcopiedgeneration_configinstead of passing alongside it, addressing transformers v5 deprecation warningsto_mono()helper for consistent multichannel audio handling; updatedprepare_wav_bytes()to use real training collator patternshlexquoting for all caller-supplied values (tokens, W&B IDs, Hydra overrides) to prevent shell expansionjiwer.process_words()withword_errors()Levenshtein distance for efficiencyNotable Implementation Details
DataCollatorconfiguration as training, including left-padding of text and proper<audio>token emissionattn_implementation=eagerto avoid flash-attn dependency while maintaining gradient flow independence from attention implementationhttps://claude.ai/code/session_014D4gdQCgNfW9ZWGdgMuEVg