Feat/hf mammoth lumi helper adds tools/helper to feat/hf_mammoth. - #146
Open
amikael wants to merge 60 commits into
Open
Feat/hf mammoth lumi helper adds tools/helper to feat/hf_mammoth. #146amikael wants to merge 60 commits into
amikael wants to merge 60 commits into
Conversation
- Add x_transformers integration module for HF compatibility - Add hf_converter.py for model conversion utilities - Add translation_config.yaml for translation configuration - Refactor imports and code style across codebase - Update constants and token handling for consistency - Add .ipynb to gitignore for notebook files - Fix file permissions and formatting issues 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2. Added "--valid_metrics" to enable in-training-validation.
- Add LUMI environment setup, training, and translation scripts - Improve HuggingFace hub upload to include config push - Add save_model path validation with proper error handling - Include minimal requirements file for LUMI deployment
- Add validation at start option for early model assessment - Fix task sampling distribution logging when no samples exist - Add subword type configuration for denoising transforms - Support BPE and SentencePiece type specification for word boundaries 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
…omputers
- Add SLURM job scripts for 2-node distributed training on Puhti and LUMI clusters
- Configure 8-GPU training setup with shared encoder and language-specific decoders
- Implement validation improvements with BLEU score reporting and master-only execution
- Add comprehensive distributed training flowchart documentation for Puhti
- Remove deprecated requirements_lumi.txt file
- Fix NCCL timeout issues by restricting validation to master rank only
The configuration supports 8 translation tasks (en→{ar,eu,fi,he,hi,hr,is,sw}) with
shared encoder components synchronized via all-reduce and independent language decoders.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
Move all LUMI and Puhti configuration files into a centralized csc_env/ directory structure to improve organization and maintainability of supercomputer-specific configurations. This consolidates environment setup scripts, training configurations, and documentation for both LUMI and Puhti systems. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…ng scripts Restructure CSC supercomputer configuration files to separate single-node and multi-node training workflows. Create unified documentation hub with environment-specific quickstart guides for both LUMI and Puhti systems. Changes: - Add csc_env/README.md as central documentation hub - Create PUHTI_QUICKSTART.md with single-node and multi-node training guides - Rewrite LUMI_QUICKSTART.md with improved structure and hardware specifications - Rename training scripts for clarity (train.sh → multi_node_train.sh, etc.) - Add single_node_train.yaml configurations for both environments - Add requirements_puhti.txt for Puhti-specific dependencies - Update script references to match new file organization 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit enables MAMMOTH to use HuggingFace's tokenizers library as an alternative to traditional vocabulary files, supporting modern tokenization algorithms (BPE, WordPiece, Unigram) with faster performance and flexible configuration. Core changes: - New HFTokenizerVocab class providing MAMMOTH-compatible interface for HF tokenizers - Automatic .json file detection for seamless tokenizer loading - Modified dataset pipeline to handle both traditional vocabs and HF tokenizers - Updated translation decoding to use tokenizer's built-in decoder - Added --use_hf_tokenizer configuration flag with backward compatibility Documentation: - Comprehensive guide (docs/HF_TOKENIZERS.md) covering installation, usage, and troubleshooting - Example script (examples/hf_tokenizers/train.py) for MARIAN-style BPE tokenizer training - Updated README.md with link to HF tokenizers guide - Updated CHANGELOG.md with detailed feature description 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Establish formal mammoth/hf_integration/ module structure with from_hf/ and to_hf/ subdirectories - Add ModernBERT conversion support with RoPE and unpadding capabilities - Reorganize Puhti/LUMI configs by node count (one_node/, two_nodes/, four_nodes/) - Add four-node 16-GPU training configurations with multi-node SLURM scripts - Update x-transformers to version 2.9.2 with sliding window attention and local RoPE support - Enhance vocabulary system and transforms for improved HF tokenizer compatibility - Move legacy conversion scripts from root to proper module locations 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…3 as single decoder - Now Mammoth supports encoder and decoder are using different setup by prefixing "enc" and "dec" for more x_transformers parameters - Exposed x_transformer parameter 'rotary_pos_emb_base' to Mammoth
…ma3 as decoder - Added parameter 'scaled_embedding' to accomodate Gemma3 training (the original x-transformer library only supports scaled embedding in the class 'AbsolutePositionalEmbedding'. - Update the logic of 'reset_optim', now it has four scenarios that can correctly load optimizer states, momentum and steps from the previous ckpt. - Now the 'filtertoolong' filter supports to filter out the sentences that is longer than certain tokens and shorter than certain tokens.
…nd decoder's model dim are not aligned.
- QKV bias - Layernorm bias - final logits bias - tie embedding (share embedding between encoder and decoder) - Implemented the freezing mechanism (left unimplemented), now the training can freeze: - encoder embedding - encoder (does not include embedding) - cross attention only - decoder (include cross attention, not include embedding) - decoder (exclude cross attention) - decoder embedding Removed the legacy freezing embedding mechanism. Currently, the output projection layer is always trainable. - Reinstated sliding window attention - Reinstated local/global RoPE theta value for sliding window attention layer
…ectly starts from the last stopping point. The training log will show the first 5 lines in the source training text for debugging purpose - Removed the Puhti related configs and docs from this branch.
… Now this message will only appear when Mammoth has stripped certain special tokens from the data.
…oint with best metrics - Fixed the bug that the option 'valid_at_start' didn't work properly - The line checker (to confirm whether the training is starting at the previous stopping point in the dataset) will only log out lines at the beginning of the training
…tion Feature Improvements: - BART decoder sequence support: Decoder sequences now correctly start with </s> (EOS) then <s> (BOS) tokens, matching BART's architecture requirements - Full sliding window attention with dual RoPE: Complete implementation for Gemma3 and ModernBERT models supporting alternating local/global attention patterns with separate RoPE theta values (global: 1M, local: 10K for Gemma3; global: 160K, local: 10K for ModernBERT) - Enhanced embedding sharing: Cross-lingual vocabulary sharing now supports bilingual models with identical vocabularies but different language labels (e.g., es→en BART models) - Comprehensive logging: Added detailed configuration logging for sliding window, dual RoPE parameters, and embedding sharing decisions Bug Fixes: - Fixed RoPE weight mapping: Removed inv_freq copying in conversion scripts as it's auto-computed from theta parameters, preventing conflicts with dual RoPE configurations - Corrected decoder layer indexing: Fixed triplet indexing for hybrid models with cross-attention enabled Implementation Details: - Added `decoder_start_with_eos` parameter throughout the pipeline (train.py, vocab.py, dataset.py) - Extended model_opts with sliding window parameters: enc_/dec_sliding_window, enc_/dec_global_attn_every_n_layers, enc_/dec_global_rope_theta, enc_/dec_local_rope_theta - Updated conversion scripts (gemma2mammoth.py, modernBERT_gemma3_2mammoth.py) with full sliding window configuration extraction and mapping - Improved embedding sharing logic in model_builder.py to detect vocabulary equivalence across different language tags 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
amikael
force-pushed
the
feat/hf_mammoth_lumi_helper
branch
from
December 17, 2025 19:01
ef6d1fe to
c614526
Compare
added 5 commits
December 17, 2025 21:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch aims to store helper functions tools/helper added to MAMMOTH
and facilitate merging of them to feat/hf_mammoth_lumi.