Feat/hf mammoth lumi helper adds tools/helper to feat/hf_mammoth. - #146

Open

amikael wants to merge 60 commits into

mainfrom

feat/hf_mammoth_lumi_helper

Commits on Aug 22, 2025

Add HuggingFace integration support and code refactoring

chaowang0524
and
claude
committed
1429e40

Commits on Aug 23, 2025

Commits on Aug 26, 2025

1. Added training configuration template.
chaowang0524
committed
c9a6f72

Commits on Aug 27, 2025

Updated readme and changelog file
chaowang0524
committed
d141848
Updated the readme file
chaowang0524
committed
5569f4d

Commits on Sep 5, 2025

Commits on Sep 15, 2025

Commits on Sep 16, 2025

Updated lumi quickstart
chaowang0524
committed
c13aca8

Commits on Sep 26, 2025

Add validation and tokenization improvements

chaowang0524
and
claude
committed
178b8d0
Updated the changelog.md
chaowang0524
committed
3473fc5

Commits on Sep 30, 2025

Add multi-node distributed training support for Puhti and LUMI supercomputers

chaowang0524
and
claude
committed
4d9bdcc
Reorganize CSC supercomputer environment files under csc_env/ directory

chaowang0524
and
claude
committed
49e3d49
Add comprehensive CSC environment documentation and reorganize training scripts

chaowang0524
and
claude
committed
42941d7
Updated the readme
chaowang0524
committed
6ebe7a0
Updated the readme
chaowang0524
committed
430ae2d

Commits on Oct 6, 2025

Add HuggingFace tokenizers library integration for training from scratch

chaowang0524
and
claude
committed
ddebefb
Updated the Readme
chaowang0524
committed
9b69a82
Update changelog.md
chaowang0524
committed
e677fa5

Commits on Oct 20, 2025

Reorganize HuggingFace integration and CSC environment configurations

chaowang0524
and
claude
committed
91531c3
code clean up
chaowang0524
committed
bf88557

Commits on Oct 29, 2025

Added new features to accomodate ModernBert
chaowang0524
committed
69f8d1f
updated 'bert_padding.py'
chaowang0524
committed
1f953d3

Commits on Nov 10, 2025

Code clean up
chaowang0524
committed
dd755f3

Commits on Nov 11, 2025

bug fixes
chaowang0524
committed
e488548
code clean up, ready to implement ModernBert features
chaowang0524
committed
2c309b2

Commits on Nov 17, 2025

Commits on Nov 19, 2025

Commits on Nov 25, 2025

Update the changelog.md
chaowang0524
committed
5367141
- Added features to accomodate HF BART model:
chaowang0524
committed
e8601c4
Updated the change log
chaowang0524
committed
33d175f

Commits on Dec 5, 2025

- Fix the dataset continuation in continue training: now mammoth correctly starts from the last stopping point. The training log will show the first 5 lines in the source training text for debuggin…
chaowang0524
committed
4a5a943

Commits on Dec 8, 2025

Commits on Dec 9, 2025

Commits on Dec 10, 2025

Commits on Dec 17, 2025

Add BART decoder support and full sliding window attention implementation

chaowang0524
and
claude
committed
ab2ca8d
Updated landing page and `Loading_HF_models.md`
chaowang0524
committed
f5ac46e
Added tools/helper tools/helper/venv/.gitignore and HELPER.md

Anssi Yli-Jyra
authored and
Anssi Yli-Jyra
committed
c614526
Updated sbatch-{dev-g,small-g,standard-g,dry-run}, bin/slurm, and updated examples (added 4-node one)

Anssi Yli-Jyra
authored and
Anssi Yli-Jyra
committed
0dc0f26

Commits on Dec 18, 2025

Updated/added tools/helper/examples

Anssi Yli-Jyra
authored and
Anssi Yli-Jyra
committed
9b2db14
Updated task-wrapper.sh with 1>&2

Anssi Yli-Jyra
authored and
Anssi Yli-Jyra
committed
4e1a006

Commits on Jan 27, 2026

Updated build-venv, module-loads and pytorch-rocm-mammoth

Anssi Yli-Jyra
authored and
Anssi Yli-Jyra
committed
480e14f

Commits on Feb 25, 2026

lifted the helper dir

Anssi Yli-Jyra
authored and
Anssi Yli-Jyra
committed
b54cda1