MiRA: Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding.

Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding (ECCV 2026)
Seongro Yoon¹   , Donghyeon Cho²   , Jinsun Park³   , François Brémond^1
1 Inria, France    ² Hanyang University, South Korea    ³ Pusan National University, South Korea

Contact: seong-ro.yoon@inria.fr

We are actively conducting interdisciplinary research at the intersection of affective computing and related fields, and are always open to discussions and collaborations. Feel free to reach out!

MiRA (Marginal-induced Attention Redistribution) is a parameter-free, lightweight plug-in framework for foundational video transformers that introduces frame-marginal attention redistribution for facial expression understanding. It encourages the model to focus on subtle intra-face spatio-temporal dynamics, enabling more effective representation learning during large-scale self-supervised pre-training as well as downstream fine-tuning. MiRA provides both Exact mode for principled post-softmax attention redistribution and FlashLite mode, which seamlessly integrates with FlashAttention kernels to provide an efficient approximation of the exact formulation while preserving high training and inference efficiency.

Our implementation follows the official VideoMAE framework, augmenting each transformer attention block with the proposed AttentionMiRA module. The core implementation of AttentionMiRA(nn.Module) is provided in modeling_finetune.py. The same module is reused during self-supervised pre-training in modeling_pretrain.py.

Datasets

Pre-training: million-scale unlabeled facial videos

• VoxCeleb2 (1.2M videos)

Fine-tuning: downstream facial expression recognition

• DFEW (12K video clips)
• MAFW (10K video clips)
• FERV39k (39K video clips)

k-NN Probing: micro-expression recognition

• SAMM
• MMEW

The original datasets should be downloaded from their respective official websites due to their licensing policies.

We provide the metadata CSV files (in datasets/dataSpecCSV_combined/) used in our experiments, containing dataset-specific information required for training and evaluation.

Note

The metadata file vox-celeb-dataset_video_specs_updated_only_inliers.csv is not included in this repository due to its large size. It can be downloaded from our Hugging Face repository. Metadata fields that depend on the local dataset (e.g., bool_file) should be updated according to your downloaded VoxCeleb2 data.

For convenience, we also provide a sample directory structure (one example path per dataset) illustrating the expected organization of the downloaded datasets. See the example directory layouts under datasets/.

Environment

Our implementation was developed and tested with the following environment:

• Python 3.8.19
• CUDA 12.1
• cuDNN 9.2
• GCC 11.3
• PyTorch 2.6.0
• torchvision 0.21.0
• timm 0.4.12
• decord 0.6.0
• deepspeed 0.16.6
• NVIDIA H100 GPUs (40 GPUs for pre-training; 1–4 GPUs for fine-tuning)

Install the required Python packages via:

pip install -r requirements.txt

Main Results with Confusion Matrices

Representative confusion matrices after VoxCeleb2 pre-training and subsequent fine-tuning on each target dataset: DFEW (Fold 1), MAFW (Fold 1), and FERV39k, using FlashLite with ViT-B, ViT-L, and ViT-H. Click each image to view it in full size. Additional confusion matrices for other folds are available under the corresponding confusion_matrix/ subdirectories.

Backbone	DFEW (Fold 1)	MAFW (Fold 1)	FERV39k
ViT-B FlashLite
ViT-L FlashLite
ViT-H FlashLite

Pre-training

Pretrained Models

The pretrained checkpoints are available on Hugging Face.

Backbone	Mode	Pretraining Dataset	Download Checkpoint	Training Script
ViT-B/16	Exact	VoxCeleb2	BASE	script
ViT-B/16	FlashLite	VoxCeleb2	BASE-FlashLite	script
ViT-L/16	FlashLite	VoxCeleb2	LARGE-FlashLite	script
ViT-H/16	FlashLite	VoxCeleb2	HUGE-FlashLite	script

Pre-training from Scratch

TBA

Fine-tuning

Finetuned Models

TBA

Fine-tuning from Pretrained Checkpoints

TBA

Citation

If you find this work useful in your research, please consider citing:

@article{yoon2026reweighting,
  title={Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding},
  author={Yoon, Seongro and Cho, Donghyeon and Park, Jinsun and Br{\'e}mond, Fran{\c{c}}ois},
  journal={arXiv preprint arXiv:2606.30611},
  year={2026},
  note={Accepted to ECCV 2026}
}