"elemental install" fails during rsync

[Swassie]

elemental-toolkit version:

I have tried 2.1.0 and 2.2.0

CPU architecture, OS, and Version:

Linux UD-MS0 6.6.36.3-syntheticmr+ #16 SMP Wed Aug 28 14:57:01 CEST 2024 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug

rsync exits with error "No space left on device (28)" during snapshot creation, when syncing data to the image mounted to /run/elemental/workingtree. The error started when we added some new files (1GB total size) to our docker image. The host system has lots of free disk space.

Logs

�[36mINFO�[0m[2025-09-09T07:24:54Z] Creating image /run/elemental/state/.snapshots/1/snapshot.img from rootDir /run/elemental/state/.snapshots/1/snapshot.workDir 
�[37mDEBU�[0m[2025-09-09T07:24:55Z] Image size 4282M                             
�[37mDEBU�[0m[2025-09-09T07:24:55Z] Running cmd: 'mkfs.ext2 -L EL_SNAP1 /run/elemental/state/.snapshots/1/snapshot.img' 
�[37mDEBU�[0m[2025-09-09T07:24:55Z] Mounting image EL_SNAP1 to /run/elemental/workingtree 
�[37mDEBU�[0m[2025-09-09T07:24:55Z] Running cmd: 'losetup --show -f /run/elemental/state/.snapshots/1/snapshot.img' 
�[36mINFO�[0m[2025-09-09T07:24:55Z] Sync /run/elemental/state/.snapshots/1/snapshot.workDir to /run/elemental/workingtree 
�[36mINFO�[0m[2025-09-09T07:24:55Z] Starting rsync...                            
�[37mDEBU�[0m[2025-09-09T07:24:55Z] Running cmd: 'rsync --progress --partial --human-readable --archive --xattrs --acls --exclude=/run/elemental/state/.snapshots/1/snapshot.workDir/.snapshots --exclude=/run/elemental/state/.snapshots/1/snapshot.workDir/mnt --exclude=/run/elemental/state/.snapshots/1/snapshot.workDir/proc --exclude=/run/elemental/state/.snapshots/1/snapshot.workDir/sys --exclude=/run/elemental/state/.snapshots/1/snapshot.workDir/dev --exclude=/run/elemental/state/.snapshots/1/snapshot.workDir/tmp --exclude=/run/elemental/state/.snapshots/1/snapshot.workDir/run --exclude=/run/elemental/state/.snapshots/1/snapshot.workDir/host --exclude=/run/elemental/state/.snapshots/1/snapshot.workDir/etc/resolv.conf /run/elemental/state/.snapshots/1/snapshot.workDir/ /run/elemental/workingtree/' 
�[37mDEBU�[0m[2025-09-09T07:25:00Z] Syncing data...                              
�[37mDEBU�[0m[2025-09-09T07:25:05Z] Syncing data...                              
�[37mDEBU�[0m[2025-09-09T07:25:10Z] Syncing data...                              
�[37mDEBU�[0m[2025-09-09T07:25:15Z] Syncing data...                              
�[37mDEBU�[0m[2025-09-09T07:25:20Z] Syncing data...                              
�[37mDEBU�[0m[2025-09-09T07:25:25Z] Syncing data...                              
�[37mDEBU�[0m[2025-09-09T07:25:27Z] 'rsync' command reported an error: exit status 11 
...
rsync: [receiver] write failed on "/run/elemental/workingtree/var/lib/dpkg/info/udev.md5sums": No space left on device (28)

Additional context

``

[frelon]

Hi @Swassie!

Can you post more of your configuration? Did you try extending the state partition size (config pointer)

[Swassie]

Hey @frelon, thanks for the quick response!

Config:

# Reference:
# https://rancher.github.io/elemental-toolkit/docs/customizing/general_configuration/

install:
  partitions:  # All sizes are in MiB
    state:
      size: 131072
      fs: ext4
    recovery:
      size: 32768
      fs: ext4
    oem:
      size: 4096
      fs: ext4
    persistent:
      size: 0  # zero to use all remaining space
      fs: ext4
  grub-entry-name: test

reset:
  grub-entry-name: test

upgrade:
  grub-entry-name: test

mount:
  sysroot: /sysroot
  write-fstab: false
  ephemeral:
    type: tmpfs
    size: 25%
    paths:
      - /etc
      - /tmp
      - /var
  persistent:
    mode: bind
    paths:
      - /root
      - /home
      - /etc
      - /opt
      - /usr/local/bin
      - /var/log
      - /var/opt
      - /var/lib/docker

Yes, I have tried extending the state partition. Some more info, just before the error occurs:

root@8c54da496d93:/# du -s /run/elemental/state/.snapshots/1/snapshot.workDir; df
4271776 /run/elemental/state/.snapshots/1/snapshot.workDir
Filesystem      1K-blocks      Used Available Use% Mounted on
overlay        1055762868 260279968 741779428  26% /
none             12302852         8  12302844   1% /dev/shm
/dev/sdc       1055762868 260279988 741779408  26% /etc/hosts
none             12302852      3068  12299784   1% /run/docker.sock
/dev/loop0p1        65374      8866     56508  14% /run/elemental/efi
/dev/loop0p2      8058252        28   7627328   1% /run/elemental/oem
/dev/loop0p5    135148224        28 128210188   1% /run/elemental/persistent
/dev/loop0p3     32716560        24  31022432   1% /run/elemental/recovery
/dev/loop0p4    131001348   8519464 115754616   7% /run/elemental/state
/dev/loop1        4288192   4227696         0 100% /run/elemental/workingtree

[frelon]

I'm trying to replicate this on HEAD and I'm also getting errors when the image size is larger than around ~4Gb and using the loopdevice snapshotter.

It works using the default partition sizes and a 5Gb image and the btrfs snapshotter, so the problem seems specific to the loopdevice snapshotter.

The /dev/loop1 size in the above logs seem suspect.. we try to calculate the size of the loopback image and add 256Mb overhead here. Maybe we have an error in the size calculation for larger images.

[Swassie]

If we have previously used loopdevice snapshots, is it possible to upgrade to an image with btrfs snapshots? Are the two types compatible?

[frelon]

There is no tested way to migrate from one snapshotter to another.

With that said since we have the recovery system and a elemental reset command that resets the state partition, it might be possible to switch snapshotter in that way, but we have not tested that yet!