Backup failed since GitLab version 19.1.0

[tollscoherent]

Hi everyone,
thanks for maintaining this gitlab image, highly appreciated.
We are hosting it internally and since version 19.1.0 of the image encountered that
our nightly backup crons failed and did not generate current backup-files anymore.

After some investigation we found the following reaon for this on our own instance:

+0200 -- Deleting backup and restore PID file at [/home/git/gitlab/tmp/backup_restore.pid] ... done
rake aborted!
Errno::ENOENT: No such file or directory @ rb_check_realpath_internal - /home/git/data/shared/agent_plan_content (Errno::ENOENT)
/home/git/gitlab/lib/backup/targets/files.rb:182:in realpath'
/home/git/gitlab/lib/backup/targets/files.rb:182:in storage_realpath'
/home/git/gitlab/lib/backup/targets/files.rb:68:in dump'
/home/git/gitlab/lib/backup/tasks/task.rb:25:in backup!'
/home/git/gitlab/lib/backup/manager.rb:57:in run_create_task'
/home/git/gitlab/lib/backup/manager.rb:149:in block in run_all_create_tasks'
/home/git/gitlab/lib/backup/manager.rb:149:in each_value'
/home/git/gitlab/lib/backup/manager.rb:149:in run_all_create_tasks'
/home/git/gitlab/lib/backup/manager.rb:32:in create'
/home/git/gitlab/lib/tasks/gitlab/backup.rb:12:in block in create_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rb:83:in lock_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rb:10:in create_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:8:in block (3 levels) in <main>'
/home/git/gitlab/vendor/bundle/ruby/3.3.0/gems/rake-13.4.2/exe/rake:27:in <top (required)>'
/usr/local/bin/bundle:25:in <main>'

so it seems that the "agent_plan_content" internal path was missing, resulting in a failure of the backup-task.

As a workaround we applied the following local Workaround:

docker compose exec gitlab bash -lc 'install -d -o git -g git -m 0700 /home/git/data/shared/agent_plan_content'

and then reran the backup-creation task, which now finished succesfully.
I'm not sure if this is a problem with new images or only applies to our upgraded
version. But maybe it would make sense to ensure that the setup/migration scripts
of this image could ensure that the required path is created/available.

[th-2021]

Upstream issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/604413

[th-2021]

The AI overview suggests another workaround:

sudo gitlab-backup create SKIP=tar,agent_plan_content

[kkimurak]

Just an Idea: how about adding rake task such as gitlab:backup:create_dirs?

Backup tasks can be found in lib/backup/manager.rb and each task have target. For example, the task agent_plan_content have target ::Backup::Targets::Files.new(progress, storage_path, options: options, excludes: ['tmp']) and the exact path is storage_path. If we can get target path from ::Backup::Targets::Files then we can iterate whole targets defined in lib/backup/manager.rb and iterate to create required directories.

[th-2021]

There is also GITLAB_BACKUP_SKIP that could be set to "tar,agent_plan_content"