What is the execution order of substeps?

[jjqq2013]

There seems no documentation about the execution order of substeps such as files, commands.

I have done some experiments, turns out that there are some rules existing in the internal implementation, I hope we can add the rules to the documentation.

==> /oem/91_custom.yml <==
name: "====test====1"
stages:
  initramfs:
   - name: "test step1"
     files:
       - path: /etc/file1
         permissions: 644
         owner: 0
         group: 0
         content: "file1"
     commands:
       - ls /etc/file1 2>&1 > /etc/cmd1.out

==> /oem/92_custom.yml <==
name: "====test====2"
stages:
  initramfs:
   - name: "test step1 of test2"
     commands:
       - ls /etc/file2 2>&1 > /etc/cmd2.out
     files:
       - path: /etc/file2
         permissions: 644
         owner: 0
         group: 0
         content: "file2"

==> /etc/cmd1.out <==
/etc/file1

==> /etc/cmd2.out <==
/etc/file2

Regardless of the order of files and commands, the files is always executed before commands.

So I am wondering, what is the rule?

[jjqq2013]

Having checked some source code, it seems the order of substeps is hardcoded in the source code, if that is true, then it is a problem.
https://github.com/rancher/elemental-toolkit/blob/v2.3.0-dev/vendor/github.com/rancher/yip/pkg/executor/default.go#L97

func (e *DefaultExecutor) applyStage(stage schema.Stage, fs vfs.FS, console plugins.Console) error {
	var errs error
	for _, p := range e.conditionals {
		if err := p(e.logger, stage, fs, console); err != nil {
			e.logger.Warnf("(conditional) Skip '%s' stage name: %s",
				err.Error(), stage.Name)
			return nil
		}
	}

... ...
	for _, p := range e.plugins {
		if err := p(e.logger, stage, fs, console); err != nil {
			e.logger.Error(err.Error())
			errs = multierror.Append(errs, err)
		}
	}
	return errs
}

The plugins seems is defined in https://github.com/rancher/elemental-toolkit/blob/v2.3.0-dev/vendor/github.com/rancher/yip/pkg/executor/executor.go#L74-L74

func NewExecutor(opts ...Options) Executor {
	d := &DefaultExecutor{
		logger: logrus.New(),
		conditionals: []Plugin{
			plugins.NodeConditional,
			plugins.IfConditional,
		},
		plugins: []Plugin{
			plugins.DNS,
			plugins.Download,
			plugins.Entities,
			plugins.EnsureDirectories,
			plugins.EnsureFiles,
			plugins.Commands,
			plugins.DeleteEntities,
			plugins.Hostname,
			plugins.Sysctl,
			plugins.User,
			plugins.SSH,
			plugins.Timesyncd,
			plugins.Systemctl,
			plugins.Environment,
			plugins.SystemdFirstboot,
			plugins.DataSources,
			plugins.Layout,
		},
	}
...

[jjqq2013]

It seems that mixing defining multiple substeps in a single step make things unclear, such as

name: "====test====1"
stages:
  initramfs:
   - name: "test step1"
     files: ...
     commands: ...        => is the `environment` substeps executed before me?
     environment: ... 

Fortunately, defining multiple steps instead of multiple substeps make things clear, since the steps is an array, they executes in the definition order, very clear.

name: "====test====1"
stages:
  initramfs:
   - name: "test step1"
     files: ...
   - name: "test step2"
     environment: ... 
   - name: "test step3"
     commands: ...

So it is not a big problem, just need some documentation.

Hope this discussion can help those who are wondering about the execution order of substeps.

[jjqq2013]

Added some documentation in PR #2294