feat(examples): comprehensive node prerequisites audit (fixes #19)

[lexfrei]

Summary

Comprehensive node prerequisites audit for the three example prepare playbooks. Closes issue #19 (LINSTOR ZFS failure) and fills many adjacent gaps uncovered during research: kernel modules for containerd/Kube-OVN, sysctl for bridge netfilter, multipath DRBD blacklist, kernel headers pinned to the running kernel, ZFS backend + KubeVirt modules behind opt-out toggles, and a broadly rewritten README that documents every Cozystack node requirement per subsystem.

The collection role itself is unchanged; all new work lives in examples/*/prepare-*.yml and the README.

Changes

• examples/ubuntu/prepare-ubuntu.yml, examples/rhel/prepare-rhel.yml, examples/suse/prepare-suse.yml — full prerequisite coverage:
  • Base packages: lvm2, thin-provisioning-tools / device-mapper-persistent-data, and kernel headers pinned to the running kernel (Ubuntu/RHEL); openSUSE uses the unversioned kernel-default-devel (zypper rejects the version-suffixed form).
  • Ubuntu also installs linux-modules-extra-{{ ansible_kernel }} which ships openvswitch and geneve on cloud/minimal kernels.
  • RHEL family installs iptables-nft — Rocky 10 / Alma 10 do not ship the iptables binary by default, blocking the optional flush task and k3s kube-proxy replacement.
  • Kernel modules loaded via /etc/modules-load.d/cozystack.conf: overlay, br_netfilter, openvswitch, geneve, ip_tables, iptable_nat.
  • Additional sysctl: net.bridge.bridge-nf-call-iptables, net.bridge.bridge-nf-call-ip6tables, net.ipv6.conf.all.forwarding.
  • Multipath DRBD blacklist drop-in — without it multipathd grabs DRBD devices and LINSTOR volumes become inaccessible after reboot.
• New opt-out toggles (both default true):
  • cozystack_enable_zfs — Ubuntu installs zfsutils-linux; RHEL imports the OpenZFS GPG key then installs the release RPM from a per-major map (cozystack_zfs_release_rpm_by_major_default + additive cozystack_zfs_release_rpm_extra for user overrides without wiping the base dict); openSUSE adds the OBS filesystems repo with a distro-detected path. Debian is not automated (contrib + zfs-dkms required manually). UEK kernels skip with a visible notice (no OpenZFS kmod builds for UEK).
  • cozystack_enable_kubevirt — loads vhost_net, tun, and a best-effort kvm_intel/kvm_amd. The CPU-vendor module is only persisted to modules-load.d if it actually loaded (checked via /sys/module), so systemd-modules-load does not fail on every boot. No host userspace packages are installed: QEMU and libvirt are bundled in KubeVirt pods.
• cozystack_flush_iptables (default false) documented — set to true on cloud images where the default INPUT chain blocks k3s inter-node traffic.
• README.md — new Node Prerequisites section with per-subsystem tables (verified package names for Ubuntu 22.04/24.04, RHEL 9, openSUSE Leap 15.6), Known limitations matrix for ZFS per OS + kernel flavor, and a supported-targets/validated-end-to-end summary table.
• CHANGELOG.rst — expanded Unreleased entry.

Validated end-to-end on OCI (3-node multi-master k3s, 87/87 Cozystack HelmReleases Ready):

• Ubuntu 22.04
• Ubuntu 24.04
• Debian 12
• Rocky Linux 10

Test plan

• ansible-lint passes (production profile)
• ansible-playbook --syntax-check passes on all three prepare playbooks
• Tested on a live cluster — OCI 3-node multi-master, 4 distros above
• Idempotency verified on re-run

Notes for reviewers

• The collection role itself is unchanged; galaxy.yml version is not bumped (per project convention it tracks the upstream Cozystack chart version). Current changes stay under the Unreleased CHANGELOG section until the next upstream chart bump.
• examples/rhel/prepare-rhel.yml retains best-effort Oracle Linux UEK handling (kernel-uek-devel, ZFS skip) even though Oracle Linux is not on the validated-end-to-end list — documented in README.

Summary by CodeRabbit

• New Features

  • Optional ZFS and KubeVirt support with automatic kernel-module probing and persistent loading
  • Expanded node prep to install broader package set and enable bridge/IPv6 sysctls; multipath DRBD blacklist prevents post-reboot access issues

• Documentation

  • Expanded node prerequisites, supported-targets validation matrix, and example playbook variables table
  • Added project working/release conventions

• Chores

  • Added development docs and ignored local tooling directory in VCS

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR expands example preparation playbooks and docs to add distro-aware package installs, kernel module management (optional/required with persistent boot loading), multipath DRBD blacklisting, conditional ZFS and KubeVirt flows, and extended sysctl settings across RHEL, SUSE, and Ubuntu.

Changes

Cohort / File(s)	Summary
Configuration & Documentation \.gitignore, CLAUDE.md, CHANGELOG.rst, README.md	Added .claude/ to ignore; introduced CLAUDE.md with release/review/planning conventions; updated CHANGELOG.rst Unreleased notes for expanded node prerequisites, new opt-out variables (cozystack_enable_zfs, cozystack_enable_kubevirt) and validation details; replaced “Tested on” with a “Supported targets” matrix and expanded node prerequisites in README.md.
RHEL Playbook examples/rhel/prepare-rhel.yml	Expanded cozystack_packages (iptables-nft, lvm2, device-mapper-persistent-data, kernel-devel/kernel-uek-devel, modules-extra); added optional/required kernel module probing, persistent /etc/modules-load.d/cozystack.conf creation, multipathd restart handler and DRBD blacklist drop-in, gated ZFS (GPG import, release RPM install, modprobe/persist) and gated KubeVirt (CPU vendor detection, module probe/persist).
SUSE Playbook examples/suse/prepare-suse.yml	Added lvm2, thin-provisioning-tools, kernel-default-devel; expanded sysctl set; added optional/required module probing with persistence, multipathd restart handler and DRBD blacklist drop-in; OBS-based ZFS repo/GPG key flow and gated KubeVirt handling with CPU vendor detection and persisted modules.
Ubuntu Playbook examples/ubuntu/prepare-ubuntu.yml	Added linux-modules-extra-{{ ansible_kernel }}, linux-headers-{{ ansible_kernel }}, LVM/thin tools; implemented optional/required module probing and persisted /etc/modules-load.d/cozystack.conf; added DRBD blacklist and multipathd restart handler; gated ZFS (zfsutils-linux) and gated KubeVirt with CPU vendor detection and module verification.

Sequence Diagram(s)

mermaid
sequenceDiagram
participant Admin
participant Playbook
participant PackageRepo
participant Kernel
participant Systemd
Admin->>Playbook: run prepare--playbook
Playbook->>PackageRepo: install distro-specific packages (ZFS, headers, extras)
Playbook->>Kernel: modprobe optional modules (best-effort)
Playbook->>Kernel: modprobe required modules (fail on error)
Kernel-->>Playbook: report available modules
Playbook->>Playbook: write /etc/modules-load.d/.conf for present modules
Playbook->>Systemd: drop in DRBD blacklist to /etc/multipath/conf.d/
Playbook->>Systemd: notify Restart multipathd handler
Systemd-->>Admin: multipathd start result (warn on failure)

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~70 minutes

Poem

🐰 Hopping through kernels and modules so bright,
ZFS and KubeVirt now working just right!
With multipath blacklists and sysctl so fine,
Three distros prepared—what a wonderful line!
CozyStack nodes snug, boot modules in tow.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: a comprehensive node prerequisites audit for example playbooks, fixing issue #19. It is specific, concise, and accurately reflects the primary scope of the changeset.
Description check	✅ Passed	The PR description is comprehensive and well-structured, covering summary, detailed changes across all modified files, validated end-to-end testing, and a clear test plan matching the template structure with checkboxes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/node-prerequisites

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request significantly enhances the node preparation playbooks for RHEL, SUSE, and Ubuntu/Debian to ensure all Cozystack prerequisites are met. Key changes include the installation of necessary storage and LVM packages, pinning kernel headers to the running kernel, applying a critical multipath blacklist for DRBD, and managing kernel modules for Kube-OVN and KubeVirt. Additionally, a new CLAUDE.md file establishes project conventions, and the documentation is updated to reflect these comprehensive requirements. Feedback focuses on improving security by pinning GPG key URLs to specific commits and ensuring that optional kernel modules are only persisted if they are actually available to prevent systemd service failures.

[gemini-code-assist]

The GPG key URL points to the master branch of the repository. For better security and stability, it is recommended to use a specific commit hash (pinning) to ensure the integrity of the key and prevent unexpected failures or security risks if the file is modified or moved upstream.

[lexfrei]

This trade-off is already documented at lines 65–69 just above the variable. Pinning to a specific commit would protect against upstream moving the file, but it would also go stale the moment OpenZFS rotates the signing key — at which point rpm_key would silently import a superseded key and later install steps would fail GPG verification on new packages. Tracking master means the playbook follows whatever the OpenZFS project currently publishes, which is what matters for the rpm_key import step.

The fetch itself is over HTTPS against raw.githubusercontent.com, and the key is validated by rpm_key before use. I'd rather keep the current URL (with the inline note telling users what to do if it 404s) than add a moving target that needs manual bumping.

[gemini-code-assist]

The cozystack_kernel_modules_optional list (which includes overlay and openvswitch) is persisted to /etc/modules-load.d/cozystack.conf without verifying if the modules are actually available or built-in. If a module is missing from the kernel, systemd-modules-load.service will fail on every boot. Given that you've implemented careful checks for ZFS and KVM modules to avoid this exact issue (e.g., lines 298 and 375), you should consider applying similar logic here—only persisting modules that are confirmed to be loadable or present in /sys/module.

[lexfrei]

Good catch — applied the same stat/warn/persist pattern used for ZFS and the KVM modules. The optional modules are now loaded with failed_when: false, then stat'd under /sys/module/<name>; only modules that are actually present (loadable or built-in) end up in /etc/modules-load.d/cozystack.conf. Missing modules emit a visible warning pointing at kernel-modules-extra so the user can fix the environment, and systemd-modules-load.service no longer fails every boot on an absent module.

[gemini-code-assist]

The cozystack_kernel_modules_optional list is persisted to /etc/modules-load.d/cozystack.conf without verifying if the modules are actually available. If a module is missing from the kernel, systemd-modules-load.service will fail on every boot. Consider only persisting modules that are confirmed to be loadable, consistent with the logic used for ZFS and KVM modules later in the playbook.

[lexfrei]

Fixed with the same pattern used elsewhere in the playbook: load optional modules with failed_when: false, stat each under /sys/module/<name>, then render /etc/modules-load.d/cozystack.conf with only the modules that are actually present. Missing ones emit a warning pointing at kernel-default-extra. This keeps systemd-modules-load.service from failing every boot on an absent module.

[gemini-code-assist]

The cozystack_kernel_modules_optional list is persisted to /etc/modules-load.d/cozystack.conf without verifying if the modules are actually available. If a module is missing from the kernel, systemd-modules-load.service will fail on every boot. Consider only persisting modules that are confirmed to be loadable, consistent with the logic used for ZFS and KVM modules later in the playbook.

[lexfrei]

Fixed with the same pattern used for ZFS and KVM: optional modules are probed with failed_when: false, stat'd under /sys/module/<name>, and only the ones that are actually present end up in /etc/modules-load.d/cozystack.conf. Missing modules emit a warning pointing at linux-modules-extra so the user can fix the environment, and systemd-modules-load.service no longer fails every boot on an absent module.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@examples/rhel/prepare-rhel.yml`:
- Around line 57-63: The default mapping lacks an EL10 entry causing
cozystack_zfs_release_rpm to be empty and ZFS tasks to fail; add a safe default
and guard the ZFS install flow: add a "10": "" entry to
cozystack_zfs_release_rpm_by_major_default (so cozystack_zfs_release_rpm
resolves to empty string rather than undefined) and update the ZFS-related tasks
that reference cozystack_zfs_release_rpm to run only when
cozystack_zfs_release_rpm | length > 0, and emit a debug/warn message when it's
empty so unsupported majors are skipped non‑fatally (references: variables
cozystack_zfs_release_rpm_by_major_default, cozystack_zfs_release_rpm_by_major,
and cozystack_zfs_release_rpm and the ZFS install tasks that currently consume
cozystack_zfs_release_rpm).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e30137e0-7d8e-4302-9711-2f1262080f8b

📥 Commits

Reviewing files that changed from the base of the PR and between f85d016 and 664d7e6.

📒 Files selected for processing (3)

• examples/rhel/prepare-rhel.yml
• examples/suse/prepare-suse.yml
• examples/ubuntu/prepare-ubuntu.yml

🚧 Files skipped from review as they are similar to previous changes (1)

• examples/ubuntu/prepare-ubuntu.yml

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Default EL10 path hard-fails ZFS flow.

With defaults, cozystack_enable_zfs is true. On EL10, Line 63 resolves cozystack_zfs_release_rpm to empty (no "10" mapping), and Line 277 then fails the play. That makes default prepare behavior fail on EL10-class hosts unless users pre-override variables.

Please make unsupported-major behavior non-fatal by default (warn + skip ZFS tasks), or provide an EL10 mapping when available.

Also applies to: 277-287

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@examples/rhel/prepare-rhel.yml` around lines 57 - 63, The default mapping
lacks an EL10 entry causing cozystack_zfs_release_rpm to be empty and ZFS tasks
to fail; add a safe default and guard the ZFS install flow: add a "10": "" entry
to cozystack_zfs_release_rpm_by_major_default (so cozystack_zfs_release_rpm
resolves to empty string rather than undefined) and update the ZFS-related tasks
that reference cozystack_zfs_release_rpm to run only when
cozystack_zfs_release_rpm | length > 0, and emit a debug/warn message when it's
empty so unsupported majors are skipped non‑fatally (references: variables
cozystack_zfs_release_rpm_by_major_default, cozystack_zfs_release_rpm_by_major,
and cozystack_zfs_release_rpm and the ZFS install tasks that currently consume
cozystack_zfs_release_rpm).

[lexfrei]

This is intentional and documented. See the "Known limitations" table in README.md (RHEL 10 row) and the v1.2.2 entry in CHANGELOG.rst: on EL10 with the default cozystack_enable_zfs: true, prepare fails fast on the missing OpenZFS release RPM.

Silent warn-and-skip was rejected because ZFS is a primary LINSTOR backend in Cozystack. If prepare quietly skipped the ZFS install on EL10 while the toggle stayed at its default, LINSTOR would later refuse to create ZFS-backed volumes and the user would have no breadcrumb pointing back at prepare. Fail-fast forces an explicit cozystack_enable_zfs: false (or cozystack_zfs_release_rpm_extra: {"10": "<url>"} once OpenZFS ships an el10 release RPM).

[coderabbitai]

@lexfrei, understood — the fail-fast is a deliberate UX contract, not an oversight. Failing loudly with a clear message is preferable to silently skipping ZFS and leaving LINSTOR in a broken state without any diagnostic trail. The escape hatches (cozystack_enable_zfs: false or cozystack_zfs_release_rpm_extra) give users a clean path forward. I'll retract the concern.

( \_
(='.'=)
(")_(") ← noted and filed away

---

✏️ Learnings added

Learnt from: lexfrei
Repo: cozystack/ansible-cozystack PR: 27
File: examples/rhel/prepare-rhel.yml:57-63
Timestamp: 2026-04-13T20:30:27.306Z
Learning: In `examples/rhel/prepare-rhel.yml`, the ZFS setup intentionally fails fast on RHEL 10 (EL10) when no OpenZFS release RPM URL is configured for that major version (cozystack_zfs_release_rpm resolves to empty string). This is by design: ZFS is a primary LINSTOR backend in Cozystack, so silently skipping it would leave LINSTOR unable to create ZFS-backed volumes with no diagnostic breadcrumb. Users must explicitly set `cozystack_enable_zfs: false` or supply `cozystack_zfs_release_rpm_extra: {"10": "<url>"}` once OpenZFS publishes an el10 release RPM. This design decision is documented in README.md "Known limitations" and CHANGELOG.rst.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.