feat(examples): comprehensive node prerequisites audit (fixes #19)

.gitignore

@@ -6,3 +6,4 @@ __pycache__/
 *.pyc
 FILES.json
 MANIFEST.json
+.claude/

CHANGELOG.rst

@@ -2,6 +2,62 @@
 cozystack.installer Release Notes
 ============================
 
+Unreleased
+==========
+
+Node prerequisites: comprehensive audit and install in examples.
+
+- Example prepare playbooks now install the full set of node prerequisites.
+  Base additions: ``lvm2``, ``thin-provisioning-tools`` /
+  ``device-mapper-persistent-data``, and kernel headers. Ubuntu and RHEL
+  pin headers to the running kernel (``linux-headers-{{ ansible_kernel }}``
+  / ``kernel-devel-{{ ansible_kernel }}``). openSUSE installs
+  ``kernel-default-devel`` unversioned — SUSE's NVR format differs from
+  ``uname -r`` so zypper rejects the version-suffixed form, but zypper
+  resolves the unversioned name to the version matching the running
+  kernel. On Ubuntu the playbook also installs
+  ``linux-modules-extra-{{ ansible_kernel }}`` which provides
+  ``openvswitch`` and ``geneve`` on cloud/minimal kernels.
+  Debian 12 remains a supported target for ``prepare-ubuntu.yml`` and is
+  validated end-to-end, but ZFS automation is Ubuntu-only: on Debian the
+  playbook skips the ZFS block (with a visible notice), since ``zfsutils``
+  lives in ``contrib`` and the kernel module requires ``zfs-dkms``. Users
+  who want ZFS on Debian must enable contrib + install ``zfs-dkms``
+  manually, or set ``cozystack_enable_zfs: false``.
+- Kernel modules for containerd, Kubernetes bridge networking, and Kube-OVN
+  loaded via ``/etc/modules-load.d/cozystack.conf``: ``overlay``,
+  ``br_netfilter``, ``openvswitch``, ``geneve``, ``ip_tables``, ``iptable_nat``.
+- Additional sysctl parameters: ``net.bridge.bridge-nf-call-iptables``,
+  ``net.bridge.bridge-nf-call-ip6tables``, ``net.ipv6.conf.all.forwarding``.
+- Critical fix: ``multipathd`` DRBD device blacklist at
+  ``/etc/multipath/conf.d/cozystack-drbd-blacklist.conf``. Without it
+  LINSTOR volumes become inaccessible after node reboot.
+- New opt-out variable ``cozystack_enable_zfs`` (default ``true``).
+  Ubuntu installs ``zfsutils-linux`` from the main repo. RHEL imports the
+  OpenZFS GPG key and installs the release RPM before installing ``zfs``.
+  openSUSE adds the OBS ``filesystems`` repo with a distro-detected path
+  segment (Leap / Tumbleweed / SLE). Debian is not automated — contrib +
+  zfs-dkms must be installed manually. Persists the ``zfs`` module via
+  ``/etc/modules-load.d/``.
+- New opt-out variable ``cozystack_enable_kubevirt`` (default ``true``) loads
+  ``vhost_net``, ``tun``, and ``kvm_intel``/``kvm_amd`` kernel modules.
+  QEMU and libvirt are bundled in KubeVirt pods; no host userspace packages
+  are installed.
+- README now documents every node prerequisite per subsystem with exact
+  package names for Ubuntu 22.04/24.04, RHEL 9, and openSUSE Leap 15.6.
+- ``prepare-rhel.yml`` now installs ``iptables-nft``. Rocky 10 / Alma 10 (and
+  other RHEL 10 rebuilds) do not ship the ``iptables`` userspace binary by
+  default, which made the ``cozystack_flush_iptables`` task fail on cloud
+  images. ``iptables-nft`` provides an ``iptables`` wrapper over nftables
+  and is also required for k3s kube-proxy replacement.
+- Validation matrix extended. End-to-end tested on OCI with 3-node
+  multi-master k3s + 87/87 Cozystack HelmReleases Ready: Ubuntu 22.04,
+  Ubuntu 24.04, Debian 12, Rocky Linux 10. For Rocky 10 / Alma 10 (and
+  other RHEL 10 rebuilds) ``cozystack_enable_zfs: false`` is currently
+  required because OpenZFS has not yet published an el10 release RPM;
+  ``prepare-rhel.yml`` fails fast with a clear message until an entry is
+  added to ``cozystack_zfs_release_rpm_by_major``.
+
 v1.1.2
 ======
 

CLAUDE.md

@@ -0,0 +1,114 @@
+# Project conventions: cozystack/ansible-cozystack
+
+This file captures project-specific working conventions that aren't obvious
+from the code.
+
+## Release strategy
+
+- `galaxy.yml` version **tracks the upstream Cozystack chart version** (the
+  value in `roles/cozystack/defaults/main.yml:cozystack_chart_version`).
+- Do **NOT** bump the collection version just because a PR adds features
+  or fixes bugs. The collection version moves only when upstream Cozystack
+  releases a new chart version.
+- Between upstream releases, changes accumulate on `main`. Users pin to
+  git tags for stable installs; they reference `main` if they want the
+  current feature set.
+- Release workflow (`.github/workflows/release.yml`) triggers on
+  `galaxy.yml` changes and creates a tag named `v<version>`. Never edit
+  `galaxy.yml` version to force a release — the upstream Cozystack version
+  is the trigger.
+- Git tags are **`v`-prefixed** (`v1.2.2`), but `galaxy.yml` and
+  `requirements.yml` version fields **omit the `v`** for Ansible Galaxy.
+  Example requirements files that use `type: git` reference tags and must
+  include the `v` prefix.
+
+## CHANGELOG discipline
+
+- `CHANGELOG.rst` uses per-version sections in reverse chronological order.
+- For changes on `main` without a version bump, use an `Unreleased`
+  section at the top. Rename it to the new version when upstream bumps
+  and the release workflow fires.
+- Never attribute new changes to an already-tagged version — releases are
+  immutable.
+
+## Review workflow for pull requests
+
+Every PR must receive at least two independent review passes before
+merging — one from a second LLM (run in parallel, in the background, so
+it does not block the main review) and one from a branch-diff-focused
+reviewer. Iterate on any blocking findings until both pass. False
+positives must be explicitly justified before dismissing.
+
+Only then enable auto-merge:
+
+```bash
+gh pr merge --auto --squash --delete-branch
+```
+
+## Planning discipline
+
+When the user invokes plan mode (or the task is non-trivial):
+
+- Research must be **exhaustive, not summary-level**. For anything touching
+  OS packages, kernel modules, or upstream components:
+  - Verify each package name against the **latest LTS** repos (Ubuntu
+    22.04/24.04, RHEL 9, openSUSE Leap 15.6).
+  - Follow the dependency chain: e.g., LINSTOR needs DRBD, DRBD needs
+    headers, Piraeus compiles DRBD at runtime, multipathd grabs DRBD
+    devices, etc.
+  - Cover every Cozystack subsystem (storage, networking, virtualization,
+    observability) even if the immediate issue touches only one.
+- Produce per-OS tables when cross-distro work is involved.
+- Cite upstream documentation URLs in research output.
+
+Shallow "looks right" research will be rejected in plan review.
+
+## Ansible collection conventions
+
+- Node-side work lives in `examples/*/prepare-*.yml`, not in the role.
+  The role handles only the Cozystack chart and Platform Package install.
+- Kernel modules on nodes:
+  - Drop a file in `/etc/modules-load.d/cozystack*.conf` for persistence.
+  - `community.general.modprobe` for immediate load.
+  - Use `failed_when: false` only when the failure is benign (module
+    built into kernel, only one of two vendor-specific modules applies);
+    document the reason inline.
+- Kernel headers: Piraeus builds DRBD against the running kernel, not
+  the staged one, so pin to the running kernel when the distro allows.
+  - Ubuntu/Debian: `linux-headers-{{ ansible_kernel }}` works.
+  - RHEL: `kernel-devel-{{ ansible_kernel }}` works.
+  - openSUSE/SLE: zypper rejects `kernel-default-devel-{{ ansible_kernel }}`
+    because SUSE package names use a different NVR format than `uname -r`.
+    Use the plain `kernel-default-devel` metapackage — zypper resolves
+    it to the version matching the installed kernel. Note: if a kernel
+    update is staged but not yet booted, this pulls headers for the
+    newer installed kernel, not the running one; the user should reboot
+    before running the playbook in that case.
+- External repos (OpenZFS release RPM, OBS `filesystems`):
+  - Import the GPG key explicitly; **never** `disable_gpg_check: true`.
+  - OBS path segments are canonical distro names (e.g.
+    `openSUSE_Leap_15.6`, `SLE_15_SP6`), not bare version numbers.
+
+## What NOT to install on hosts
+
+KubeVirt, Kube-OVN, and Piraeus all bundle their userspace components in
+their own pods. Do **not** install on the host:
+
+- `qemu-kvm`, `libvirt*` — KubeVirt bundles these.
+- `openvswitch-switch` / `openvswitch` userspace — Kube-OVN bundles OVS.
+- `drbd-utils`, `drbd-dkms`, `kmod-drbd*` — Piraeus init-container
+  compiles DRBD 9.x from source at runtime (kernel headers are enough).
+
+The host only needs the kernel modules and, for KVM, a working `/dev/kvm`.
+
+## Critical silent-failure traps
+
+- **Multipath DRBD blacklist**: without
+  `/etc/multipath/conf.d/cozystack-drbd-blacklist.conf` blocking
+  `^drbd[0-9]+`, `multipathd` grabs DRBD devices and LINSTOR volumes
+  become inaccessible after reboot. Always apply.
+- **`vhost_net` not loaded by default**: KubeVirt VMIs stay Pending until
+  it's present.
+- **`br_netfilter` missing**: `net.bridge.bridge-nf-call-*` sysctls fail
+  with "No such file or directory". Load the module before applying the
+  sysctl.