Feature/multiserver plugin

[Muddyblack]

Reference: #3420

Summary

This PR adds the multiserver plugin to distribute a single Netlab topology across multiple physical servers.
Sadly for now containerlab-provider only.

Key Details

• Self-contained: The implementation is entirely within netsim/extra/multiserver/ and doesn't modify any core Netlab engine logic.
• Consistent Allocations: IP, interface, and VNI allocations are computed once on the workstation. The plugin then generates a per-server directory with a filtered clab.yml and netlab.snapshot.pickle.
• Native Remote Deployments: Remote servers launch using standard sudo netlab up --snapshot -vv without needing custom CLI options.

For the test-files I am not sure if they make any sense. But they show at least it does not interfere with the normal netlab workflow.

Explanations on how it works can be found in docs/plugins/multiserver.md.

[ipspace]

Super-awesome-job!!! Thanks a million.

Tons of comments (as you expected ;). Some of them are just suggestions or pointers to existing helper functions, in other cases I think we can make the whole thing a lot more streamlined with significant rewrites.

[ipspace]

We might want to automate that in the future, but this is definitely more than good enough for version 1

[Muddyblack]

Yeah currently i mostly do it externally automated. So an extra custom script.

[Muddyblack]

I guess about 99% of them are valid 😁
I will have a look at it next week 👍

[Muddyblack]

Refactor & review-response changelog

clab.yml generation

• Dropped ~200 lines of hand-rolled Python clab generation (_build_server_clab & co).
• output() now filters the topology and renders via the standard clab.j2 template.
• Cross-server P2P links carry a link.clab.vxlan attribute; clab.j2 got one extra
  branch under node_count == 2. Regular topologies are unaffected.

Data types

• Added must_be_group_id and must_be_node_or_group (mirroring must_be_node_id).
• Schema now validates groups (group_id), members (node_id), replicate (node_or_group).
• Removed the manual existence-check loops in _resolve_assignments /
  _resolve_replicated; they only expand references now.

Servers as a dictionary

• multiserver.servers is a name-keyed dict — duplicate names impossible by construction.
• Dropped the seen_ids loop; _validate_servers uses _dataplane IDs (like VLANs/VRFs).

CLI hooks for VXLAN setup/teardown

• Hooks (post_start_clab / pre_stop_clab) are baked into each server's snapshot with a
  relative vxlan-setup.sh path, so they fire locally where netlab up --snapshot runs.

Single-server netlab up --snapshot fixes

• _classify_links marked a link cross-server if endpoints spanned >1 server,
  ignoring clab.uplink — uplink NIC + VXLAN on the same bridge, no STP →
  broadcast storm → 100 % CPU. Fix: clab.uplink links are now forced cross: False.
• set -e + bare ip link add vxlanN aborted on "File exists" after a crash.
  Fix: ip link del vxlanN 2>/dev/null || true before each add in
  vxlan-setup.j2 / generated script.
• New multiserver.vxlan.auto_start bool (default true). When false, scripts
  are generated but CLI hooks are not registered — tunnels stay down until run
  manually (staged convergence or single-server local testing).

Remote-server snapshots (superseded by _resolve_snapshot_paths above)

• make_paths_absolute() no longer touches per-server snapshots.

Bug fixes

• Fixed two Box crashes in _build_server_topo (default_box=True missing → BoxKeyError).
• Box cleanup per AGENTS.md: to_dict() and dotted-path .get().
• Module config render failed in per-server dirs ('frr.j2' not found, OSPF/etc.):
  snapshot load skips make_paths_absolute(), so a bare relative templates dir
  reached render_template(), whose "not .//" branch re-based it onto the netlab
  install dir → in-template {% import %} lookups missed. Fix: _resolve_snapshot_paths
  now anchors relative entries with ./ via _cwd_relative() (stays portable across hosts).

Tests & docs

• Merged srlinux/frr coverage into multiserver-explicit.yml; dropped the separate test.
• docs/plugins/multiserver.md updated for the dict format, CLI hooks, and clab.j2 flow.

Open / outlook

• Waiting on a core post-output callback (offered by maintainer) to replace the atexit
  file-copy handler.
• Future: optional remote orchestration — control node pushes dirs and runs netlab
  remotely (SSH), so the whole lab comes up from one command. Out of scope here.

---

I am not sure on the package: (netlab installation path) is it baked into the snapshot file?

This would require the user to have on all their machines the same path where netlab is installed onto.
Although I would say usually you do similar setups of your machines anyways (just cloning them) .

if "package:" in entry:
  resolved.append(str(moddir / entry.replace("package:", "")))

Also unhappy with _cwd_relative and _resolve_snapshot_paths

[ipspace]

A few quick thoughts -- more details after I implement the hooks this needs and we rebase it.

Added must_be_group_id and must_be_node_or_group (mirroring must_be_node_id).

Will cherry-pick this change into a new PR. Don't want to have this hidden in a large blob of unrelated code.

• Hooks (post_start_clab / pre_stop_clab) are baked into each server's snapshot with a
  relative vxlan-setup.sh path, so they fire locally where netlab up --snapshot runs.

👍

• New multiserver.vxlan.auto_start bool (default true). When false, scripts
  are generated but CLI hooks are not registered — tunnels stay down until run
  manually (staged convergence or single-server local testing).

👍

• make_paths_absolute() no longer touches per-server snapshots.

That's a huge can of worms. We need the absolute paths in the snapshot and in the Ansible inventory. I think it would be best to have a plugin hook executed very early in "netlab up" so it can adjust the topology data and recreate the snapshot and Ansible inventory before "netlab up" does some real work.

I am not sure on the package: (netlab installation path) is it baked into the snapshot file?

It's resolved from the current installation path in make_paths_absolute

This would require the user to have on all their machines the same path where netlab is installed onto. Although I would say usually you do similar setups of your machines anyways (just cloning them) .

Unless you install netlab as root on lab VMs and in virtual environment on your local machine.

Also unhappy with _cwd_relative and _resolve_snapshot_paths

Agreed. That whole thing has to be solved in a different way.