feat: add gateway-api support

[l0wl3vel]

Description

• Add kind-cloud-controller-manager to provide Type: Loadbalancer services
• Introduce envoy-gateway as the Gateway API implementation
• Move metal-stack control plane kind cluster into the mini_lab_external docker network
  • can't select container IP in the default docker bridge, which we need for the pre-defined *.nip.io DNS records
• Kept ingress-nginx for now. Still required for Dex, Thanos, Gardener, PowerDNS

WIPs

• Certificates are a bit messed up still (using default-gateway cert for grcp termination)
• Link metal-roles pr branch to run ci in pull request metal-roles PR is merged
• CI is failing - looks like an unrelated timeout

Used AI-Tools ✨

• none used for generation

Closes: #297

Requires: metal-stack/helm-charts#156 and metal-stack/metal-roles#594

Tested configurations

• Sonic
• Dell Sonic
• Gardener (looks good, deploys correctly with metal-stack on GWAPI and Gardener components still on ingress-nginx, further testing required)
• Kamaji
  • non-functional. So likely a wontfix, unless it gets integreated into mini-lab. Only usable in capi-lab, which uses an old pinned version of mini-lab.

[ma-hartma]

Dell Sonic does actually work, but you need credentials to pull from r.metal-stack.io.

[vknabel]

Sadly I got the following error:

deploy-control-plane  | TASK [ansible-common/roles/helm-chart : Copy over custom helm charts] **********
deploy-control-plane  | fatal: [localhost]: FAILED! => 
deploy-control-plane  |     changed: false
deploy-control-plane  |     cmd: /usr/bin/rsync --delay-updates -F --compress --delete-after --archive --out-format='<<CHANGED>>%i
deploy-control-plane  |         %n%L' /helm-charts/charts/metal-control-plane /tmp/helm-chart
deploy-control-plane  |     msg: |-
deploy-control-plane  |         rsync: [sender] change_dir "/helm-charts/charts" failed: No such file or directory (2)
deploy-control-plane  |         rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.4.1]
deploy-control-plane  |     rc: 23

I had the following overrides metal_roles_version: gatewayapi

[l0wl3vel]

Sadly I got the following error:

deploy-control-plane  | TASK [ansible-common/roles/helm-chart : Copy over custom helm charts] **********
deploy-control-plane  | fatal: [localhost]: FAILED! => 
deploy-control-plane  |     changed: false
deploy-control-plane  |     cmd: /usr/bin/rsync --delay-updates -F --compress --delete-after --archive --out-format='<<CHANGED>>%i
deploy-control-plane  |         %n%L' /helm-charts/charts/metal-control-plane /tmp/helm-chart
deploy-control-plane  |     msg: |-
deploy-control-plane  |         rsync: [sender] change_dir "/helm-charts/charts" failed: No such file or directory (2)
deploy-control-plane  |         rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.4.1]
deploy-control-plane  |     rc: 23

I had the following overrides metal_roles_version: gatewayapi

@vknabel fixed in 629cb02

[l0wl3vel]

@Sven-Ric Would you mind taking a look at the network changes?

[Sven-Ric]

It seems like the kind node always ends up in the default kind network on a clean first run. The kind network is read from .env, which is written by env.sh. However the Makefile reads .env before env.sh is invoked and the kind node network falls back to default. Because .env is persistent the bug is masked on all subsequent runs.

On initial run:

# docker inspect metal-control-plane-control-plane
[
    {
        <SNIP>
        "NetworkSettings": {
            <SNIP>
            "Networks": {
                "kind": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "DriverOpts": null,
                    "GwPriority": 0,
                    "NetworkID": "6530b19e41b397d41d37f6a38d6b1bbd74c9ba2b7478df95f6a6270cc84c4d0e",
                    "EndpointID": "6d56f5f0fa83330b85e0b0ebbd04175a93d8586c48492c9b545beb7eeecce015",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.2",
                    "MacAddress": "12:98:42:c8:e4:ec",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "fc00:f853:ccd:e793::1",
                    "GlobalIPv6Address": "fc00:f853:ccd:e793::2",
                    "GlobalIPv6PrefixLen": 64,
                    "DNSNames": [
                        "metal-control-plane-control-plane",
                        "bd976835cec0"
                    ]
                }
            }
        },
        "ImageManifestDescriptor": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "digest": "sha256:21c46cf61fd45873f89e6a1bfcba4b7904dffa84c2bec88aeeca9a0409af4725",
            "size": 743,
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        }
    }
]

On all subsequent runs:

# docker inspect metal-control-plane-control-plane
[
    {
        <SNIP>
        "NetworkSettings": {
            <SNIP>
            "Networks": {
                "mini_lab_internal": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "DriverOpts": null,
                    "GwPriority": 0,
                    "NetworkID": "2734b8f942cae84d8693ecd43ab3bb9d5cd71905faf992fbfe5c3df17ddc376b",
                    "EndpointID": "62f8b2a6eb379bb65f13f6441a9249417fc9ce754218a29b699cd7511b393d29",
                    "Gateway": "172.42.0.1",
                    "IPAddress": "172.42.0.2",
                    "MacAddress": "66:e7:b9:9c:2e:39",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DNSNames": [
                        "metal-control-plane-control-plane",
                        "5b12fbbedfdc"
                    ]
                }
            }
        },
        "ImageManifestDescriptor": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "digest": "sha256:21c46cf61fd45873f89e6a1bfcba4b7904dffa84c2bec88aeeca9a0409af4725",
            "size": 743,
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        }
    }
]

[l0wl3vel]

Thank you so much for checking it out @Sven-Ric. Fixed in baddf29. A few people checked this PR and it worked fine but CI was failing and I had no clue why. You saved me a lot of time 😅