Skip to content

testing-supported-clusters.md

Latest commit

 

History

History
536 lines (483 loc) · 18.8 KB

testing-supported-clusters.md

File metadata and controls

536 lines (483 loc) · 18.8 KB

import Pool from "../partials/cmx/_openshift-pool.mdx"

CMX cluster and VM types

This topic lists the supported cluster and VM types that you can provision with Compatibility Matrix (CMX).

Overview

CMX can create VMs, VM-based clusters (such as kind, k3s, RKE2, and Red Hat OpenShift OKD), and cloud-managed clusters (such as EKS, GKE and AKS):

  • Cloud-based Kubernetes distributions are run in a Replicated managed and controlled cloud account to optimize and deliver a clusters quickly and reliably. The Replicated account has control planes ready and adds a node group when you request it, making the cluster available much faster than if you try to create your own cluster with your own cloud account.

  • VMs and VM-based clusters run on Replicated bare metal servers located in several data centers, including data centers physically in the European Union.

VM clusters

This section lists the supported VM cluster distributions for clusters created with CMX.

VM-based clusters refers to clusters that run on Hetzner servers with the CMX cluster provisioner. This allows for greater flexibility than with Cloud Clusters like AWS, EKS, etc. For example, with VM-based distributions, CMX offers warm pools to make Openshift startup times very fast.

kind

CMX supports creating kind clusters.

Type Description
Supported Kubernetes Versions {/* START_kind_VERSIONS */}1.30.13, 1.31.14, 1.32.11, 1.33.7, 1.34.3, 1.35.1{/* END_kind_VERSIONS */}
Supported Instance Types See Replicated Instance Types
Node Groups No
Node Auto Scaling No
Nodes Supports a single node.
IP Family Supports `ipv4` or `dual`.
Limitations See Limitations
Common Use Cases Smoke tests

k3s

CMX supports creating k3s clusters.

Type Description
Supported k3s Versions The upstream k8s version that matches the Kubernetes version requested.
Supported Kubernetes Versions {/* START_k3s_VERSIONS */}1.32.0, 1.32.1, 1.32.2, 1.32.3, 1.32.4, 1.32.5, 1.32.6, 1.32.7, 1.32.8, 1.32.9, 1.32.10, 1.32.11, 1.32.12, 1.32.13, 1.33.0, 1.33.1, 1.33.2, 1.33.3, 1.33.4, 1.33.5, 1.33.6, 1.33.7, 1.33.8, 1.33.9, 1.33.10, 1.34.1, 1.34.2, 1.34.3, 1.34.4, 1.34.5, 1.34.6, 1.35.0, 1.35.1, 1.35.2, 1.35.3{/* END_k3s_VERSIONS */}
Supported Instance Types See Replicated Instance Types
Node Groups Yes
Node Auto Scaling No
Nodes Supports multiple nodes.
IP Family Supports `ipv4`.
Limitations For additional limitations that apply to all distributions, see Limitations.
Common Use Cases
  • Smoke tests
  • Customer release tests

RKE2 (Beta)

CMX supports creating RKE2 clusters.

Type Description
Supported RKE2 Versions The upstream k8s version that matches the Kubernetes version requested.
Supported Kubernetes Versions {/* START_rke2_VERSIONS */}1.32.0, 1.32.1, 1.32.2, 1.32.3, 1.32.4, 1.32.5, 1.32.6, 1.32.7, 1.32.8, 1.32.9, 1.32.10, 1.32.11, 1.32.12, 1.32.13, 1.33.0, 1.33.1, 1.33.2, 1.33.3, 1.33.4, 1.33.5, 1.33.6, 1.33.7, 1.33.8, 1.33.9, 1.33.10, 1.34.1, 1.34.2, 1.34.3, 1.34.4, 1.34.5, 1.34.6, 1.35.0, 1.35.1, 1.35.2, 1.35.3{/* END_rke2_VERSIONS */}
Supported Instance Types See Replicated Instance Types
Node Groups No
Node Auto Scaling No
Nodes Supports a single node.
IP Family Supports `ipv4`.
Limitations For additional limitations that apply to all distributions, see Limitations.
Common Use Cases
  • Smoke tests
  • Customer release tests

OpenShift OKD

CMX supports creating Red Hat OpenShift OKD clusters, which is the community distribution of OpenShift, using CodeReady Containers (CRC).

OpenShift clusters are provisioned with two users:

  • (Default) A kubeadmin user with cluster-admin priviledges. Use the kubeadmin user only for administrative tasks such as creating new users or setting roles.
  • A developer user with namespace-scoped priviledges. The developer user can be used to better simulate access in end-customer environments.

By default, kubeconfig context is set to the kubeadmin user. To switch to the developer user, run the command oc login --username developer.

Type Description
Supported OpenShift Versions {/* START_openshift_VERSIONS */}4.16.0-okd, 4.17.0-okd, 4.18.0-okd, 4.19.0-okd, 4.20.0-okd, 4.21.0-okd{/* END_openshift_VERSIONS */}
Supported Instance Types See Replicated Instance Types
Node Groups Yes
Node Auto Scaling No
Nodes Supports multiple nodes for versions 4.13.0-okd and later.
IP Family Supports `ipv4`.
Limitations
  • OpenShift does not support r1.small instance types.

  • OpenShift builds take approximately 17 minutes.

For additional limitations that apply to all distributions, see Limitations.
Common Use Cases Customer release tests

Embedded Cluster

CMX supports creating clusters with Replicated Embedded Cluster. For more information, see Embedded Cluster Overview.

Type Description
Supported Embedded Cluster Versions Any valid release sequence that has previously been promoted to the channel where the customer license is assigned. Version is optional and defaults to the latest available release on the channel.
Supported Instance Types See Replicated Instance Types
Node Groups Yes
Nodes Supports multiple nodes (alpha).
IP Family Supports `ipv4`.
Limitations
  • The Admin Console UI is not exposed publicly and must be exposed via `kubectl -n kotsadm port-forward svc/kurl-proxy-kotsadm 38800:8800`. The password for the Admin Console is `password`.
  • A valid customer license is required to create an Embedded Cluster.
  • The [cluster prepare](/vendor/testing-how-to#prepare-clusters) command is not supported.

For additional limitations that apply to all distributions, see Limitations.
Common Use Cases Customer release tests

kURL

CMX supports creating kURL clusters.

Type Description
Supported kURL Versions Any promoted kURL installer. Version is optional. For an installer version other than "latest", you can find the specific Installer ID for a previously promoted installer under the relevant **Install Command** (ID after kurl.sh/) on the **Channels > kURL Installer History** page in the Vendor Portal. For more information about viewing the history of kURL installers promoted to a channel, see [Installer History](/vendor/installer-history).
Supported Instance Types See Replicated Instance Types
Node Groups Yes
Node Auto Scaling No
Nodes Supports multiple nodes.
IP Family Supports `ipv4`.
Limitations

Does not work with the Longhorn add-on.

For additional limitations that apply to all distributions, see Limitations.
Common Use Cases Customer release tests

Cloud clusters

This section lists the supported cloud clusters for compatibility testing.

EKS

CMX supports creating AWS EKS clusters.

Type Description
Supported Kubernetes Versions

{/* START_eks_VERSIONS */}1.30, 1.31, 1.32, 1.33, 1.34, 1.35{/* END_eks_VERSIONS */}

Extended Support Versions: 1.30, 1.31, 1.32
Supported Instance Types

m6i.large, m6i.xlarge, m6i.2xlarge, m6i.4xlarge, m6i.8xlarge, m7i.large, m7i.xlarge, m7i.2xlarge, m7i.4xlarge, m7i.8xlarge, m5.large, m5.xlarge, m5.2xlarge, m5.4xlarge, m5.8xlarge, m7g.large (arm), m7g.xlarge (arm), m7g.2xlarge (arm), m7g.4xlarge (arm), m7g.8xlarge (arm), c5.large, c5.xlarge, c5.2xlarge, c5.4xlarge, c5.9xlarge, g4dn.xlarge (gpu), g4dn.2xlarge (gpu), g4dn.4xlarge (gpu), g4dn.8xlarge (gpu), g4dn.12xlarge (gpu), g4dn.16xlarge (gpu)

g4dn instance types depend on available capacity. After a g4dn cluster is running, you also need to install your version of the NVIDIA device plugin for Kubernetes. See [Amazon EKS optimized accelerated Amazon Linux AMIs](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html#gpu-ami) in the AWS documentation.
Node Groups Yes
Node Auto Scaling Yes. Cost will be based on the max number of nodes.
Nodes Supports multiple nodes.
IP Family Supports `ipv4`.
Limitations

You can only choose a minor version, not a patch version. The EKS installer chooses the latest patch for that minor version.

For additional limitations that apply to all distributions, see Limitations.
Common Use Cases Customer release tests

GKE

CMX supports creating Google GKE clusters.

Type Description
Supported Kubernetes Versions {/* START_gke_VERSIONS */}1.32, 1.33, 1.34, 1.35{/* END_gke_VERSIONS */}
Supported Instance Types

n2-standard-2, n2-standard-4, n2-standard-8, n2-standard-16, n2-standard-32, t2a-standard-2 (arm), t2a-standard-4 (arm), t2a-standard-8 (arm), t2a-standard-16 (arm), t2a-standard-32 (arm), t2a-standard-48 (arm), e2-standard-2, e2-standard-4, e2-standard-8, e2-standard-16, e2-standard-32, n1-standard-1+nvidia-tesla-t4+1 (gpu), n1-standard-1+nvidia-tesla-t4+2 (gpu), n1-standard-1+nvidia-tesla-t4+4 (gpu), n1-standard-2+nvidia-tesla-t4+1 (gpu), n1-standard-2+nvidia-tesla-t4+2 (gpu), n1-standard-2+nvidia-tesla-t4+4 (gpu), n1-standard-4+nvidia-tesla-t4+1 (gpu), n1-standard-4+nvidia-tesla-t4+2 (gpu), n1-standard-4+nvidia-tesla-t4+4 (gpu), n1-standard-8+nvidia-tesla-t4+1 (gpu), n1-standard-8+nvidia-tesla-t4+2 (gpu), n1-standard-8+nvidia-tesla-t4+4 (gpu), n1-standard-16+nvidia-tesla-t4+1 (gpu), n1-standard-16+nvidia-tesla-t4+2 (gpu), n1-standard-16+nvidia-tesla-t4+4 (gpu), n1-standard-32+nvidia-tesla-t4+1 (gpu), n1-standard-32+nvidia-tesla-t4+2 (gpu), n1-standard-32+nvidia-tesla-t4+4 (gpu), n1-standard-64+nvidia-tesla-t4+1 (gpu), n1-standard-64+nvidia-tesla-t4+2 (gpu), n1-standard-64+nvidia-tesla-t4+4 (gpu), n1-standard-96+nvidia-tesla-t4+1 (gpu), n1-standard-96+nvidia-tesla-t4+2 (gpu), n1-standard-96+nvidia-tesla-t4+4 (gpu)

You can specify more than one node.
Node Groups Yes
Node Auto Scaling Yes. Cost will be based on the max number of nodes.
Nodes Supports multiple nodes.
IP Family Supports `ipv4`.
Limitations

You can choose only a minor version, not a patch version. The GKE installer chooses the latest patch for that minor version.

For additional limitations that apply to all distributions, see Limitations.
Common Use Cases Customer release tests

AKS

CMX supports creating Azure AKS clusters.

Type Description
Supported Kubernetes Versions {/* START_aks_VERSIONS */}1.33, 1.34, 1.35{/* END_aks_VERSIONS */}
Supported Instance Types

Standard_B2ms, Standard_B4ms, Standard_B8ms, Standard_B16ms, Standard_D2S_v5, Standard_D4S_v5, Standard_D8S_v5, Standard_D16S_v5, Standard_D32S_v5, Standard_D48S_v5, Standard_D2ps_v5 (arm), Standard_D4ps_v5 (arm), Standard_D8ps_v5 (arm), Standard_D16ps_v5 (arm), Standard_D32ps_v5 (arm), Standard_D48ps_v5 (arm), Standard_NC4as_T4_v3 (gpu), Standard_NC8as_T4_v3 (gpu), Standard_NC16as_T4_v3 (gpu), Standard_NC64as_T4_v3 (gpu)

GPU instance types depend on available capacity. After a GPU cluster is running, you also need to install your version of the NVIDIA device plugin for Kubernetes. See [NVIDIA GPU Operator with Azure Kubernetes Service](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/microsoft-aks.html) in the NVIDIA documentation.
Node Groups Yes
Node Auto Scaling Yes. Cost will be based on the max number of nodes.
Nodes Supports multiple nodes.
IP Family Supports `ipv4`.
Limitations

You can choose only a minor version, not a patch version. The AKS installer chooses the latest patch for that minor version.

For additional limitations that apply to all distributions, see Limitations.
Common Use Cases Customer release tests

OKE (Beta)

CMX supports creating Oracle Container Engine for Kubernetes (OKE) clusters.

Type Description
Supported Kubernetes Versions {/* START_oke_VERSIONS */}1.32.10, 1.33.1, 1.34.2, 1.35.0{/* END_oke_VERSIONS */}
Supported Instance Types

VM.Standard2.1, VM.Standard2.2, VM.Standard2.4, VM.Standard2.8, VM.Standard2.16, VM.Standard3.Flex.1, VM.Standard3.Flex.2, VM.Standard3.Flex.4, VM.Standard3.Flex.8, VM.Standard3.Flex.16, VM.Standard.A1.Flex.1 (arm), VM.Standard.A1.Flex.2 (arm), VM.Standard.A1.Flex.4 (arm), VM.Standard.A1.Flex.8 (arm), VM.Standard.A1.Flex.16 (arm)
Node Groups Yes
Node Auto Scaling No.
Nodes Supports multiple nodes.
IP Family Supports `ipv4`.
Limitations

Provising an OKE cluster does take between 8 to 10 minutes. If needed, some timeouts in your CI pipelines might have to be adjusted.

For additional limitations that apply to all distributions, see Limitations.
Common Use Cases Customer release tests

Supported VM types

The following VM types are supported:

Distribution Versions Instance Types
ubuntu 24.04, 22.04 r1.small, r1.medium, r1.large, r1.xlarge, r1.2xlarge. See Replicated Instance Types.
almalinux 8, 9, 10 r1.small, r1.medium, r1.large, r1.xlarge, r1.2xlarge. See Replicated Instance Types.

Replicated instance types {#types}

When creating a VM-based cluster with CMX, you must specify a Replicated instance type.

Type Memory (GiB) VCPU Count
r1.small 8 GB 2 VCPUs
r1.medium 16 GB 4 VCPUs
r1.large 32 GB 8 VCPUs
r1.xlarge 64 GB 16 VCPUs
r1.2xlarge 128 GB 32 VCPUs

Kubernetes version support policy

We do not maintain forks or patches of the supported distributions. When a Kubernetes version in CMX is out of support (EOL), Replicated will attempt to continue to support this version for six months for compatibility testing to support customers who are running out-of-date versions of Kubernetes. In the event that a critical security issue or bug is found and unresolved, we might discontinue support for EOL versions of Kubernetes prior to 6 months post EOL.