#Managing Nodes on Huawei Cloud Stack

This document explains how to manage worker nodes using Cluster API Machine resources on the Huawei Cloud Stack platform.

#Prerequisites

#Overview

Worker nodes are managed through Cluster API Machine resources, providing declarative and automated node lifecycle management. The deployment process involves:

1. Machine Configuration Pool - Network settings for worker nodes
2. Machine Template - VM specifications
3. Bootstrap Configuration - Node initialization settings
4. Machine Deployment - Orchestration of node creation and management

#Worker Node Deployment

#Step 1: Configure Machine Configuration Pool

The HCSMachineConfigPool defines the network configuration for worker node VMs. You must plan and configure the IP addresses, hostnames, and other network parameters before deployment.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HCSMachineConfigPool
metadata:
  name: <worker-pool-name>
  namespace: cpaas-system
spec:
  configs:
    - hostname: worker-1
      networks:
        - subnetName: <subnet-name>
          ipAddress: 192.168.1.21
    - hostname: worker-2
      networks:
        - subnetName: <subnet-name>
          ipAddress: 192.168.1.22
    - hostname: worker-3
      networks:
        - subnetName: <subnet-name>
          ipAddress: 192.168.1.23

#Step 2: Configure Machine Template

The HCSMachineTemplate defines the VM specifications for worker nodes.

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: HCSMachineTemplate
metadata:
  name: <worker-machine-template>
  namespace: cpaas-system
spec:
  template:
    spec:
      imageName: <vm-image-name>
      flavorName: <instance-flavor>
      availabilityZone: <availability-zone>
      rootVolume:
        type: SSD
        size: 100
      configPoolRef:
        name: <worker-pool-name>
      dataVolumes:
        - size: 100
          type: SSD
          mountPath: /var/lib/kubelet
          format: xfs
        - size: 100
          type: SSD
          mountPath: /var/lib/containerd
          format: xfs
        - size: 100
          type: SSD
          mountPath: /var/cpaas
          format: xfs

*Required when dataVolumes is specified.

#Step 3: Configure Bootstrap Template

The KubeadmConfigTemplate defines the bootstrap configuration for worker nodes.

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: <worker-config-template>
  namespace: cpaas-system
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"

#Step 4: Configure Machine Deployment

The MachineDeployment orchestrates the creation and management of worker nodes.

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: <worker-machine-deployment>
  namespace: cpaas-system
spec:
  clusterName: <cluster-name>
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
  selector:
    matchLabels: null
  template:
    spec:
      clusterName: <cluster-name>
      version: <kubernetes-version>
      nodeDrainTimeout: 1m
      nodeDeletionTimeout: 5m
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: <worker-config-template>
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: HCSMachineTemplate
        name: <worker-machine-template>

#Node Management Operations

This section covers common operational tasks for managing worker nodes.

#Scaling Worker Nodes

Worker node scaling allows you to adjust cluster capacity based on workload demands.

#Adding Worker Nodes

Increase the number of worker nodes to handle increased workload.

Procedure:

1. Check Current Node Status

   # List all machines in the cluster
   kubectl get machines -n cpaas-system
   
   # List machines for a specific MachineDeployment
   kubectl get machines -n cpaas-system -l cluster.x-k8s.io/deployment-name=<worker-machine-deployment>

2. Extend Configuration Pool

   Add new IP configurations to the pool for the additional nodes.

   kubectl get hcsmachineconfigpool <worker-pool-name> -n cpaas-system -o yaml

   Modify the pool to include new IP entries, then apply:

   kubectl apply -f <updated-pool-config.yaml>

3. Scale Up the MachineDeployment

   Update the replicas field to the desired number of nodes:

   kubectl patch machinedeployment <worker-machine-deployment> -n cpaas-system \
     --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": <new-replica-count>}]'

4. Monitor the Scaling Progress

   # Watch machines being created
   kubectl get machines -n cpaas-system -w
   
   # Check MachineDeployment status
   kubectl get machinedeployment <worker-machine-deployment> -n cpaas-system

#Removing Worker Nodes

Decrease the number of worker nodes to reduce cluster capacity.

Procedure:

1. Scale Down the MachineDeployment

   kubectl patch machinedeployment <worker-machine-deployment> -n cpaas-system \
     --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": <new-replica-count>}]'

2. Monitor the Removal Progress

   kubectl get machines -n cpaas-system -w

   The Cluster API controller will:

   • Drain the selected nodes (evict pods if possible)
   • Delete the underlying VMs from the HCS platform
   • Remove the machine resources

#Upgrading Machine Infrastructure

To upgrade worker machine specifications (CPU, memory, disk, VM image), follow these steps:

1. Create New Machine Template

   Copy the existing HCSMachineTemplate and modify the required values:

   • imageName - VM image

   • flavorName - Instance type

   • rootVolume.size - System disk size

   • dataVolumes - Data disk configurations

     kubectl get hcsmachinetemplate <current-template> -n cpaas-system -o yaml > new-template.yaml

2. Deploy New Template

   kubectl apply -f new-template.yaml -n cpaas-system

3. Update Machine Deployment

   Modify the MachineDeployment to reference the new template:

   kubectl patch machinedeployment <worker-machine-deployment> -n cpaas-system \
     --type='merge' -p='{"spec":{"template":{"spec":{"infrastructureRef":{"name":"<new-template>"}}}}}'

4. Monitor Rolling Update

   kubectl get machines -n cpaas-system -w

#Upgrading Kubernetes Version

Kubernetes version upgrades require coordinated updates to both the MachineDeployment and the underlying VM template.

Procedure:

1. Update Machine Template

   Create a new HCSMachineTemplate with an updated imageName that supports the target Kubernetes version.

2. Update MachineDeployment

   Modify the following fields:

   • spec.template.spec.version - Target Kubernetes version

   • spec.template.spec.infrastructureRef.name - New machine template name

     kubectl patch machinedeployment <worker-machine-deployment> -n cpaas-system \
       --type='merge' -p='{"spec":{"template":{"spec":{"version":"<kubernetes-version>","infrastructureRef":{"name":"<new-template>"}}}}}'

3. Monitor Upgrade

   Verify that new nodes join the cluster with the correct Kubernetes version:

   kubectl get nodes

#Verification

After deploying worker nodes, verify the deployment:

# Check machine status
kubectl get machines -n cpaas-system

# Verify nodes are Ready
kubectl get nodes

# Check MachineDeployment status
kubectl get machinedeployment -n cpaas-system

#Troubleshooting

#Viewing Controller Logs

# View HCS controller logs
kubectl logs -n cpaas-system deployment/hcs-controller-manager

# View machine details
kubectl describe hcsmachine <machine-name> -n cpaas-system

#Common Issues

Node fails to join cluster

• Verify the VM template matches the Kubernetes version
• Check network connectivity between nodes
• Ensure the configuration pool has available entries

Machine stuck in provisioning

• Check HCS platform for resource availability
• Verify credentials and permissions
• Review controller logs for error messages