跳到主要内容

Application Migration

Kosmos provides application migration functionality to help users migrate existing applications from subclusters to the Kosmos control plane cluster.

Application Migration Solution​

Introduction​

In the Kosmos multi-cluster design architecture, users can interact with the kube-apiserver of the control plane cluster using tools such as kubectl and client-go to create deployments or statefulset applications, etc. The actual Pod instances are running in the subclusters.

However, for existing applications in the subclusters that were not created through the Kosmos control plane, these applications cannot be viewed and managed in the control plane cluster.

Kosmos provides application migration functionality to support the migration of applications in a namespace to the control plane cluster. The entire process does not require restarting the application Pod instances, ensuring minimal impact on business operations.

Design Details​

Application migration mainly consists of three processes: application backup -> deletion of owner objects -> rebuilding applications in the control plane.

Application Backup​

Kosmos first backs up all namespaced-level resources in the target namespace, as well as the dependent cluster-level resources, such as cluster roles, cluster role bindings, persistent volumes, etc. The backup files are stored in PVCs in Kosmos.

Deletion of Owner Objects​

The subclusters in Kosmos only run Pods, and their owner StatefulSets or ReplicaSets need to be deleted and rebuilt in the Kosmos control plane. Similarly, the owner Deployment of a ReplicaSet, as well as the owner StatefulSet and Deployment, need to be deleted and rebuilt in the Kosmos control plane.

By using a top-down cascading deletion of owner objects (e.g., deleting the Deployment first and then the ReplicaSet), the Pods are not affected and remain in a running state.

Rebuilding Applications in the Control Plane​

Based on the backup resources, the control plane cluster creates all the migrated resources, including namespaces, pods, deployments, config maps, service accounts, etc. In order to maintain consistency with the subcluster Pods and keep them in a running state, the applications are rebuilt using a bottom-up approach (e.g., creating the Pod first and then the ReplicaSet).

CRD API​

The PromotePolicy CRD API is provided for configuring the migration policy. PromotePolicy is a cluster-wide CRD API. Here is an example of how to use it:

apiVersion: kosmos.io/v1alpha1
kind: PromotePolicy
metadata:
  name: promote-policy-sample
spec:
  includedNamespaces:
    - namespace1
    - namespace2
  excludedNamespaceScopedResources:
    - events
    - events.events.k8s.io
    - endpoints
    - endpointslices.discovery.k8s.io
  clusterName: member-cluster1

where:

  • includedNamespaces: The namespaces to be migrated.
  • excludedNamespaceScopedResources: The namespace-level resources that should not be migrated. It is recommended to keep the example configuration and add additional configurations as needed based on actual requirements.
  • clusterName: The name of the Kosmos subcluster.

Rollback​

Kosmos supports rollback functionality for migrations. After a successful migration, the existing applications in the subcluster can be restored to their initial state. Simply edit the PromotePolicy YAML file and add the configuration 'rollback'=true.

apiVersion: kosmos.io/v1alpha1
kind: PromotePolicy
metadata:
  name: promote-policy-sample
spec:
  rollback: "true"
  includedNamespaces:
    - namespace1
    - namespace2
  excludedNamespaceScopedResources:
    - events
    - events.events.k8s.io
    - endpoints
    - endpointslices.discovery.k8s.io
  clusterName: member-cluster1

Test Plans​

Preparation​

First, you need to deploy Kosmos (the clustertree module must be installed) and add a subcluster.

Create an Existing Application in the Subcluster​

Deploy an nginx application in the subcluster as an example.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: nginx-test
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
      deletionGracePeriodSeconds: 30
    spec:
      containers:
      - name: nginx
        image: nginx:1.14-alpine
        ports:
        - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: nginx-test
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP

Create a Migration Policy​

# kubectl apply -f promote-nginx.yaml
apiVersion: kosmos.io/v1alpha1
kind: PromotePolicy
metadata:
  name: promote-policy-example
spec:
  includedNamespaces:
    - nginx-test
  excludedNamespaceScopedResources:
    - events
    - events.events.k8s.io
    - endpoints
    - endpointslices.discovery.k8s.io
  clusterName: cluster-36-28

Check the Migration Result​

Check the progress of the migration:

# kubectl describe promotepolicy promote-policy-example
Name:         promote-policy-example
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  kosmos.io/v1alpha1
Kind:         PromotePolicy
Metadata:
  Creation Timestamp:  2024-03-11T10:57:47Z
  Generation:          3
  Resource Version:  405947183
  UID:               0e32dd93-c370-4874-b9a7-37a6894cd373
Spec:
  Cluster Name:  cluster-36-28
  Excluded Namespace Scoped Resources:
    events
    events.events.k8s.io
    endpoints
    endpointslices.discovery.k8s.io
    controllerrevisions.apps
  Included Namespaces:
    nginx-test
Status:
  Backedup File:  /data/backup/promote-policy-sample20240311-104907
  Phase:          Completed
Events:           <none>

When Status.Phase is 'Completed', it means the migration is successful. At this point, you can view and manage all applications in the nginx-test namespace in the control plane cluster.

Rollback​

Edit the promote-nginx.yml file and add the configuration ‘rollback’=true:

# kubectl apply -f promote-nginx.yaml
apiVersion: kosmos.io/v1alpha1
kind: PromotePolicy
metadata:
  name: promote-policy-example
spec:
  rollback: "true"
  includedNamespaces:
    - nginx-test
  excludedNamespaceScopedResources:
    - events
    - events.events.k8s.io
    - endpoints
    - endpointslices.discovery.k8s.io
  clusterName: cluster-36-28

Check the rollback result:

# kubectl describe promotepolicy promote-policy-example
Name:         promote-policy-example
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  kosmos.io/v1alpha1
Kind:         PromotePolicy
Metadata:
  Creation Timestamp:  2024-03-11T10:57:47Z
  Generation:          5
  Resource Version:  405953692
  UID:               0e32dd93-c370-4874-b9a7-37a6894cd373
Spec:
  Cluster Name:  cluster-36-28
  Excluded Namespace Scoped Resources:
    events
    events.events.k8s.io
    endpoints
    endpointslices.discovery.k8s.io
    controllerrevisions.apps
  Included Namespaces:
    nginx-test
Status:
  Backedup File:  /data/backup/promote-policy-sample20240311-104907
  Phase:          RolledBack
Events:           <none>

When Status.Phase is 'RolledBack', it means the rollback is successful. At this point, the applications in the nginx-test namespace cannot be queried in the control plane cluster.