Restoring your first topic

In this guide, you will restore a topic from a Backup to a Kafka cluster. This is a continuation from the Creating your first backup guide, where you created a backup of a Kafka topic.

You will configure a Restore using:

The Storage where the topic data is backed up to.
The EventHub where the data will be restored to.
The Credentials to authenticate with the Kafka cluster.

Pre-requisites

To follow this guide, make sure you have the following tools available:

A Kubernetes cluster with Kannika Armory installed.
A running Kafka cluster
The kubectl CLI tool
Access to the Kubernetes API.

Step 1: Draft a Restore

First, you will start with defining a Draft Restore. A Draft Restore contains some basic information about the Restore, but does not contain any topics to restore yet. This allows you to use it as a working document and make changes before starting the Restore.

Create the following Restore resource in Kubernetes:

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: "my-restore"
spec:
  source: "my-volume-storage"

  sink: "my-kafka-cluster"
  sinkCredentialsFrom:
    credentialsRef:
      name: "sasl-creds"

  enabled: false            # Optional, defaults to false

  config: {}

Configure the restore using the configureRestore GraphQL mutation:

mutation {
  configureRestore(
    input: {
      id: "my-restore",
      sourceStorageName: "my-volume-storage",
      # or sourceBackupName: "my-backup",

      sinkEventHubName: "my-kafka-cluster",
17 collapsed lines
      sinkCredentialsName: "sasl-creds",

      topics: [] #
    }
  ) {
    ...on RestoreConfigured {
      restore {
        name
     }
     ...on RestoreConfigurationFailed {
       problems {
         message
         type
       }
     }
  }
}

First, create the restore using the POST /rest/restores endpoint:

POST /rest/restores HTTP/1.1
Content-Type: application/json
{
  "id": "my-restore",
  "source": "my-volume-storage",
  "sink": "my-kafka-cluster",
  "sink_credentials": "sasl-creds",
}

In this example:

The Restore is named my-restore.
The Restore will read data from the Storage my-volume-storage.
The Restore will restore data back to the EventHub my-kafka-cluster.
The Restore will use the Credentials sasl-creds to authenticate with the Kafka cluster.

Step 2: Configure the topics

Next, you will configure the topics to restore.

Let’s add the topic my-topic to the Restore:

Add the following configuration to the Restore:

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: "my-restore"
spec:
9 collapsed lines
  sink: "my-volume-storage"

  source: "my-kafka-cluster"
  sourceCredentialsFrom:
    credentialsRef:
      name: "sasl-creds"

  enabled: false

  config:
    mapping:
      my-topic:              # Topic in the Storage
        target: "my-topic"   # Topic to restore to in the Kafka cluster

In this example:

The Restore will read the data from my-topic, as defined by the key used in the mapping field. which was backed up in the previous guide, and stored in the Storage my-volume-storage.
The Restore will restore the data back to the topic my-topic in the EventHub my-kafka-cluster, defined by the target field.

Configure the topics to restore using the configureRestoreTopics GraphQL mutation:

mutation {
  configureRestoreTopics(
    input: {
      restoreId: "my-restore",

      topicsToAdd: [
        {
          topic: "my-topic"
        }
      ]
14 collapsed lines
    }
  ) {
    ...on RestoreTopicsConfigured {
      restore {
        id
     }
     ...on ConfigureRestoreTopicsFailed {
       problems {
         message
         type
       }
     }
  }
}

In this example:

The Restore will restore the topic my-topic back to the the EventHub my-kafka-cluster.

Mapping features are not fully supported in the GraphQL API yet.

Add the topics to restore using the POST /rest/restores endpoint:

POST /rest/restores HTTP/1.1
Content-Type: application/json
{
  "name": "my-restore",
  "source": "my-volume-storage",
  "sink": "my-kafka-cluster",
  "sink_credentials": "sasl-creds",
  "topics": ["my-topic"]
}

In this example:

The Restore will restore the topic my-topic back to the the EventHub my-kafka-cluster.

Mapping features are not fully supported in the REST API yet.

Step 3: Start the Restore

Finally, now that all preparations are done, you are ready to start the restore!

Start the Restore by setting the enabled field to true:

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: "my-restore"
spec:
  sink: "my-volume-storage"

  source: "my-kafka-cluster"
  sourceCredentialsFrom:
    credentialsRef:
      name: "sasl-creds"

 enabled: false
 enabled: true

  config:
    mapping:
      my-topic:
        target: "my-topic"

Start the restore using the startRestore GraphQL mutation:

mutation {
  startRestore(
    input: {
      id: "my-restore"
    }
  ) {
    ...on RestoreStarted {
      restore {
10 collapsed lines
        id
     }
     ...on StartRestoreFailed {
       problems {
         message
         type
       }
     }
  }
}

Start the restore using the POST /rest/restores/{id}/start endpoint:

POST /rest/restores/my-restore/start HTTP/1.1

Step 4: Verify that the Restore is running

To verify that the restore is running, run the following command:

$ kubectl get restores

You should see output similar to the following:

NAME         STATUS
my-restore   🚀 Restoring

You should also see the associated Job running:

$ kubectl get jobs

You should see output similar to the following:

NAME                COMPLETIONS   DURATION   AGE
my-restore-a1b2c3   0/1           1s         1m

To verify that the restore is running, run the following query:

query {
  getRestores(first: 10) {
    edges {
      node {
        id
        uuid
        status
      }
    }
2 collapsed lines
  }
}

You should see output similar to this:

{
  "data": {
    "getRestores": {
      "edges": [
        {
          "node": {
            "id": "my-restore",
            "uuid": "40b31171-913e-4b95-9c77-ad71db981dc0",
            "status": "RESTORING",
6 collapsed lines
          }
        }
      ]
    }
  }
}

To verify that the restore is running, run the following command:

GET /rest/restores HTTP/1.1

You should see output similar to this:

HTTP/1.1 200
13 collapsed lines
Vary: Origin
Vary: Access-Control-Request-Method
Vary: Access-Control-Request-Headers
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY
Content-Type: application/json
Transfer-Encoding: chunked
Date: Thu, 30 May 2024 12:41:00 GMT

{
  "restores": [
    {
      "id": "my-restore",
      "uuid": "40b31171-913e-4b95-9c77-ad71db981dc0",
      "source": "my-volume-storage",
      "sink": "my-kafka-cluster",
      "sink_credentials": "RESTORING",
      "status": "RESTORING",
      "topics": ["my-topic"],
    }
  ]
}

Once the restore completes, all data has been restored to the my-topic topic in Kafka cluster.

Step 5: Clean up the Restore (optional)

To remove the Restore and its associated resources, simply delete it:

$ kubectl delete restore my-restore

To remove the Restore and its associated resources, simply delete it using the deleteRestore mutation:

query {
  deleteRestore(
    input: {
      id: "my-restore"
    }
  ) {
    ...on RestoreDeleted {
      restore {
        id
10 collapsed lines
      }
    }
    ...on DeleteRestoreFailed {
      problems {
        message
        type
      }
    }
  }
}

To remove the Restore and its associated resources, simply delete it using the DELETE /rest/restores/my-restore endpoint:

DELETE /rest/restores/my-restore HTTP/1.1

Conclusion

You have successfully restored a Kafka topic from a Backup to a Kafka cluster!

Have a look at the Restoring data from a Backup for a more thorough step-by-step guide on restoring data from a backup, or the Restore reference for more information on the available configuration options.

If you require further assistance, don’t hesitate to reach out to us on Slack!