Creating your first backup
In this guide, you will create a backup of a Kafka topic. The backup will be stored on a Persistent Volume .
You will configure:
- The EventHub to define the Kafka cluster.
- The Credentials to authenticate with the Kafka cluster.
- The Storage to store data on.
- The Backup using all of the above.
Pre-requisites
To follow this guide, make sure you have the following tools available:
- A Kubernetes cluster with Kannika Armory installed.
- A running Kafka cluster
- The kubectl CLI tool
- Access to the Kubernetes API.
- A Kubernetes cluster with Kannika Armory installed.
- A running Kafka cluster
- A GraphQL client like Postman.
- Access to the GraphQL API.
- A Kubernetes cluster with Kannika Armory installed.
- A running Kafka cluster
- An HTTP client like Postman.
- Access to the REST API.
Step 1: Configure an EventHub
An EventHub is a Kannika Armory resource that is used to depict an external system that contains event data. In this step, you will create a Kafka EventHub.
Create the following resource in Kubernetes:
apiVersion: kannika.io/v1alphakind: EventHubmetadata: name: my-kafka-clusterspec: kafka: properties: bootstrap.servers: broker:9092
mutation { configureKafkaEventHub( input: { name: "my-kafka-cluster", bootstrapServers: "broker:9092" }14 collapsed lines
) { ...on KafkaEventHubConfigured { eventHub { name } } ...on KafkaEventHubConfigurationFailed { problems { message type } } }}
POST /rest/eventhubs HTTP/1.1Content-Type: application/json
{ "name": "my-kafka-cluster", "kafka" : { "bootstrap_servers": "broker:9092" }}
In this example:
-
A Kafka EventHub named
my-kafka-cluster
is created. -
The EventHub points to a Kafka cluster at
broker:9092
.
To authenticate with the Kafka cluster, you’ll need to configure Credentials. In this example, you’ll use SASL/PLAIN Credentials with a username and password.
Create the following resources in Kubernetes:
# Define the SASL CredentialsapiVersion: kannika.io/v1alphakind: Credentialsmetadata: name: sasl-credsspec: sasl: mechanism: PLAIN usernameFrom: secretKeyRef: name: sasl-creds # References the Secret below key: username passwordFrom: secretKeyRef: name: sasl-creds # References the Secret below key: password sslConf: enabled: true---# Define the Secret that contains the username and passwordapiVersion: v1kind: Secrettype: Opaquemetadata: name: sasl-credsdata: username: c2FzbC11c2VybmFtZQ== # sasl-username, base64 encoded password: c2FzbC1wYXNzd29yZA=== # sasl-password, base64 encoded
mutation { configureSaslCredentials( input: { name: "sasl-creds", mechanism: PLAIN, username: "sasl-username", password: "sasl-password", useSsl: true }15 collapsed lines
) { ... on SaslCredentialsConfigured { credentials { name } } ... on SaslCredentialsConfigurationFailed { problems { message type } }
}}
POST /rest/credentials HTTP/1.1Content-Type: application/json
{ "name": "sasl-creds", "sasl": { "mechanism": "PLAIN", "username": "sasl-username", "password": "sasl-password", "use_ssl": true }}
In this example:
-
The Credentials are used for SASL/PLAIN authentication with the username
sasl-username
and passwordsasl-password
, which are stored in a Secret namedsasl-creds
. -
SSL encryption is enabled. This correlates to the Kafka property
security.protocol
being set toSASL_SSL
.
Step 2: Configure a Storage
Next, configure the Storage which will be used to store the backup data. In this example, a Volume Storage is used, which will store the data on a Persistent Volume .
Create the following Storage resource in Kubernetes:
apiVersion: kannika.io/v1alphakind: Storagemetadata: name: "my-volume-storage"spec: volume: capacity: 10Gi
mutation { configureVolumeStorage( input: { name: "my-volume-storage", capacity: "10Gi" }14 collapsed lines
) { ... on VolumeStorageConfigured { storage { name } } ... on VolumeStorageConfigurationFailed { problems { message type } } }}
POST /rest/storages HTTP/1.1Content-Type: application/json
{ "name": "my-volume-storage", "volume": { "capacity": "10Gi" }}
In this example:
-
A Volume Storage named
my-volume-storage
is created. -
The volume has a capacity of
10Gi
.
Step 3: Configure the backup
Finally, create a Backup which will back up data from the EventHub on the Storage.
Create the following Backup resource in Kubernetes:
apiVersion: kannika.io/v1alphakind: Backupmetadata: name: "my-backup"spec: source: "my-kafka-cluster" sourceCredentialsFrom: credentialsRef: name: "sasl-creds" sink: "my-volume-storage" enabled: true # Optional, defaults to true streams: - topic: "my-topic" enabled: true # Optional, defaults to true
Configure the backup using the configureBackup
GraphQL mutation:
mutation { configureBackup( input: { name: "my-backup", sourceEventHubName: "my-kafka-cluster", sourceCredentialsName: "sasl-creds", sinkStorageName: "my-volume-storage", }13 collapsed lines
) { ...on BackupConfigured { backup { name } ...on BackupConfigurationFailed { problems { message type } } }}
Then,
import topics using the enableBackupStreams
GraphQL mutation:
mutation { configureBackupTopics( input: { backupName: "my-backup", topics: ["my-topic"] enable: true }14 collapsed lines
) { ...on BackupTopicsConfigured { backup { name } } ...on ConfigureBackupTopicsFailed { problems { message type } } }}
First,
create the backup using the POST /rest/backups
endpoint:
POST /rest/backups HTTP/1.1Content-Type: application/json{ "name": "my-backup", "source": "my-kafka-cluster", "source_credentials": "sasl-creds", "sink": "my-volume-storage",}
Then,
import topics using the POST /rest/backups/{backupName}/topics
endpoint:
POST /rest/backups/my-backup/topics HTTP/1.1Content-Type: application/json
{ "topics": [ "my-topic" ], "enable": true}
In this example:
-
The Backup is named
my-backup
. -
The Backup uses the Credentials
sasl-creds
to authenticate with the Kafka cluster. -
The topic
my-topic
is added to the Backup.
Step 4: Verify that the backup is running
Congratulations on making it this far! You have now created your first backup, and it should be running.
To verify that the backup is running, run the following command:
$ kubectl get backups
You should see output similar to the following:
NAME STATUSmy-backup 🚀 Streaming
You should also see the associated Pod running:
$ kubectl get pods
You should see output similar to the following:
NAME READY STATUS RESTARTS AGEmy-backup-75465ff49-kcw8p 1/1 Running 0 1s
To verify that the backup is running, run the following query:
query { getBackups(first: 10) { edges { node { name uuid status paused config {17 collapsed lines
sourceRef { name eventHub { ...on KafkaEventHub { name properties { key value } } } } sinkRef { name } } } } }}
You should see output similar to this:
{ "data": { "getBackups": { "edges": [ { "node": { "name": "my-backup", "uuid": "40b31171-913e-4b95-9c77-ad71db981dc0", "status": "RUNNING",21 collapsed lines
"paused": false, "config": { "sourceRef": { "name": "my-kafka-cluster", "eventHub": { "name": "my-kafka-cluster", "bootstrapServers": "broker:9092" ] } }, "sinkRef": { "name": "my-volume-storage", }, "streams": [ "topic": "my-topic" ] } } } ] } }}
To verify that the backup is running, run the following command:
GET /rest/backups HTTP/1.1
You should see output similar to this:
HTTP/1.1 20013 collapsed lines
Vary: OriginVary: Access-Control-Request-MethodVary: Access-Control-Request-HeadersX-Content-Type-Options: nosniffX-XSS-Protection: 0Cache-Control: no-cache, no-store, max-age=0, must-revalidatePragma: no-cacheExpires: 0X-Frame-Options: DENYContent-Type: application/jsonTransfer-Encoding: chunkedDate: Thu, 30 May 2024 12:41:00 GMT
{ "backups": [ { "name": "my-backup", "uuid": "40b31171-913e-4b95-9c77-ad71db981dc0", "config": { "source": "my-kafka-cluster", "sink": "my-volume-storage", "topics": ["my-topic"], "source_credentials": "sasl-creds" } } ]}
The backup is now running, and will continue to run until you delete it, or pause it.