Back up compacted topics

This tutorial shows how to use topic property selectors to back up every compacted topic on a cluster in a separate Backup.

Kannika Armory does not compact the backed up data. The storage of a compacted topic keeps every old value and tombstone, and keeps growing over time. With the compacted topics in a separate Backup, you can rotate the storage of that Backup, as described in Schedule backup snapshots, without affecting the backups of the other topics.

Prerequisites

• A Kannika Armory instance available, running on a Kubernetes environment.
• Local installation of the kubectl binary.
• An EventHub named prod-kafka and a Storage named prod-storage in the kannika-data namespace.
• When the Kafka cluster uses ACLs, the DescribeConfigs permission on the topics, in addition to the permissions that are needed to back them up.

Create the Backup

Create a file named compacted-backup.backup.yaml:

compacted-backup.backup.yaml

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: compacted-backup
  namespace: kannika-data
spec:
  source: prod-kafka
  sink: prod-storage
  enabled: true
  topicSelectors:
    matchers:
    - property:
        key: "cleanup.policy"
        operator: Contains
        value: "compact"

The matchers rule selects every topic whose cleanup.policy contains compact. It uses Contains instead of Equals, because cleanup.policy can hold a comma-separated list such as compact,delete.

Full example with EventHub, Storage, and Backup

# EventHub - defines the Kafka cluster connection
apiVersion: kannika.io/v1alpha
kind: EventHub
metadata:
  name: prod-kafka
  namespace: kannika-data
spec:
  kafka:
    properties:
      bootstrap.servers: broker:9092
---
# Storage - defines where backups are stored
apiVersion: kannika.io/v1alpha
kind: Storage
metadata:
  name: prod-storage
  namespace: kannika-data
spec:
  volume:
    capacity: 10Gi
---
# Backup - backs up every compacted topic from the EventHub to the Storage
apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: compacted-backup
  namespace: kannika-data
spec:
  source: prod-kafka
  sink: prod-storage
  enabled: true
  topicSelectors:
    matchers:
    - property:
        key: "cleanup.policy"
        operator: Contains
        value: "compact"

Apply the Backup:

kubectl apply -f compacted-backup.backup.yaml

Using the console

The same rule can be added to an existing Backup in the console:

1. Go to Backups, open the Backup, and select the Topics tab.
2. Click Topic selectors.
3. Under Include topics, click Add rule and select Property.
4. Enter cleanup.policy as the property key, select contains, and enter compact as the value.
5. Click Save changes.

Matching compacted topics in the console.

Summary

In this tutorial, you learned how to back up every compacted topic in a separate Backup, by matching on the cleanup.policy property of the topics.

To keep the storage of the compacted topics from growing indefinitely, rotate the storage of this Backup as described in Schedule backup snapshots.