Data Retention

You can configure a Backup to delete old data periodically. Currently, we support these retention policies:

• Keep (the default): all data is kept forever;
• Delete: the data is deleted after some duration you provide.

Policies are defined on a backup using a set of labels, the main one being io.kannika/data-retention-policy to select the policy. See details for each policy below.

The ‘Keep’ policy

When using this policy, your data won’t be deleted at any point. This is the default policy if none is defined, but if you wish to set it explicitly, then set the io.kannika/data-retention-policy to Keep.

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup-example
  labels:
    io.kannika/data-retention-policy: Keep
spec:
  source: "my-eventhub"
  sink: "my-storage"
  streams:
    - topic: "active-stream"

The Delete policy

The Delete policy will mark Kannika Armory partitions as expired after some duration of your choosing. The expiration date of a partition is computed using the latest record timestamp it contains.

When a partition’s expiration date has passed, it may be deleted by the Backup job at any time.

If a partition’s expiration date has not passed yet, it can be updated by restarting the backup job with a different duration.

If the Backup job has been stopped, then the physical files of expired partitions won’t be deleted as it is the Backup’s responsibility to remove them, however, those files will be ignored by an eventual Restore.

The duration is expected to match this regular expression: ([0-9]+(ns|us|ms|[smhdwy]))+.

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup-example
  labels:
    # Mark partitions as expired after 30 days, and remove them.
    io.kannika/data-retention-policy: Delete
    io.kannika/data-retention-policy-delete-after: 30d
spec:
  source: "my-eventhub"
  sink: "my-storage"
  streams:
    - topic: "active-stream"