Partitions

Armory writes its data to a series of files called partitions.

This section describes configuration options available to tune a Backup’s behaviour when generating these files.

Partition size

When a partition reaches a certain size, it is closed and a new partition is opened. The size of these partitions can be configured with the .spec.partitionRolloverTriggers.size setting:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup
spec:
  sink: "sink"
  source: "source"
  partitionRolloverTriggers:
    size: 10 MiB

Note

Armory uses 100 MiB for its default partition size.

When using a remote storage (S3, GCS, etc.), it may be useful to decrease the default value to force the backup process to close partitions (i.e. to commit the data) more frequently.

Periodic Rollover

Slow streams may not fill an entire partition for a long time, thus leaving old data uncommitted to persistent storage for a while.

You can force a Backup to perform a rollover after some duration if the max partition size has not yet been reached:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup
spec:
  sink: "sink"
  source: "source"
  partitionRolloverTriggers:
    # Close a partition once it reaches 25MiB ...
    size: 25 MiB
    # ... or after 1h, whichever happens first.
    timeoutSeconds: 3600

Note

This timeout setting only takes effect when a stream becomes idle, and you may notice the rollover doesn’t happen on a timely manner for busy streams.

This is the intended behaviour as this setting is meant to protect against data loss for slow streams.

For example, say you have a stream of events that occur every few seconds, and you have configured timeoutSeconds: 3600 and size: 100MiB. Initially, the backup process will be busy processing the history of events, and rollovers will be triggered when partitions reach 100MiB.

When the backup has caught up with the latest events, things will become quieter and partitions will fill much more slowly. Only then will the timeout become useful as partitions will be closed and committed to persistent storage every hour.