Segments

Armory writes its data to a series of files called segments.

Note

In previous versions, segments used to be called partitions. They have been renamed to avoid confusing them with Kafka partitions.

This section describes configuration options available to tune a Backup’s behaviour when generating these files.

Segment size

When a segment reaches a certain size, it is closed and a new segment is opened. The size of these segments can be configured with the .spec.segmentRolloverTriggers.size setting:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup
spec:
  sink: "sink"
  source: "source"
  segmentRolloverTriggers:
    size: 10 MiB

Note

Armory uses 100 MiB for its default segment size.

When using a remote storage (S3, GCS, etc.), it may be useful to decrease the default value to force the backup process to close segments (i.e. to commit the data) more frequently.

Periodic Rollover

Slow streams may not fill an entire segment for a long time, thus leaving old data uncommitted to persistent storage for a while.

You can force a Backup to perform a rollover after some duration if the max segment size has not yet been reached:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup
spec:
  sink: "sink"
  source: "source"
  segmentRolloverTriggers:
    # Close a segment once it reaches 25MiB ...
    size: 25 MiB
    # ... or after 1h, whichever happens first.
    timeoutSeconds: 3600

This setting protects against data loss for slow streams, where a segment would otherwise stay open and uncommitted for a long time before reaching its size limit. The timeout starts counting from the first record written to a segment. Rollovers are evaluated as records arrive and once per second while the topic is idle, so a segment may close up to a second after the configured timeout.

For example, say you have a stream of events that occur every few seconds, and you have configured timeoutSeconds: 3600 and size: 100MiB. Initially, the backup process will be busy processing the history of events, and rollovers will be triggered when segments reach 100MiB.

When the backup has caught up with the latest events, segments fill much more slowly and the timeout takes over, so each segment is closed and committed to persistent storage every hour.