Segments

Armory writes its data to a series of files called segments.

Note

In previous versions, segments used to be called partitions. They have been renamed to avoid confusing them with Kafka partitions.

This section describes configuration options available to tune a Backup’s behaviour when generating these files.

Segment size

When a segment reaches a certain size, it is closed and a new segment is opened. The size of these segments can be configured with the .spec.segmentRolloverTriggers.size setting:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup
spec:
  sink: "sink"
  source: "source"
  segmentRolloverTriggers:
    size: 10 MiB

Note

Armory uses 100 MiB for its default segment size.

When using a remote storage (S3, GCS, etc.), it may be useful to decrease the default value to force the backup process to close segments (i.e. to commit the data) more frequently.

Periodic Rollover

Slow streams may not fill an entire segment for a long time, thus leaving old data uncommitted to persistent storage for a while.

You can force a Backup to perform a rollover after some duration if the max segment size has not yet been reached:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup
spec:
  sink: "sink"
  source: "source"
  segmentRolloverTriggers:
    # Close a segment once it reaches 25MiB ...
    size: 25 MiB
    # ... or after 1h, whichever happens first.
    timeoutSeconds: 3600

Note

This timeout setting only takes effect when a stream becomes idle, and you may notice the rollover doesn’t happen on a timely manner for busy streams.

This is the intended behaviour as this setting is meant to protect against data loss for slow streams.

For example, say you have a stream of events that occur every few seconds, and you have configured timeoutSeconds: 3600 and size: 100MiB. Initially, the backup process will be busy processing the history of events, and rollovers will be triggered when segments reach 100MiB.

When the backup has caught up with the latest events, things will become quieter and segments will fill much more slowly. Only then will the timeout become useful as segments will be closed and committed to persistent storage every hour.