Tuning

This page lists the settings you can change to fit a Backup to your topics and your cluster.

They live in three places:

• Kafka consumer properties, set on the EventHub or per Backup through sourceAdditionalProps.
• Segment settings, set on the Backup itself and described on the Segments page.
• Pod resources, set on the Backup and described on the Pod Configuration page.

Memory

The memory a Backup uses depends on how much data it holds in flight between the EventHub and the Storage.

The following consumer properties bound that amount. Armory sets its own values for them when you do not.

Property	librdkafka default	Effect
queued.max.messages.kbytes	65536	The maximum amount of data, in kibibytes, the Backup keeps waiting to be written. Counted after decompression, so on a compressed topic this is larger than the same records on the broker.
queued.min.messages	100000	The number of records per partition the Backup tries to keep waiting to be written.
fetch.message.max.bytes	1048576	The maximum amount of data requested per partition from the EventHub in one go.
fetch.max.bytes	52428800	The maximum amount of data requested from the EventHub in one go, across all partitions.

See librdkafka’s configuration reference for the full description of each property.

To reduce memory, lower queued.max.messages.kbytes and fetch.message.max.bytes together. On topics with small records, lower queued.min.messages as well. Leave fetch.max.bytes unset unless you need it; librdkafka derives it from queued.max.messages.kbytes.

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup
spec:
  sink: "sink"
  source: "source"
  sourceAdditionalProps:
    queued.max.messages.kbytes: "32768"
    fetch.message.max.bytes: "1048576"

Compressed topics

Topics that use compression on the EventHub need more memory than their size on the broker suggests. Records are decompressed once they arrive, and the consumer properties above count the decompressed data. The better a topic compresses, the more memory the same settings need.

Size the pod for the decompressed data, not for the topic’s size on disk, and lower queued.max.messages.kbytes and fetch.message.max.bytes if the Backup uses more memory than expected.

As an illustration, this is what 64 MiB of data fetched from the broker becomes in the Backup, depending on the compression ratio of the topic:

Compression ratio of the topic	Held in the Backup
1:1	64 MiB
3:1	192 MiB
10:1	640 MiB
30:1	1.9 GiB

The ratio of a topic depends on its content and on how it is compressed: either by the topic’s compression.type on the broker, or, when that is left at producer, by the codec the producers use (compression.codec in librdkafka, compression.type in the Java client). Compare a topic’s size on the broker with the size of its records to find yours.

CPU

The compression settings of a Backup and the CPU request of its pod go together. Compression is set with spec.compression.algorithm and spec.compression.quality, and the CPU with spec.resources.requests.cpu:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup
spec:
  sink: "s3-storage"
  source: "production-kafka"
  compression:
    algorithm: zstd
    quality: 3
  resources:
    requests:
      cpu: "1"

Raise the CPU request when you raise the quality or pick one of the slower algorithms, and leave the CPU limit unset. See Compression for the algorithms and Pod resources for the other fields.

Throughput

EventHub

Raising queued.max.messages.kbytes and fetch.message.max.bytes lets a Backup read further ahead on fast topics, at the cost of the memory described above. Give the pod the memory to match.

Storage

The segment size and the compression decide how fast records are written and how many requests the Storage receives.

Segments

A Backup writes records to segment files and closes a segment when it reaches a size or an age.

Larger segments mean fewer files and a higher throughput on busy topics. On a remote Storage such as S3, where every request has a cost and a latency, this is the first setting to raise for a high-volume Backup:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: high-volume-backup
spec:
  sink: "s3-storage"
  source: "production-kafka"
  segmentRolloverTriggers:
    size: 256 MiB

Smaller segments are closed sooner. For slow topics, add a timeout so an open segment is closed after a while even if it never fills up:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: audit-backup
spec:
  sink: "s3-storage"
  source: "production-kafka"
  segmentRolloverTriggers:
    size: 25 MiB
    timeoutSeconds: 900

The timeout is evaluated as records arrive and while the topic is idle, so a segment closes on whichever trigger fires first. See the Segments page for the details.

Compression

Compressing segments reduces the amount of data written to the Storage, which on most Storages raises throughput and lowers cost, in exchange for CPU in the Backup pod.

zstd is a good default: it compresses text-like records several times over at little CPU cost.

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup
spec:
  sink: "s3-storage"
  source: "production-kafka"
  compression:
    algorithm: zstd
    quality: 3

A higher quality compresses smaller and slower. Keep it low when the Backup must keep up with a fast topic, and raise it when Storage size matters more than CPU. Compression applies to segments closed from then on; existing segments are not rewritten. See the Compression page for the other algorithms and a comparison.

Pod resources

A Backup pod runs two containers: the Backup itself and the monitoring sidecar. Each has its own resource requirements.

Set the memory request and limit of the Backup container to the same value. A limit below what the Backup needs is not a throttle: the container is restarted, and the Backup resumes from its last committed offsets. Leave its CPU limit unset, because compression and decompression are CPU bound and a CPU limit slows the Backup down without saving memory.

The sidecar needs far less than the Backup and is not affected by the consumer properties, so its resources can stay small and fixed:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup
spec:
  sink: "s3-storage"
  source: "production-kafka"
  resources:
    requests:
      cpu: "1"
      memory: "2Gi"
    limits:
      memory: "2Gi"
  monitoring:
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
      limits:
        memory: "256Mi"

To set the same resources for every Backup, use the operator’s default resource requirements. See Resource Requirements for the fields, and the Backup Requirements for a starting point.

Many topics

The topics of one Backup share its memory and CPU. To spread a large set of topics over several pods, use Worker Groups.

A good rule of thumb is 50 topics per worker. For a Backup of 150 topics:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup
spec:
  sink: "s3-storage"
  source: "production-kafka"
  workGroup:
    workers: 3

Each worker is its own pod with the resources set on the Backup, so the memory and CPU above apply per worker.

Monitoring sidecar

The monitoring sidecar is not affected by the consumer properties you set on the EventHub or the Backup. Its CPU and memory are set separately through spec.monitoring.resources.