Tuning

This page lists the settings you can change to fit a Restore to the topics it restores and to the target cluster.

They live in three places:

• Kafka producer properties, set on the EventHub or per Restore through sinkAdditionalProps.
• The Restore itself: which topics and partitions it restores, and how many at a time.
• The Backup it restores from: how its segments were written and compressed.

Producer settings

A Restore produces the records it reads from the Storage to the target cluster. The following producer properties decide how fast and how safely it does so. Armory sets its own values for some of them when you do not.

Property	librdkafka default	Effect
acks	all	How many replicas must have written a record before it counts as delivered. all is the safest; 1 is faster and risks losing records if the leader fails before replicating.
compression.codec	none	Compresses batches before they are sent, which lowers network use and broker disk at the cost of CPU in the Restore pod.
linger.ms	5	How long the producer waits to fill a batch before sending it. Longer waits mean larger batches and fewer requests.
batch.size	1000000	The maximum size of one batch in bytes.
batch.num.messages	10000	The maximum number of records in one batch.
queue.buffering.max.kbytes	1048576	The maximum amount of data, in kibibytes, waiting to be sent. Bounds the memory of the Restore.
queue.buffering.max.messages	100000	The maximum number of records waiting to be sent.

See librdkafka’s configuration reference for the full description of each property.

Keep acks at all unless you can afford to run the Restore again. To raise throughput, let batches grow with linger.ms and batch.size, and compress them with compression.codec if the network to the brokers is the bottleneck:

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore
spec:
  sink: "production-kafka"
  source: "s3-storage"
  enabled: true
  sinkAdditionalProps:
    acks: "all"
    compression.codec: "zstd"
    linger.ms: "50"
  config:
    topics:
      - source: orders
        target: orders

The same properties can be set once on the EventHub, where they apply to every Restore that targets it:

apiVersion: kannika.io/v1alpha
kind: EventHub
metadata:
  name: production-kafka
spec:
  kafka:
    properties:
      bootstrap.servers: "broker1:9092,broker2:9092"
      acks: "all"
      compression.codec: "zstd"
      linger.ms: "50"

A value in sinkAdditionalProps on a Restore takes precedence over the EventHub’s.

To reduce memory, lower queue.buffering.max.kbytes.

Note

A future release adds an option to honour the target topic’s compression settings, so a Restore produces with the codec the topic is configured for without setting compression.codec yourself.

Memory

The memory of a Restore is set by how much it buffers towards the target cluster and by how many topics it restores at once. queue.buffering.max.kbytes applies to each producer, so the pod needs that amount for every producer maxProducers allows, on top of what each topic in flight under parallelism holds while it is read from the Storage.

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore
spec:
  sink: "production-kafka"
  source: "s3-storage"
  enabled: true
  sinkAdditionalProps:
    queue.buffering.max.kbytes: "262144"
  config:
    parallelism: 2
    maxProducers: 2
  resources:
    requests:
      memory: "2Gi"
    limits:
      memory: "2Gi"

CPU

A Restore spends CPU on reading the Backup’s segments, on compression.codec if it is set, and on TLS, all multiplied by parallelism. The CPU request of the pod goes with those three settings:

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore
spec:
  sink: "production-kafka"
  source: "s3-storage"
  enabled: true
  sinkAdditionalProps:
    compression.codec: "zstd"
  config:
    parallelism: 4
  resources:
    requests:
      cpu: "2"

See Pod resources for the other fields.

Parallelism

Within one Restore

By default a Restore restores one topic at a time with one producer. parallelism sets how many topics are restored at once, and maxProducers how many producer clients the Restore may keep open for them, each with its own buffer and its own connections to every broker.

Raise parallelism when a Restore covers many topics, or a few large ones. Raise maxProducers with it when the target topics need different producer settings, for instance a different message.max.bytes; otherwise the topics share one producer.

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore
spec:
  sink: "production-kafka"
  source: "s3-storage"
  enabled: true
  config:
    parallelism: 4
    maxProducers: 2

Every topic restored at once adds its own read buffers and producer queue, so give the pod the memory and CPU to match. See Restore to multiple topics in parallel for the details.

Several Restores

A Restore is one pod. To use more than one node, or to restore independent sets of topics on their own schedule, split the topics over several Restore resources. Each Restore has its own resources, its own parallelism and its own report.

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore-orders
spec:
  sink: "production-kafka"
  source: "s3-storage"
  enabled: true
  config:
    topics:
      - source: orders
        target: orders
---
apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore-payments
spec:
  sink: "production-kafka"
  source: "s3-storage"
  enabled: true
  config:
    topics:
      - source: payments
        target: payments

One topic over several Restores

A single large topic can be split the same way, by giving each Restore a subset of its partitions.

Caution

This causes read amplification. A Backup’s segments hold records of every partition of a topic, so each of these Restores reads the whole topic from the Storage and keeps only its partitions. Splitting a topic over N Restores reads it N times. Use it when producing to the target cluster is the bottleneck, not reading from the Storage, and keep N small.

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore-orders-a
spec:
  sink: "production-kafka"
  source: "s3-storage"
  enabled: true
  config:
    topics:
      - source: orders
        target: orders
        partitions:
          - number: 0
          - number: 1
          - number: 2
---
apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore-orders-b
spec:
  sink: "production-kafka"
  source: "s3-storage"
  enabled: true
  config:
    topics:
      - source: orders
        target: orders
        partitions:
          - number: 3
          - number: 4
          - number: 5

Backup segments and compression

A Restore reads the segments the Backup wrote, so two settings of the Backup decide part of a Restore’s speed.

The segment size decides how many objects a Restore fetches from the Storage. Larger segments mean fewer, larger requests, which suits remote Storages.

The compression algorithm of the Backup works both ways on a Restore. A compressed segment holds several times more records per byte, so every request to the Storage returns more records and the Restore needs fewer requests and less bandwidth per record. In exchange the Restore spends CPU decompressing. Algorithms such as zstd decompress fast; xz and lzma compress smaller but are several times slower to read back. Choose the Backup’s compression with the Restore in mind when restore time matters.

Pod resources

Set the memory request and limit of the Restore pod to the same value, and leave the CPU limit unset. Memory grows with parallelism and queue.buffering.max.kbytes; CPU grows with parallelism and with compression on either side.

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore
spec:
  sink: "production-kafka"
  source: "s3-storage"
  enabled: true
  resources:
    requests:
      cpu: "1"
      memory: "2Gi"
    limits:
      memory: "2Gi"
  config:
    parallelism: 4

See Resource Requirements for the fields, and the Restore Requirements for a starting point.