Topic Repartitioning Plugin

The topic-repartitioning plugin is used to repartition a topic.

This plugin is useful when:

• you want to change the partitioning scheme of a topic from one scheme to another.
• you want to increase or decrease the number of partitions in a topic.

Synopsys

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore-with-plugin
spec:
  source: "source"
  sink: "sink"
  config:
    plugins:
      - name: topic-repartitioning
        spec:
          topic-foo-restore:
            scheme: keyHash
            algorithm: murmur2
            partitions: 10
          topic-bar-restore:
            scheme: automatic
          topic-baz-restore:
            scheme: headerHash
            name: "myheader"
            algorithm: crc32
            partitions: 3
    mapping:
      topic-foo:
        target: topic-foo-restore
        disablePreflightChecks: true # Needed when the target topic has different partitions than the source topic
      topic-bar:
        target: topic-bar-restore
      topic-baz:
        target: topic-baz-restore
        disablePreflightChecks: true # Needed when the target topic has different partitions than the source topic

Configuring the partitioning scheme

The topic-repartitioning plugin supports the following partitioning schemes:

• Automatic
• Hash-based partitioning
• Random partitioning
• Round-robin based partitioning

Automatic repartitioning

In this mode, Armory will simply strip the original partition and offset information from the record and it will therefore delegate the partitioning strategy to the event broker.

Hash-based partitioning

This is the default partitioning scheme in Kafka. A hash of the key (or a header’s value) is used to determine the partition.

The following algorithms are currently supported:

• murmur2
• fnv1a
• crc32

When a key is present in the message, the key is used to determine the partition. If the key is not present, Armory will fall-back to the automatic scheme. An empty key is considered to be present, therefore all messages with an empty key will be assigned the same partition.

plugins:
  - name: topic-repartitioning
    spec:
      target-topic-foo:
        scheme: keyHash
        algorithm: murmur2
        partitions: 10

The algorithm field is optional and defaults to murmur2.

The partition is calculated as follows:

partition = hash(key) % partitions

A header’s value can also used instead of the key:

plugins:
  - name: topic-repartitioning
    spec:
      target-topic-foo:
        scheme: headerHash
        name: "myHeader"
        algorithm: crc32
        partitions: 10

Random partitioning

Messages are distributed to partitions randomly.

plugins:
  - name: topic-repartitioning
    spec:
      target-topic-baz:
        scheme: random
        partitions: 10

Round-robin based partitioning

Messages are distributed to partitions in a round-robin fashion. This is useful when you want to distribute messages evenly across partitions.

plugins:
  - name: topic-repartitioning
    spec:
      target-topic-foo:
        scheme: roundRobin
        partitions: 10