Topic Repartitioning Plugin

The topic-repartitioning plugin is used to repartition a topic.

This plugin is useful when:

• you want to change the partitioning scheme of a topic from one scheme to another.
• you want to increase or decrease the number of partitions in a topic.

Synopsys

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore-with-plugin
spec:
  source: "source"
  sink: "sink"
  config:
    plugins:
      - name: topic-repartitioning
        topicSelectors:
          - name:
              regex: "^topic-foo-restore$"
        spec:
          scheme: keyHash
          algorithm: murmur2
          partitions: 10
      - name: topic-repartitioning
        topicSelectors:
          - name:
              regex: "^topic-bar-restore$"
        spec:
          scheme: automatic
      - name: topic-repartitioning
        topicSelectors:
          - name:
              glob: "topic-baz-*"
        spec:
          scheme: headerHash
          name: "myheader"
          algorithm: crc32
          partitions: 3
    topics:
      - target: topic-foo-restore
        source: topic-foo
      - target: topic-bar-restore
        source: topic-bar
      - target: topic-baz-restore
        source: topic-baz

Configuring the partitioning scheme

The topic-repartitioning plugin supports the following partitioning schemes:

• Automatic
• Hash-based partitioning
• Random partitioning
• Round-robin based partitioning

Automatic repartitioning

In this mode, Armory will simply strip the original partition and offset information from the record and it will therefore delegate the partitioning strategy to the event broker.

Hash-based partitioning

This is the default partitioning scheme in Kafka. A hash of the key (or a header’s value) is used to determine the partition.

The following algorithms are currently supported:

• murmur2
• fnv1a
• crc32

When a key is present in the message, the key is used to determine the partition. If the key is not present, Armory will fall-back to the automatic scheme. An empty key is considered to be present, therefore all messages with an empty key will be assigned the same partition.

plugins:
  - name: topic-repartitioning
    spec:
      scheme: keyHash
      algorithm: murmur2
      partitions: 10

The algorithm field is optional and defaults to murmur2.

The partition is calculated as follows:

partition = hash(key) % partitions

A header’s value can also used instead of the key:

plugins:
  - name: topic-repartitioning
    spec:
      scheme: headerHash
      name: "myHeader"
      algorithm: crc32
      partitions: 10

Random partitioning

Messages are distributed to partitions randomly.

plugins:
  - name: topic-repartitioning
    spec:
      scheme: random
      partitions: 10

Round-robin based partitioning

Messages are distributed to partitions in a round-robin fashion. This is useful when you want to distribute messages evenly across partitions.

plugins:
  - name: topic-repartitioning
    spec:
      scheme: roundRobin
      partitions: 10