Skip to content

    Topic Repartitioning Plugin

    The topic-repartitioning plugin is used to repartition a topic.

    This plugin is useful when:

    • you want to change the partitioning scheme of a topic from one scheme to another.
    • you want to increase or decrease the number of partitions in a topic.

    Synopsys

    apiVersion: kannika.io/v1alpha
    kind: Restore
    metadata:
    name: restore-with-plugin
    spec:
    source: "source"
    sink: "sink"
    config:
    plugins:
    - name: topic-repartitioning
    spec:
    topic-foo-restore:
    scheme: keyHash
    algorithm: murmur2
    partitions: 10
    topic-bar-restore:
    scheme: automatic
    topic-baz-restore:
    scheme: headerHash
    name: "myheader"
    algorithm: crc32
    partitions: 3
    mapping:
    topic-foo:
    target: topic-foo-restore
    disablePreflightChecks: true # Needed when the target topic has different partitions than the source topic
    topic-bar:
    target: topic-bar-restore
    topic-baz:
    target: topic-baz-restore
    disablePreflightChecks: true # Needed when the target topic has different partitions than the source topic

    Configuring the partitioning scheme

    The topic-repartitioning plugin supports the following partitioning schemes:

    • Automatic
    • Hash-based partitioning
    • Random partitioning
    • Round-robin based partitioning

    Automatic repartitioning

    In this mode, Armory will simply strip the original partition and offset information from the record and it will therefore delegate the partitioning strategy to the event broker.

    Hash-based partitioning

    This is the default partitioning scheme in Kafka. A hash of the key (or a header’s value) is used to determine the partition.

    The following algorithms are currently supported:

    • murmur2
    • fnv1a
    • crc32

    When a key is present in the message, the key is used to determine the partition. If the key is not present, Armory will fall-back to the automatic scheme. An empty key is considered to be present, therefore all messages with an empty key will be assigned the same partition.

    plugins:
    - name: topic-repartitioning
    spec:
    target-topic-foo:
    scheme: keyHash
    algorithm: murmur2
    partitions: 10

    The algorithm field is optional and defaults to murmur2.

    The partition is calculated as follows:

    partition = hash(key) % partitions

    A header’s value can also used instead of the key:

    plugins:
    - name: topic-repartitioning
    spec:
    target-topic-foo:
    scheme: headerHash
    name: "myHeader"
    algorithm: crc32
    partitions: 10

    Random partitioning

    Messages are distributed to partitions randomly.

    plugins:
    - name: topic-repartitioning
    spec:
    target-topic-baz:
    scheme: random
    partitions: 10

    Round-robin based partitioning

    Messages are distributed to partitions in a round-robin fashion. This is useful when you want to distribute messages evenly across partitions.

    plugins:
    - name: topic-repartitioning
    spec:
    target-topic-foo:
    scheme: roundRobin
    partitions: 10