Skip to content
Kannika

    Topic Repartitioning Plugin

    The topic-repartitioning plugin is used to repartition a topic.

    This plugin is useful when:

    • you want to change the partitioning scheme of a topic from one scheme to another.
    • you want to increase or decrease the number of partitions in a topic.

    Synopsys

    apiVersion: kannika.io/v1alpha
    kind: Restore
    metadata:
      name: restore-with-plugin
    spec:
      source: "source"
      sink: "sink"
      config:
        plugins:
          - name: topic-repartitioning
            topicSelectors:
              - name:
                  regex: "^topic-foo-restore$"
            spec:
              scheme: keyHash
              algorithm: murmur2
              partitions: 10
          - name: topic-repartitioning
            topicSelectors:
              - name:
                  regex: "^topic-bar-restore$"
            spec:
              scheme: automatic
          - name: topic-repartitioning
            topicSelectors:
              - name:
                  glob: "topic-baz-*"
            spec:
              scheme: headerHash
              name: "myheader"
              algorithm: crc32
              partitions: 3
        topics:
          - target: topic-foo-restore
            source: topic-foo
          - target: topic-bar-restore
            source: topic-bar
          - target: topic-baz-restore
            source: topic-baz

    Configuring the partitioning scheme

    The topic-repartitioning plugin supports the following partitioning schemes:

    • Automatic
    • Hash-based partitioning
    • Random partitioning
    • Round-robin based partitioning

    Automatic repartitioning

    In this mode, Armory will simply strip the original partition and offset information from the record and it will therefore delegate the partitioning strategy to the event broker.

    Hash-based partitioning

    This is the default partitioning scheme in Kafka. A hash of the key (or a header’s value) is used to determine the partition.

    The following algorithms are currently supported:

    • murmur2
    • fnv1a
    • crc32

    When a key is present in the message, the key is used to determine the partition. If the key is not present, Armory will fall-back to the automatic scheme. An empty key is considered to be present, therefore all messages with an empty key will be assigned the same partition.

    plugins:
      - name: topic-repartitioning
        spec:
          scheme: keyHash
          algorithm: murmur2
          partitions: 10

    The algorithm field is optional and defaults to murmur2.

    The partition is calculated as follows:

    partition = hash(key) % partitions

    A header’s value can also used instead of the key:

    plugins:
      - name: topic-repartitioning
        spec:
          scheme: headerHash
          name: "myHeader"
          algorithm: crc32
          partitions: 10

    Random partitioning

    Messages are distributed to partitions randomly.

    plugins:
      - name: topic-repartitioning
        spec:
          scheme: random
          partitions: 10

    Round-robin based partitioning

    Messages are distributed to partitions in a round-robin fashion. This is useful when you want to distribute messages evenly across partitions.

    plugins:
      - name: topic-repartitioning
        spec:
          scheme: roundRobin
          partitions: 10