Overview

Plugins can be used to alter the behaviour of a Backup or a Restore. They allow the user to transform the data between the source and the sink, or to skip certain data. It is possible to configure multiple plugins on a Backup or a Restore in a specific order.

The following plugins are available:

• Key Schema Mapping Plugin
• Payload Schema Mapping Plugin
• Topic Repartitioning Plugin

Configuring a plugin on a Backup

Plugins can be configured on a Backup with the .spec.plugins field. The ‘spec’ field contains the plugin’s configuration and its content will vary from plugin to plugin.

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup-with-plugin
spec:
  source: "source"
  sink: "sink"
  plugins:
    - name: some-plugin
      spec: # ...
  streams:
    - topic: "topic-foo"
    - topic: "topic-baz"

Configuring a plugin on a Restore

Plugins can be configured on a Restore with the .spec.config.plugins field: The ‘spec’ field contains the plugin’s configuration and its content will vary from plugin to plugin.

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore-with-plugin
spec:
  source: "source"
  sink: "sink"
  config:
    plugins:
      - name: some-plugin
        spec: # ...
    topics:
      - target: target-topic-name
        source: source-topic-name

Disabling pre-flight checks

A restore job will inspect the target topic to make sure it has enough partitions for the restore to succeed. However some plugins such as the topic-repartitioning plugin will change the partitioning of the restored records. For such cases, the disablePreflightChecks flag can be used to bypass this kind of verifications performed before the restore can start.

Example:

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore-with-plugin
spec:
  sink: "sink"
  source: "source"
  config:
    topics:
      - target: target-topic-name
        source: source-topic-name
        disablePreflightChecks: true    # New

Enabling plugins for a subset of topics

By default, plugins will be enabled for all topics listed in a Backup or a Restore. You can restrict which topic should be affected by a plugin using the topicSelectors attribute when configuring a plugin:

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore-with-plugin
spec:
  source: "source"
  sink: "sink"
  config:
    plugins:
      - name: some-plugin
        spec: # ...
        topicSelectors:
          - name:
              literal: 'products.uk.v2'
          - name:
              regex: '^orders\.'
          - name:
              glob: 'invoices*'
    topics:
      - target: products.uk.v2
        source: products.uk
      - target: invoices.uk.v2
        source: invoices.uk
      - target: orders.uk.v2
        source: orders.uk
      - target: customers.uk.v2
        source: customers.uk

In the example above, the some-plugin plugin will only be enabled for topics that match one of the selectors: products.uk.v2, orders.uk.v2 and invoices.uk.v2 in this case; the some-plugin plugin won’t be enabled for customers.uk.v2.

The literal selector rule performs an exact match on a topic name. glob and regex rules will match a topic’s name against a pattern.

The accepted syntax for regular expressions is documented here , and the one for glob patterns can be found here .

Note

Notice that in the case of a Restore, the selector is tested against the target topic, not the source topic.

The ambiguity doesn’t exist for Backups as there’s only a ‘source’ topic.

You can also use selectors to use different instances of the same plugin with different configurations. However, a particular plugin can only be used once for a given topic. This means that, if multiple instances of a plugin match a single topic, only one will be used (typically, the last one defined in spec.config.plugins).

Plugin Specification

Each plugin instance can be configured using the spec field. This field takes an object that is specific to the plugin. It can contain key-value pairs, nested objects and arrays, and so on.

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup-with-a-lot-of-plugin-configuration
spec:
  source: "source"
  sink: "sink"
  plugins:
    - name: my-plugin
      spec:
        key: "value" # string
        bool: true # boolean
        numbers: 123 # number
        nested: # object
          key: "value"
          boolean: true
          array_of_strings: ["value1", "value2"] # array of strings
          array_of_objects: # array of objects
          - name: object1
            property: property1
          - name: object2
            property: property2

Danger

When initialized, plugins used in Backups will receive the name of the source topic as an argument, however, plugins used in Restores will receive the name of the target topic. So make sure you don’t mix them up.

Writing custom plugins

You can write your own plugins. See the was-plugin section of this guide.