Overview

Plugins can be used to alter the behaviour of a Backup or a Restore. They allow the user to transform the data between the source and the sink, or to skip certain data. It is possible to configure multiple plugins on a Backup or a Restore in a specific order.

The following plugins are available:

• Key Schema Mapping Plugin
• Payload Schema Mapping Plugin
• Topic Repartitioning Plugin

Configuring a plugin on a Backup

Plugins can be configured on a Backup with the .spec.plugins field:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup-with-plugin
spec:
  source: "source"
  sink: "sink"
  plugins:
    - name: some-plugin
      spec: # ...
  streams:
    - topic: "topic-foo"
    - topic: "topic-baz"

Configuring a plugin on a Restore

Plugins can be configured on a Restore with the .spec.plugins field:

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore-with-plugin
spec:
  source: "source"
  sink: "sink"
  config:
    plugins:
      - name: some-plugin
        spec: # ...
    mapping:
      source-topic-name:
        target: target-topic-name

Disabling pre-flight checks

A restore job will inspect the target topic to make sure it has enough partitions for the restore to succeed. However some plugins such as the topic-repartitioning plugin will change the partitioning of the restored records. For such cases, the disablePreflightChecks flag can be used to bypass this kind of verifications performed before the restore can start.

Example:

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore-with-plugin
spec:
  sink: "sink"
  source: "source"
  config:
    mapping:
      source-topic-name:
        disablePreflightChecks: true    # New
        target: target-topic-name

Plugin Specification

Each plugin instance can be configured using the spec field. This field takes an object that is specific to the plugin. It can contain key-value pairs, nested objects and arrays, and so on.

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup-with-a-lot-of-plugin-configuration
spec:
  source: "source"
  sink: "sink"
  plugins:
    - name: my-plugin
      spec:
        key: "value" # string
        bool: true # boolean
        numbers: 123 # number
        nested: # object
          key: "value"
          boolean: true
          array_of_strings: ["value1", "value2"] # array of strings
          array_of_objects: # array of objects
          - name: object1
            property: property1
          - name: object2
            property: property2

Note

Some plugins can have a different configuration for each topic.

For example:

  plugins:
    - name: my-plugin
      spec:
        topic-foo:
          # Config for topic "topic-foo"
        topic-bar:
          # Config for topic "topic-bar"

Danger

When initialized, plugins used in Backups will receive the name of the source topic as an argument, however, plugins used in Restores will receive the name of the target topic. So make sure you don’t mix them up.

Writing custom plugins

The plugins are currently statically linked into the engine during the compilation process for performance reasons. This means that you cannot write your own plugins at the moment. Dynamic plugins are not yet supported.

In the future, the core engine will be open-sourced which will open up the possibility of compiling the core engine with custom plugins statically linked into it.