Schema Mapping

This page describes how to configure schema mapping in a Restore.

Schema Mapping

Schema mapping allows migration of data from one environment to another that use schema registries.

With schema mapping, you can, for example, copy data from a production environment to a QA environment where there are different schema registries.

The restore process will map the schema IDs from the source environment to the schema IDs of the target environment. It will do so by looking up the schema mapping in a lookup table and replacing the magic byte that represents the schema ID in the message when restoring the data.

To use schema mapping, you need to:

• Define or generate a schema mapping*
• Create a ConfigMap with the schema mapping
• Reference the ConfigMap in the Restore for schema mapping

The operator will automatically configure both the Payload Schema Mapping plugin, and the Key Schema Mapping plugin with the schema mapping.

Tip

*We are working on automatically mapping the schemas, so no manual schema mapping is needed.

We already have a tool that can automatically map schemas. If you want to give it a try, give us a shout at sales@kannika.io or get in touch with us on Slack.

First, create a YAML file with a mapping field that contains pairs of schema IDs.

mapping.yaml

mapping:
  # Map from schema 10001 to schema 10002
  10001: 10002

Then, create a ConfigMap with the schema mapping:

$ kubectl create configmap mapping --from-file=schema-mapping=mapping.yaml

This will create a ConfigMap with the following content:

schema-mapping-config-map.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: mapping
data:
  schema-mapping: |
    mapping:
      # Map from schema 10001 to schema 10002
      10001: 10002

Finally, set the schema mapping in the Restore by setting the .spec.config.schemaMappingFrom field to load the mapping from the field in the ConfigMap, using a ConfigMapKeySelector.

apiVersion: kannika.io/v1alpha
kind: Restore
metadata:
  name: restore
spec:
  sink: "sink"
  source: "source"
  config:
    schemaMappingFrom:
      configMapKeyRef:
        name: mapping # Name of the ConfigMap
        key: schema-mapping # Name of the field in the ConfigMap

The operator will validate the schema mapping. To check if the schema mapping is valid, check the SchemaMappingValidated condition in the status of the Restore resource.

$ kubectl get restore [NAME] -o jsonpath='{.status.conditions[?(@.type=="SchemaMappingValidated")]}'