SchemaRegistryRestore

A SchemaRegistryRestore is used for restoring schemas to a SchemaRegistry, from the data previously backed-up by a SchemaRegistryBackup in a Storage.

It is configured by creating a SchemaRegistryRestore resource in Kubernetes.

The Kannika Armory Operator creates a Job that runs the restore process, based on the configuration in the SchemaRegistryRestore resource.

Usage

A SchemaRegistryRestore can be managed using the kubectl command line tool, and are available by the name schemaregistryrestore or schemaregistryrestores.

$ kubectl get schemaregistryrestores
NAME                         STATUS
confluent-registry-restore   🌈 Ready

Restore Status

A SchemaRegistryRestore can have the following statuses:

• Initializing The Restore process is being created;
• Ready The Restore is configured but it has not been started yet;
• Running The Restore is running and restoring to the registry;
• Error The Restore has failed;
• Done The Restore is complete.

Additionally, a Restore resource exposes a JobCompleted Condition to report on the state of the underlying Job. This condition’s status can either be ‘True’ or ‘False’, and the reason field gives additional context:

• JobInitializing the job is being started;
• JobRunning the job is running;
• JobCompleted the job has completed successfully;
• JobFailed the job has failed.

Configuring a SchemaRegistryRestore

The following is an example of a SchemaRegistryRestore.

apiVersion: kannika.io/v1alpha
kind: SchemaRegistryRestore
metadata:
  name: confluent-registry-restore
spec:
  enabled: true
  registry: confluent-registry
  registryCredentialsFrom:
    credentialsRef:
      name: "http-basic-creds"
  storage: s3-storage
  storageCredentialsFrom:
    credentialsRef:
      name: "s3-storage-creds"
  extraEnvVars:
  - name: RUST_LOG
    value: "debug"

Full example with SchemaRegistry, Credentials, and Storage

# Secret - holds the registry's HTTP Basic credentials
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: http-basic-creds-secret
data:
  username: bXktdXNlcm5hbWU=  # base64 encoded
  password: bXktcGFzc3dvcmQ=  # base64 encoded
---
# Credentials - defines authentication to the schema registry
apiVersion: kannika.io/v1alpha
kind: Credentials
metadata:
  name: http-basic-creds
spec:
  httpBasic:
    usernameFrom:
      secretKeyRef:
        name: http-basic-creds-secret
        key: username
    passwordFrom:
      secretKeyRef:
        name: http-basic-creds-secret
        key: password
---
# SchemaRegistry - defines the connection to the target schema registry
apiVersion: kannika.io/v1alpha
kind: SchemaRegistry
metadata:
  name: confluent-registry
spec:
  confluent:
    url: https://registry.example.com
---
# Secret - holds the S3 access keys
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: aws-creds
data:
  accessKeyId: bXktYWNjZXNzLWtleQ==     # base64 encoded
  secretAccessKey: bXktc2VjcmV0LWtleQ==  # base64 encoded
---
# Credentials - defines authentication to the S3 bucket
apiVersion: kannika.io/v1alpha
kind: Credentials
metadata:
  name: s3-storage-creds
spec:
  aws:
    accessKeyIdFrom:
      secretKeyRef:
        name: aws-creds
        key: accessKeyId
    secretAccessKeyFrom:
      secretKeyRef:
        name: aws-creds
        key: secretAccessKey
---
# Storage - the source of the restore (where the backed-up schemas are stored)
apiVersion: kannika.io/v1alpha
kind: Storage
metadata:
  name: s3-storage
spec:
  s3:
    bucket: my-schemas-bucket
    region: eu-west-1
---
# SchemaRegistryRestore - restores schemas from the Storage to the registry
apiVersion: kannika.io/v1alpha
kind: SchemaRegistryRestore
metadata:
  name: confluent-registry-restore
spec:
  enabled: true
  registry: confluent-registry
  registryCredentialsFrom:
    credentialsRef:
      name: http-basic-creds
  storage: s3-storage
  storageCredentialsFrom:
    credentialsRef:
      name: s3-storage-creds

In this example:

• A SchemaRegistryRestore named confluent-registry-restore is created, as indicated by the .metadata.name field. This name will inherited by the Job running the restore process.

• The spec.enabled flag is set to true to allow restore process to start. If it is omitted or set to false, the restore process will be created in a paused state until the flag is switched to true.

• The SchemaRegistryRestore will pull data from the Storage called s3-storage as defined in the spec.storage field, and use the Credentials referenced by s3-storage-creds to authenticate to it.

• The SchemaRegistryRestore will connect to the confluent-registry SchemaRegistry to restore schemas, as indicated by the .spec.registry field, and use the Credentials referenced by http-basic-creds for authentication.

• The spec.extraEnvVars field is used to pass additional environment variables to the restore process. In this case, the RUST_LOG variable is set to debug to enable debug logging.

Note

It is likely that Restored schemas do not retain their original version after being restored. If your schemas have references to each other, a SchemaRegistryRestore will patch them with the new schema version of the referenced schema.

Import mode

Some registries such as the Confluent Cloud Schema Registry support an ‘import’ mode that will ensure that schemas will be restored with their original schema ID and version.

Caution

This will overwrite any existing schema with a conflicting schema ID or version on the target registry. Use with caution.

The restore validates the target registry’s mode before writing anything and never changes it. The target registry must be switched to IMPORT mode beforehand:

$ curl -X PUT \
    -H "Content-Type: application/json" \
    -d '{"mode":"IMPORT"}' \
    https://registry.example.com/mode

If the target registry’s import mode is enabled, a SchemaRegistryRestore may be configured to try and restore the original IDs and versions:

apiVersion: kannika.io/v1alpha
kind: SchemaRegistryRestore
metadata:
  name: confluent-registry-restore
spec:
  enabled: true
  importMode: true
  registry: confluent-registry
  registryCredentialsFrom:
    credentialsRef:
      name: "http-basic-creds"
  storage: s3-storage
  storageCredentialsFrom:
    credentialsRef:
      name: "s3-storage-creds"

Schema contexts

Backed-up subjects in a named schema context are restored under their fully qualified name, for example :.production:orders-value, into that same context on the target registry.

In import mode, the global IMPORT mode covers the default context only. Every named context being restored must be switched to IMPORT mode separately:

$ curl -X PUT \
    -H "Content-Type: application/json" \
    -d '{"mode":"IMPORT"}' \
    https://registry.example.com/mode/:.production:

The restore validates this before writing anything. If a named context is not in IMPORT mode, the restore fails up front with the JobFailed condition reason and the target registry is left untouched. The same applies to registries without IMPORT mode support.

Reinstating the original mode

A registry in IMPORT mode accepts writes that bypass the usual checks, so it should not be left in that state. After the restore completes, switch the registry back to its original mode:

$ curl -X PUT \
    -H "Content-Type: application/json" \
    -d '{"mode":"READWRITE"}' \
    https://registry.example.com/mode

Repeat this for every named context that was switched to IMPORT mode:

$ curl -X PUT \
    -H "Content-Type: application/json" \
    -d '{"mode":"READWRITE"}' \
    https://registry.example.com/mode/:.production:

Can I choose which schemas are restored?

Currently, a SchemaRegistryRestore will restore all data present in the Storage. In a future release, it will be possible to configure a subset of the data to restore.

Note

If you need to restore only a subset of the data, it is possible to delete the unwanted data from the Storage before starting the restore process, or to create a copy of the data with only the desired data in a new Storage.