SchemaRegistryBackup

A SchemaRegistryBackup is used for backing up schemas from a SchemaRegistry, and offloading them to a Storage. It is configured by creating a SchemaRegistryBackup resource in Kubernetes.

The Kannika Armory Operator creates a Pod that runs the backup process, based on the configuration in the SchemaRegistryBackup resource.

Usage

SchemaRegistryBackups can be managed using the kubectl command line tool, and are available by the name schemaregistrybackup or schemaregistrybackups.

$ kubectl get schemaregistrybackups
NAME                        STATUS
confluent-registry-backup   Running

Backup Status

A SchemaRegistryBackup can have the following statuses:

• Paused The Backup is configured but it has not been started yet or it has been paused;
• Initializing The Backup process is being created;
• Running The Backup is running and backing up data to the storage;
• Error The Backup has failed, and no data is being backed up.

Additionally, a SchemaRegistryBackup resource exposes a DeploymentReady Condition to report on the state of the underlying Deployment. This condition’s status can either be ‘True’ or ‘False’, indicating whether this Backup’s deployment is healthy. The condition’s reason gives additional context and may be one of the following:

• DeploymentReady the deployment is healthy;
• DeploymentError the deployment encountered an error: check the status of this deployment to find out what happened;
• DeploymentDeleted somebody or something deleted the deployment and this should be a transient state;
• DeploymentStateUnknown the deployment state is unknown, perhaps due to an error talking with the kubernetes API.

Configuring a SchemaRegistryBackup

The following is an example of a SchemaRegistryBackup.

apiVersion: kannika.io/v1alpha
kind: SchemaRegistryBackup
metadata:
  name: confluent-registry-backup
spec:
  enabled: true
  registry: confluent-registry
  registryCredentialsFrom:
    credentialsRef:
      name: confluent-registry-creds
  storage: s3-storage
  storageCredentialsFrom:
    credentialsRef:
      name: s3-storage-creds
  # An optional explicit interval between backups in seconds
  backupIntervalSecs: 15
  extraEnvVars:
    - name: RUST_LOG
      value: debug

Full example with SchemaRegistry, Credentials, and Storage

# Secret - holds the registry's HTTP Basic credentials
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: confluent-registry-secret
data:
  username: bXktdXNlcm5hbWU=  # base64 encoded
  password: bXktcGFzc3dvcmQ=  # base64 encoded
---
# Credentials - defines authentication to the schema registry
apiVersion: kannika.io/v1alpha
kind: Credentials
metadata:
  name: confluent-registry-creds
spec:
  httpBasic:
    usernameFrom:
      secretKeyRef:
        name: confluent-registry-secret
        key: username
    passwordFrom:
      secretKeyRef:
        name: confluent-registry-secret
        key: password
---
# SchemaRegistry - defines the connection to the schema registry
apiVersion: kannika.io/v1alpha
kind: SchemaRegistry
metadata:
  name: confluent-registry
spec:
  confluent:
    url: https://registry.example.com
---
# Secret - holds the S3 access keys
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: aws-creds
data:
  accessKeyId: bXktYWNjZXNzLWtleQ==     # base64 encoded
  secretAccessKey: bXktc2VjcmV0LWtleQ==  # base64 encoded
---
# Credentials - defines authentication to the S3 bucket
apiVersion: kannika.io/v1alpha
kind: Credentials
metadata:
  name: s3-storage-creds
spec:
  aws:
    accessKeyIdFrom:
      secretKeyRef:
        name: aws-creds
        key: accessKeyId
    secretAccessKeyFrom:
      secretKeyRef:
        name: aws-creds
        key: secretAccessKey
---
# Storage - defines where the schemas are stored
apiVersion: kannika.io/v1alpha
kind: Storage
metadata:
  name: s3-storage
spec:
  s3:
    bucket: my-schemas-bucket
    region: eu-west-1
---
# SchemaRegistryBackup - backs up schemas from the registry to the Storage
apiVersion: kannika.io/v1alpha
kind: SchemaRegistryBackup
metadata:
  name: confluent-registry-backup
spec:
  enabled: true
  registry: confluent-registry
  registryCredentialsFrom:
    credentialsRef:
      name: confluent-registry-creds
  storage: s3-storage
  storageCredentialsFrom:
    credentialsRef:
      name: s3-storage-creds

In this example:

• A SchemaRegistryBackup named confluent-registry-backup is created, as indicated by the .metadata.name field. This name will inherited by the Pod running the backup process.

• The backup is enabled by setting the spec.enabled to true. Unless this flag is set, the backup container will not be started.

• The SchemaRegistryBackup will connect to the confluent-registry SchemaRegistry to fetch schemas, as indicated by the .spec.registry field, and use the Credentials referenced by confluent-registry-creds for authentication.

• The SchemaRegistryBackup will write data to the s3-storage Storage defined in the spec.storage field, and use the Credentials referenced by s3-storage-creds to authenticate to it.

• The interval between API fetches and synchronization with the remote storage is set to 15 seconds by the spec.backupIntervalSecs field. If not specified, backups will occur every 3600 seconds.

• The spec.extraEnv field is used to pass additional environment variables to the backup container. In this case, the RUST_LOG variable is set to debug to enable debug logging from the backup process.

What is backed up

A SchemaRegistryBackup backs up all subjects and all their schema versions. No additional configuration is required.

Subjects are backed up from all schema contexts of the schema registry. Subjects in a named context are backed up under their fully qualified name, for example :.production:orders-value. Subjects in the default context keep their plain name. Schema registries without context support are not affected.

Soft-deleted subjects and soft-deleted schema versions are included in the backup. Permanently deleted subjects cannot be recovered from the registry and are not included.

See Schema Registry Backups for the layout of the backed-up schemas in the storage.