Announcing the 0.9.0 Release

This new release introduces initial schema registry support, WASI plugins, and an improved console.

Installation

For new installations, see the Installation guide.

For upgrading existing installations, see the associated Upgrading to 0.9.x guide.

A backup for your schema registry

New SchemaRegistry and SchemaRegistryBackup resources allow you to back up Confluent Cloud schema registries to any storage that Armory supports, including S3, Google Cloud Storage and Azure Blob Storage.

Schema registry backups are a great way to ensure that your schemas are safe and can be restored in case of data loss.

apiVersion: kannika.io/v1alpha
kind: SchemaRegistry
metadata:
  name: confluent-registry
spec:
  confluent:
    url: https://registry-1234.confluent.cloud

apiVersion: kannika.io/v1alpha
kind: SchemaRegistryBackup
metadata:
  name: confluent-registry-backup
spec:
  registry: confluent-registry
  registryCredentialsFrom:
    credentialsRef:
      name: confluent-registry-creds
  storage: s3-storage
  # An optional explicit interval between backups in seconds
  backupIntervalSecs: 15

A new HTTP Basic type has been added to Credentials to authenticate with the Confluent Schema Registry REST API:

apiVersion: kannika.io/v1alpha
kind: Credentials
metadata:
  name: confluent-registry-creds
spec:
  httpBasic:
    usernameFrom:
      secretKeyRef:
        name: confluent-registry-creds
        key: username
    passwordFrom:
      secretKeyRef:
        name: confluent-registry-creds
        key: password
---
apiVersion: v1
kind: Secret
metadata:
  name: confluent-registry-creds
type: Opaque
data:
  username: <username base64>
  password: <password base64>

Restoring schema registries will be added in the next release. We will also be integrating schema mapping into the platform, which will replace the current manual schema mapping process.

Schema Registry Learn how to configure schema registries

Schema Registry Backup Learn how to configure schema registry backups

HTTP Basic Learn how to configure HTTP Basic authentication

Segments instead of partitions

From this release onwards, we will be referring to partitions as segments in the context of backups and restores.

This change is to avoid confusion with Kafka partitions.

A new field segmentRolloverTriggers has been added to the Backup resource to accommodate this change. The partitionRolloverTriggers field is still supported, but has been deprecated and will be removed in a future release.

Custom functionality with WASI plugins

We have added support for WASI plugins in Armory. This will allow you to write custom plugins in any language with support for the WebAssembly Component Model, and run them in the Armory platform.

WASI plugins are a great way to extend the functionality of Armory, and we are excited to see what the community will build with this new feature.

WASI Plugins Learn how to configure WASI plugins

Enabling plugins for a subset of topics

You can now enable plugins for a subset of topics by using the topics field in the plugin configuration.

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: backup-with-plugin
spec:
  source: "source"
  sink: "sink"
  plugins:
    - name: some-plugin
      spec: # ...
      topicSelectors:
        - name:
            glob: 'flights.*' # Apply plugin to all topics starting with 'flights.'

Caution

This is a breaking change. All plugins have had their specifications updated.

Enable plugins for a subset of topics Learn how to enable plugins for a subset of topics

Improved restore topic configuration

We have changed the way you configure topics on a Restore from a map-based approach to a list-based approach.

Deprecated way using the .spec.config.mapping map:

apiVersion: kannika.io/v1alpha
kind: Restore
spec:
  source: source
  sink: sink
  config:
    mapping:
      source.topic:
        target: target.topic

Using the new .spec.config.topics list:

apiVersion: kannika.io/v1alpha
kind: Restore
spec:
  source: source
  sink: sink
  config:
    topics:
      - source: source.topic
        target: target.topic

This adheres more closely to the Kubernetes API conventions , which prefers lists to maps.

In the new configuration, the target topic is the key in the new configuration, unlike the old configuration where the source topic was the key. You can not add the same target topic twice in the new configuration.

This makes a lot more sense, as you probably don’t want to restore different source topics to the same target topic. However, you can restore data from the same source topic to multiple target topics now.

The old .spec.config.mapping configuration is still supported, but will be removed in the next release. The API will automatically convert the old configuration to the new configuration when you update a Restore. The new configuration is recommended for all new Restores.

Improved Topics tab on the Restore page

The Topics tab on the Restore page has been improved to allow more fine-grained control over topics.

Inline editing of target topics

We added the ability to edit the target of a topic directly in the console, by clicking on the target topic name. This should greatly improve the user experience when configuring a restore. The changes are saved automatically.

Inline edit the target of a topic

Configure topics using YAML

For more advanced use-cases, or if you prefer to use your own editor, you can now configure topics using a YAML editor by clicking on the Source button (”<>”) in the top right corner of the Topics tab.

The YAML editor validates the input and provides auto-completion for the configuration.

Configure topics using YAML

Auto-completion is available

New Add Topic dialog

We have removed the “Configure topics” screen and replaced it with a new “Add topics” dialog.

Map multiple topics at once

You can now map multiple topics at once by selecting multiple topics, and clicking the “Topic Mapping” button.

Map multiple topics at once

Missing topics warning

The console will now display a warning if a source topic is missing from a Backup, when configuring a Restore.

Topic does not exist!

Follow the progress of a Restore in the new Progress tab

A new Progress tab has been added to the restore page in the console. This tab will show you the progress of the restore, including the number of messages restored and the number of topics restored.

Follow the progress of a Restore

Updating backup and restore settings in the console

New Settings tabs have been added to the backup and restore pages in the console.

Update the settings of a Backup

Update the settings of a Restore

View the status of each topic

The Topics tabs in the console now show the status of each topic, indicated by a colored line on the left side of the topic.

Easily view the status of each topic

Mutual exclusion on Storage

Any Armory process that writes to a storage location will now acquire a lease on the storage location, and release it when the operation is complete. This works on a per-topic basis.

If multiple backup processes were running at the same time on the same storage, it would be possible to create unnecessary, dangling segments. By adding mutual exclusion on storage, this issue is resolved.

Depending on the storage type, you may notice new files alongside the usual .kan files called __lease or __kannika_time which are used to implement the leasing mechanism, inside each topic’s directory.

Other changes

• The explicit lifecycle management of backup and restore pods has been removed in favour of the new leasing mechanism.

• The console now contains versioned links to the documentation, instead of the latest link, to ensure that the documentation matches the version of the console.

Bug fixes

• Fixed possible division by 0 in backup metrics

• Metrics of paused topics are now included in the backup metrics

• Fix parallelism and max producers options not being shown in restore overview

• Fix description not being saved when creating a backup via the API

Breaking changes

• Plugins must now be configured using topic selectors. All plugins have had their specification updated, as the topic configuration has been extracted out of the specification. See the Enable plugins for a subset of topics section for more information.

Upcoming features

Some features that we are working on include:

• Schema registry restore support: We are working on adding support for restoring schema registries.
• Integrated schema mapping in the console: We are going to be adding schema mapping to the console, which will work on top of the current Kubernetes-based schema mapping process.
• Topic discovery and automatic importing: We are working on a feature that will allow you to discover topics in your Kafka cluster, and automatically add them to your backups using regex and glob patterns.
• Improved resource management: We will continue to improve the resource management capabilities of the Armory console.

Please check the Roadmap for more information on upcoming features.

Release Notes

For a full list of changes, see the Changelog.