Skip to content
Kannika

    Announcing the 0.6.0 Release

    This new release brings Storage Drivers to Kannika Armory, giving you the ability to store your backups in S3, Azure Blob Storage and Google Cloud Storage in a robust and efficient manner, and the ability to resume a restore from where it left off.

    Out with the Sidecars, in with the Storage Drivers

    Kannika Armory has introduced the concept of storage drivers into its core engine, allowing it to directly interact with different storage providers. This greatly simplifies the backup and restore process, and allows for greatly improved error handling and efficiency.

    In the 0.5 release, we introduced the concept of storage sidecars. These were separate containers that were responsible for the synchronisation of data to a storage provider. The sidecar would set up named pipes to allow the core process to read and write data to the storage provider.

    This approach had a number of limitations and issues, like a lack of backpressure and the need to write data to disk before uploading to the storage provider. We noticed for example that the S3 sidecar was not able to cope very well with network issues.

    Therefore, the sidecars have been removed and replaced with storage drivers. Using storage drivers is totally transparent to you, no additional configuration is required to use them. The only change required is to remove all sidecar configuration from your existing resources and installation configuration.

    This new approach has many advantages:

    • Backpressure. The backup process will only consume data as fast as it can be uploaded to the storage provider.
    • Data is streamed directly. The backup process will no longer write the data to disk before uploading to the storage provider, removing the risk of running out of disk space.
    • Simplified configuration. No additional configuration is required to use storage drivers. All sidecar related configuration has been removed.
    • Improved error handling and resiliency. The backup and restore processes will now be able to handle errors more gracefully, and will be able to retry failed tasks more effectively.
    • Improved performance. The backup and restore processes will be more efficient, and will be able to use resources more effectively.
    • Improved resource usage. The backup and restore processes will use a lot less resources, and will be able to run on smaller machines. Compression has had its performance greatly improved.

    We are very happy with the improvements that storage drivers bring to Kannika Armory. We believe that this will greatly improve the user experience, and make Kannika Armory a more robust and efficient backup solution.

    Azure Blob Storage Support

    Kannika Armory now supports Azure Blob Storage as a storage provider. It builds on top of the Storage and Credentials resources introduced in the 0.5 release. It uses the new storage driver architecture to provide a way to store your backups in Azure Blob Storage.

    Here is an example of how you can configure Kannika Armory to use Azure Blob Storage:

    apiVersion: kannika.io/v1alpha
    kind: Storage
    metadata:
      name: azure-storage
    spec:
      azureContainer:
        location: https://my-account.blob.core.windows.net/
        container: my-container
    ---
    apiVersion: kannika.io/v1alpha
    kind: Credentials
    metadata:
      name: azure-container-token
    spec:
      azureContainerToken:
        sasTokenFrom:
          secretKeyRef:
            name: azure-container-sas-token
            key: secretSasToken
    ---
    apiVersion: v1
    data:
      secretSasToken: |
        <Your container SAS token base64-encoded>
    kind: Secret
    type: Opaque
    metadata:
      name: azure-container-sas-token
    ---
    apiVersion: kannika.io/v1alpha
    kind: Backup
    metadata:
      name: backup
    spec:
      source: "my-kafka-endpoint"
      sink: "azure-storage"
      sinkCredentialsFrom:
        credentialsRef:
          name: azure-container-token

    This configuration create a new Storage resource called azure-storage, and a new Credentials resource called azure-container-token, using a Secret called azure-container-sas-token to store a SAS token. It is then used in a Backup resource to back up data from my-kafka-endpoint to the azure-storage Storage.

    For more information on how to configure Azure Blob Storage, please refer to the new Azure Blob Storage documentation.

    Google Cloud Storage Support

    The next storage driver is one for Google Cloud Storage. Just like with Azure Blob Storage and S3, it builds on top of the Storage and Credentials resources introduced in the 0.5 release.

    Here is an example of how you can configure Kannika Armory to use Google Cloud Storage:

    apiVersion: kannika.io/v1alpha
    kind: Storage
    metadata:
      name: gcs-storage
    spec:
      gcsBucket:
        bucket: my-bucket
    ---
    apiVersion: kannika.io/v1alpha
    kind: Credentials
    metadata:
      name: gcp-sa-creds
    spec:
      gcp:
        serviceAccountJsonFrom:
          secretKeyRef:
            name: gcp-sa
            key: contents
    ---
    apiVersion: v1
    kind: Secret
    type: Opaque
    metadata:
      name: gcp-sa
    data:
      contents: |
        <base64-encoded service account file (JSON)>
    ---
    apiVersion: kannika.io/v1alpha
    kind: Backup
    metadata:
      name: backup
    spec:
      source: "my-kafka-endpoint"
      sink: "gcs-storage"
      sinkCredentialsFrom:
        credentialsRef:
          name: gcp-sa-creds

    This configuration create a new Storage resource called gcs-storage, and a new Credentials resource called gcp-sa-creds, using a Secret called gcp-sa to store a GCP service account file. It is then used in a Backup resource to back up data from my-kafka-endpoint to the gcs-storage Storage. A ServiceAccount with Workload Identity can also be used to authenticate with Google Cloud Storage.

    For more information on how to configure Google Cloud Storage, please refer to the new Google Cloud Storage documentation.

    Resuming a Restore

    This release adds initial support for resuming a restore from where it left off.

    Even though Kannika Armory is designed to be robust and reliable, there are still situations where a restore might fail or be interrupted. This could be due to network issues, or the restore process being killed for running out of memory, or misconfiguration of the restore process.

    Another situation where resuming a restore is useful is when you have restored data to another environment, perhaps while a Backup was running, and then you want to restore more data at a later time.

    A Restore will now track its progress in a report that is persisted between runs. This report is used to resume the restore from where it left off. Please refer to the new Resuming a Restore documentation for more information.

    Other Changes

    • Backups are recreated upon config changes. When a Backup resource is updated, any running Pod will be terminated first before a new one is created with the new configuration. This replaces the old behaviour of a rolling update which caused some issues with shared resources.
    • Kafka Endpoint ssl.key.password support. The Kafka Endpoint now supports the ssl.key.password configuration.
    • Additional Properties. The Backup and Restore resources now support additional properties for their sink and source configurations. See the Backup and Restore section for more information.

    Breaking Changes

    • Sidecar configuration removed. Please remove any related configuration from your resources and installation configuration. This includes the sidecar fields in the Backup and Restore resources, as well as the operator.config.s3 settings in the Helm chart.
    • Existing Deployment resources must be deleted. Due to the change of Backup deployment strategy mentioned above, and due to changes in the way Backup resources are patched, it is required to delete the associated Deployment resources for Backup resources. The Deployment will automatically be recreated with the new configuration.

    Wrapping up

    We are pleased with the direction that Kannika Armory is heading in with this release.

    We would love to hear your feedback on these new features. Please try it out and let us know what you think! You can reach out to us on Slack if you have any questions or feedback.

    In the next release, we will be focusing on the following features:

    • Kafka Event Hub. A new Kubernetes resource to define connections to Kafka and removing the legacy Endpoint resources.
    • The Kannika Armory console. We will release a new web-based console to manage your backups and restores.

    We hope you enjoy this release, and we are looking forward to hearing your feedback!