Skip to content
Kannika

    Overview

    A Storage resource is used to define where the data of a Backup should be stored, or where a Restore should read the data from.

    Usage

    Storage resources can be managed using the kubectl command line tool, and are available by the name storage or storages.

    Terminal window
    $ kubectl get storages
    NAME            STATUS   AGE   LOCATION
    s3-storage      Ready    1s    s3://kannika-aws-s3/backups
    gcs-storage     Ready    1s    gs://kannika-gcs/backups
    azure-storage   Ready    1s    https://kannika-azure.blob.core.windows.net/backups
    volume-storage  Ready    1s

    Supported Storage

    Currently, the following storage systems are supported:

    Storage Class

    It is also possible to configure a custom StorageClass that can be used in combination with Volume Storage. This allows you to offload your data to any storage provider that is supported by Kubernetes.

    Please check the Storage Class section for more information.

    Objects stored in an Armory Storage

    Kafka Records Backups

    Armory stores different kinds of objects in a storage. The directory structure looks like the following:

    • Directoryprefix optional prefix when defining a Storage
      • Directorytopic.A all topics have their own directory
        • index a file with metadata on this topic’s data
        • ULID_1.kan a segment containing kafka records
        • ULID_2.kan another segment containing kafka records
        • …
        • ULID_N.kan
        • __lease a temporary file to prevent multiple backups from writing to the same storage
        • __time a temporary file to query the storage’s time; used when acquiring a lease on a storage.
      • Directorytopic.B another topic’s directory
        • …

    Here is a high-level description of what they are.

    The index file

    This is the most important object in a topic’s backup. It is a yaml file that contains information on the content of the backup such as which records are stored in a particular .kan file, and expiration dates.

    If a retention period has been defined on a backup, an index may contain references to expired (and deleted) .kan files.

    An index may be rebuilt if it is deleted by accident. However, there is a risk some information (such as expiration times) will be lost in the process.

    .kan files

    These are your topic’s data. Each .kan segment present in the storage should be referenced in the index along with some metadata.

    __lease & __time

    Depending on the storage type you chose, you may find these temporary files in a topic’s directory.

    These are temporary objects used when Backups acquire a lease on a storage. The leasing mechanism prevents multiple Backups from writing to the same location.

    Armory is able to detect orphan leases after some time should a backup fail unexpectedly. Therefore, it is not necessary to remove these files manually if you notice a backup complaining about not being able to acquire a lease. The situation will resolve itself after a few minutes.

    Note that the process must be able to delete the lease files in order to clean up after itself.

    Schema Registry Backups

    Confluent

    The directory structure of confluent schema registry backup is as follows:

    • Directoryprefix optional prefix when defining a Storage
      • Directorysubject.A all subjects have their own directory
        • 1 version 1
        • 2 version 2
      • Directorysubject.B another subject’s directory
        • …

    Subject version files are stored as json documents.

    Troubleshooting storage errors

    storage error: resource already held by someone else

    This error indicates that a process is trying to acquire a lease on the storage, but it cannot because there is already lease file present in the storage.

    • Orphan lease: This is the most common reason for this error. An orphan lease is a lease file that was left behind by a process that crashed or did not clean up after itself. Armory will automatically clean up orphan leases after a few minutes, but if you need to resolve the issue immediately, you can delete the __lease file in the storage directory of the topic that is causing the error.

    • Another process is using the storage: This can happen if you have multiple processes using the same storage. In case of a Backup, check that there are no other backups writing to the same directory of each topic. See the directory structure of Backups for more information.

    • Lack of delete permissions: The process that was using the storage did not have permission to delete the lease file, leaving behind an orphan lease. Check the permissions of the storage and ensure the process has the necessary rights to delete files.

    • Process crashed without cleaning up: The process that was using the storage crashed and did not clean up after itself, leaving behind an orphan lease. In this case, the lease will be automatically cleaned up after a few minutes if the process does not crash again once restarted.

    • Resource issues: Very similar to the previous case, it is possible that the process that was using the storage did not have enough resources (memory, CPU, etc.) and did not clean up after itself, leaving behind an orphan lease. Usually with memory issues, the process will keep getting OOMKilled until the memory limits are increased. Increase the resource limits of the process, to allow it to run properly and clean up after itself.