S3 Storage

---

This page describes the configuration options for the S3 Storage resource. The S3 Storage resource is used to configure an S3 bucket as a storage backend. It supports AWS S3 as well as S3-compatible storages such as MinIO and Ceph.

Synopsis

apiVersion: kannika.io/v1alpha
kind: Storage
metadata:
  name: s3-storage
spec:
  s3:
    bucket: my-bucket
    region: us-east-1                       # Optional, defaults to us-east-1
    prefix: /path/to/directory              # Optional
    description: "This is an S3 bucket"     # Optional
    endpoint: https://minio:9000            # Optional, for S3-compatible storages
    forcePathStyle: true                    # Optional, defaults to false
    caCertificateFrom:                      # Optional, for S3 servers with a private CA
      secretKeyRef:
        name: minio-ca
        key: ca.crt

Credentials: Access Key and Secret Key

apiVersion: kannika.io/v1alpha
kind: Credentials
metadata:
  name: aws-creds
spec:
  aws:
    description: "Access key authentication for AWS" # Optional description
    accessKeyIdFrom:
      secretKeyRef:
        name: aws-creds          # Name of the secret below
        key: accessKeyId         # References the access key id in the secret
    secretAccessKeyFrom:
      secretKeyRef:
        name: aws-creds          # Name of the secret below
        key: secretAccessKey     # References the secret access key in the secret
---
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: aws-creds
data:
  accessKeyId: dGhla2V5aWQ= # thekeyid
  secretAccessKey: dGhlU2VjcmV0S2V5 # theSecretKey

Configuring the bucket

To configure the bucket, you need to specify the bucket name and the region.

apiVersion: kannika.io/v1alpha
kind: Storage
metadata:
  name: s3-storage
spec:
  s3:
    bucket: my-bucket
    region: us-east-1
    prefix: /path/to/directory          # Optional
    description: "This is an S3 bucket" # Optional

Caution

Changing the bucket, region or prefix will not move existing data.

Using S3-compatible storages

The S3 Storage resource can be used with S3-compatible storages such as MinIO, Ceph, or any other service that implements the S3 API. To connect to a custom S3-compatible storage, configure the endpoint and forcePathStyle fields.

apiVersion: kannika.io/v1alpha
kind: Storage
metadata:
  name: s3-minio
spec:
  s3:
    bucket: my-bucket
    endpoint: http://minio:9000
    forcePathStyle: true

The endpoint field specifies the URL of the S3-compatible storage. For example, if MinIO is running in the same Kubernetes cluster under the service name minio, the endpoint is http://minio:9000. When endpoint is not set, the standard AWS S3 endpoints are used.

The forcePathStyle field controls how bucket addresses are constructed and defaults to false. S3 supports two addressing styles:

• When forcePathStyle is false (default), virtual-hosted-style addressing is used. The bucket name is part of the hostname, e.g. https://my-bucket.s3.amazonaws.com/key.
• When forcePathStyle is true, path-style addressing is used. The bucket name is part of the URL path, e.g. https://s3.amazonaws.com/my-bucket/key.

Most S3-compatible storages such as MinIO do not support virtual-hosted-style addressing and require forcePathStyle to be set to true.

When using S3-compatible storages, the region field is optional and defaults to us-east-1 if not specified. Some S3-compatible storages ignore the region entirely, but it is still sent as part of the request signing process.

Trusting a private CA

S3-compatible storages often serve HTTPS with a certificate issued by a private or self-signed CA. Armory only trusts the system roots by default, so the certificate of such a server fails verification. Use caCertificateFrom to trust the CA that issued the S3 server’s certificate. The certificate must be PEM-encoded and is read from a key in a Secret:

apiVersion: v1
kind: Secret
metadata:
  name: minio-ca
stringData:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
---
apiVersion: kannika.io/v1alpha
kind: Storage
metadata:
  name: s3-minio
spec:
  s3:
    bucket: my-bucket
    endpoint: https://minio:9000
    forcePathStyle: true
    caCertificateFrom:
      secretKeyRef:
        name: minio-ca
        key: ca.crt

A CA certificate is public material, so it can also be read from a ConfigMap:

    caCertificateFrom:
      configMapKeyRef:
        name: minio-ca
        key: ca.crt

The CA is trusted in addition to the system roots, not instead of them. Only server certificate verification can be configured: S3 authenticates requests by signing them, so there is no client certificate or key to set.

A Backup or Restore reads the certificate when it starts. After replacing the certificate in the Secret or ConfigMap, restart the Backups and Restores using the storage.

Authentication for S3 storage

Please see the section on AWS Authentication for information on how to configure access to the S3 Storage. Authentication for S3-compatible storages is configured in the same way as for AWS S3.

The following permissions are required for backup processes:

• s3:Put*: required to write backup data and leases.
• s3:Get*: required to read backup data and leases.
• s3:List*: required to list contents of the bucket.
• s3:Delete*: required to remove leases created by the backup process and to delete expired segments under a retention policy.
• s3:AbortMultipartUpload: required to abort the multipart upload of a segment that failed to upload, see Aborting incomplete multipart uploads.

The following permissions are required for restore processes:

• s3:Get*: required to read backup data and leases.
• s3:List*: required to list the contents of the bucket.

Depending on your setup, you may need to configure the following IAM policy for the S3 bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Principal": {
        "AWS": "arn:aws:iam::<uid>:user/<username>"
      },
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*",
        "s3:Put*",
        "s3:Delete*",
        "s3:AbortMultipartUpload"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket_name>",
        "arn:aws:s3:::<bucket_name>/*"
      ]
    }
  ]
}

Replace the <uid> and <username> with the appropriate values for your IAM user. The <bucket_name> should be replaced with the name of your S3 bucket.

Configuring a data retention policy

A retention policy to delete old data can be configured in a Backup. Use it instead of a lifecycle rule on the bucket, see Lifecycle rules.

Lifecycle rules

S3 lifecycle rules act on the objects in a bucket without Armory knowing about it. Most rules interfere with Armory’s own bookkeeping, so check the lifecycle configuration of the bucket before you use it for backups.

Expiring and transitioning objects

Do not use rules that expire objects (Expiration) or transition objects to another storage class (Transition).

Besides the segments with the backed up data, Armory stores files in the bucket to keep track of the backup. A rule that expires objects can delete any of these files, and a Backup or Restore that still needs them fails. A rule that transitions objects to an archive storage class, such as S3 Glacier Flexible Retrieval, makes these files unreadable until they are restored from the archive.

To delete old data, configure a data retention policy in the Backup instead.

Aborting incomplete multipart uploads

We recommend a rule that aborts incomplete multipart uploads (AbortIncompleteMultipartUpload). It is safe to use as long as it allows more time than the rollover timeout of every Backup that writes to the bucket.

A Backup writes the records of a topic to a segment, and keeps that segment open until it is rolled over. A segment is rolled over when it reaches its maximum size or when its rollover timeout expires, whichever happens first.

How a segment is uploaded to S3 depends on its size:

• A segment that stays below 5 MiB is uploaded as a single object when it is rolled over.
• A larger segment is uploaded as a multipart upload. The multipart upload starts once the segment grows past 5 MiB, and completes when the segment is rolled over.

While a segment is open, its multipart upload is incomplete. If the rule aborts it before the segment is rolled over, S3 discards the segment and the Backup fails to commit it. If uploading a part fails, the Backup aborts the multipart upload itself. This requires the s3:AbortMultipartUpload permission. Without it, the Backup logs a warning and leaves the upload incomplete. When a Backup stops without rolling over its open segment, for example because its pod is killed, the multipart upload of that segment stays incomplete and its parts keep taking up storage in the bucket. The rule removes these incomplete uploads.

For example, this rule aborts incomplete multipart uploads after 7 days, which is safe for any rollover timeout below 7 days:

lifecycle.json

{
  "Rules": [
    {
      "ID": "abort-incomplete-multipart-uploads",
      "Filter": {},
      "Status": "Enabled",
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 7
      }
    }
  ]
}

When S3 has discarded the multipart upload of an open segment, the Backup fails with the following error:

S3 discarded the multipart upload before it completed (check the bucket's lifecycle rule for incomplete multipart uploads against the segment rollover timeout)

To fix it, increase the DaysAfterInitiation of the lifecycle rule, or lower the rollover timeout of the Backup with .spec.segmentRolloverTriggers.timeoutSeconds.