How do I back up a Kafka Streams application?

A Kafka Streams application creates two kinds of internal topics behind the scenes, and they need very different treatment when backing up.

Changelog topics (<application.id>-<store-name>-changelog) are compacted topics that mirror every state store. They are the durable, canonical copy of the application’s state.
Repartition topics (<application.id>-<name>-repartition) are transient scratch topics used to re-key data before aggregations and joins. Kafka Streams actively purges them as records are consumed, and they can always be regenerated by replaying the source topics.

Back up the changelog topics, and ignore the repartition topics. Backing up a repartition topic only captures a partial, mid-flight slice that Kafka Streams would discard and rebuild anyway.

Do I actually need to back up the changelog?

It depends on what you are protecting against.

A plain application restart is already handled by Kafka Streams natively. When an instance restarts without its local state stores (for example, when it runs on ephemeral storage), Kafka Streams replays the changelog topics that live in the Kafka cluster to rebuild them. No backup is involved: the changelog topics in the broker are doing exactly their job.

Backing up the changelog topics matters for disaster recovery: losing the cluster, accidentally deleting the topics, corruption, or migrating to a different cluster. In those cases the native mechanism has nothing to replay from, and a backup lets you restore the state stores.

Selecting the right topics

Because changelog and repartition topics follow predictable naming patterns, you can capture the changelogs and drop the repartition topics with a topic selector. Match the changelog topics and exclude the repartition topics with an excludeMatchers rule:

apiVersion: kannika.io/v1alpha
kind: Backup
metadata:
  name: streams-backup
spec:
  source: "my-kafka-cluster"
  sink: "my-storage"
  topicSelectors:
    matchers:
    - name:
        glob: "*-changelog"     # Back up the state store changelogs
    excludeMatchers:
    - name:
        glob: "*-repartition"   # Skip the transient repartition topics

The excludeMatchers rules take precedence over matchers, so a broader matchers rule (for example the application’s topic prefix) still works as long as the repartition topics are excluded.

Restore considerations

Kafka Streams state is only meaningful relative to the source topics and consumer group offsets.

For a consistent restore:

Recreate each restored changelog topic as compacted (cleanup.policy=compact), otherwise the state store semantics are lost.
Restore the changelog topics together with their source topics so the state matches the input the application has already processed.
Migrate the application’s consumer group offsets so it resumes from the right place. See Are consumer groups migrated?.