Backup: offset reset or invalid seek error

A backup may fail with offset reset warnings followed by an invalid seek error. This happens when the backup’s stored offsets no longer match the offsets available on the Kafka topic.

Common causes include:

• The topic was deleted and recreated;
• The topic was truncated (e.g. via retention policy or manual deletion of segments);
• The topic’s log segments were compacted beyond the backup’s last known offset.

Recognizing the error

The backup logs will contain messages similar to:

WARN  offset reset (at offset 42, broker 1) to offset BEGINNING:
      fetch failed due to requested offset not available on the broker:
      Broker: Offset out of range

WARN  Got an Offset Reset warning. Some records may have been missed.

WARN  Backup interrupted. err: kafka error: detected invalid seek from rdkafka

The backup will keep restarting, but it cannot make progress because the offsets it tries to resume from no longer exist on the topic.

Resolution

To resolve this, you must clear the topic’s backup data from storage so that the backup can start fresh:

1. Pause the backup, or pause the affected topic.

2. Delete the topic’s backup data from the configured Storage. Navigate to the configured storage and remove the topic directory for the affected topic.

3. Resume the backup. The backup will start consuming from the beginning of the topic.

Caution

Removing the topic’s backup data means that any previously backed-up records for that topic are lost. If you need to preserve historical data, copy the backup data to a separate location before deleting it.

Automating with storage snapshots

If you have scheduled storage snapshots configured, the backup data is automatically recreated from the latest snapshot. This means the offset mismatch resolves itself without manual intervention, as the backup starts fresh from a clean state.

Note

In a future release, Kannika will support clearing topic backup data directly from the Console, and optionally handling offset mismatches automatically.