Overview
A Restore is used to restore messages from cold storage to a message source like Kafka.
It has many different configuration options to allow for a wide range of use cases. You can use it to restore data from multiple topics in parallel, restore data until a certain point in time or until a certain offset.
Another use case is to restore or migrate data from one cluster to another cluster. It supports mapping schemas between topics, renaming topics, and much more.
Usage
Restore resources can be managed using the kubectl
command line tool,
and are available by the name restore
or restores
.
Creating a Restore
The following is an example of a Restore.
It will restore 2 topics from the my-volume-storage
Storage to the my-kafka-cluster
Endpoint.
In this example:
-
A Restore named
restore-example
is created, indicated by the.metadata.name
field. This name will become the basis for the Kubernetes Job (and other resources) which are created later. -
The Restore will connect to the
my-volume-storage
Storage to fetch data, indicated by the.spec.source
field. The Restore will restore data to themy-kafka-cluster
Event Hub Endpoint defined in thespec.sink
field. -
The
.spec.config.mapping
field contains the mapping between the source and target topics. In this example, the topicssource-topicA
andsource-topicB
will be restored totarget-topicA
andtarget-topicB
respectively. -
The
.spec.enabled
field indicates whether the Restore should be enabled or not. Note that this field is optional and defaults tofalse
. This means that the Restore will not be started unless this field is explicitly set totrue
. Please check the Drafting and starting a Restore section for more information.
Drafting and starting a Restore
The following is an example of a Draft Restore.
This is a minimal Restore definition.
In this state,
the Restore will not be started (the Job will not be created yet)
because the .spec.enabled
field is not set to true
.
The Restore is in a Draft state.
At this stage of the Restore lifecycle, the Restore can act as a working document, and be edited and modified as needed. More importantly, topics can be imported and mapped to target topics, or removed.
To start the Restore,
the .spec.enabled
field must be set to true
.
Restore to multiple topics in parallel
The following is an example of a draft Restore that can restore to multiple topics in parallel.
In this example,
the .spec.config.parallelism
field is set to 3
.
This means that the Restore will restore 3 topics in parallel.
Use cases where this is useful:
- when restoring a large number of topics
- when restoring large, single partition topics
By default, a Restore will use a single producer at any given time. This may limit the amount of concurrency achievable if the target topics are configured in a way that requires different producer configurations.
At this time,
a matching value of message.max.bytes
is the only cluster-side topic setting considered for sharing a producer,
but more may be used in the future.
To increase the number of producers a Restore is allowed to use concurrently,
set the spec.config.maxProducers
attribute:
In this example,
the .spec.config.parallelism
field is set to 3
and .spec.config.maxProducers
is set to 2
.
This means that the Restore will restore up to 3 topics in parallel
and that it is allowed to create an additional producer if required.
Adding the legacy offset to the headers
You can add the original offsets to the headers of the restored messages.
The following is an example of a Restore that adds the original offset to the headers of the restored message.
In this example,
the .spec.config.legacyOffsetHeader
field is set to original-offset
.
This means that the original offset will be added to the headers of the restored messages with the key original-offset
.
If you wish to add the header only to a specific topic and not to all topics, you must split the Restore into multiple Restores.
Restoring data between two dates
It is possible to restore records whose timestamp is within some time period.
Two settings are available to achieve this:
restoreFromDateTime
: start restoring from the given datetimerestoreUntilDateTime
: restore data up to the given datetime
When used together, those settings define a time window.
The following is an example of a Restore that restores the data up to the start of the year 2021:
In this example,
the .spec.config.restoreUntilDateTime
field is set to 2021-01-01T00:00:00Z
.
This means that the Restore job will only restore data with a timestamp earlier than January 1st, 2021.
Any messages with a timestamp from this point in time will not be restored.
In the next example, we define a time window using both parameters in order to restore 1 month worth of data:
In this example,
the .spec.config.restoreFromDateTime
and .spec.config.restoreUntilDateTime
are used together to instruct the Restore to select only records from the month of January 2021.
Selecting offsets to restore
For each partition of a given topic, you can define a range of offsets to restore.
The following is an example of a Restore using multiple filtering rules:
In this example,
we’ve defined various combinations of .spec.config.mapping.source-topic.partitions.*.restoreFromOffset
and .spec.config.mapping.source-topic.partitions.*.restoreUntilOffset
to request different filtering rules.
The Restore job will only restore:
- offsets from 0 (inclusive) to 100 (exclusive) from partition #0;
- offsets from 50 (inclusive) to 1500 (exclusive) from partition #1;
- everything from partition #2;
- nothing from every other partition (if any).
Pre-flight checks
A Restore performs pre-flight checks before starting the restore process for each topic. These checks are used to ensure that topic can be restored without any issues.
The current checks are:
- Check if the partition count of the source and target topics are the same
Future checks that are planned are:
- Check if the target topic exists
- Check if the target topic is empty
- Check if the target topic has the same configuration as the source topic
Check the roadmap for more information.
Disabling pre-flight checks
In some cases, it can be necessary to disable the pre-flight checks. For example, when using the Topic Repartitioning plugin to change the amount of partitions of a topic.
Fine-grained control over the pre-flight checks is not available yet,
but you can disable all pre-flight checks for a specific topic by setting the disablePreflightChecks
field to true
in the mapping configuration.
Example:
Mapping schemas
You can map schemas between the source and target topics when restoring data. Check the Schema Mapping page for more details