Configure incremental load to read from Kafka topics
You can configure incremental load in a mapping in advanced mode to read from Kafka source topics.
When you re-run mappings enabled for incremental load, the mapping does not read records from previous runs.
To enable incremental load, configure the consumer property group ID in the Consumer Configuration Properties source advanced property. The group ID consumer property is the name of the consumer group to which the Kafka consumer belongs.
Enter the group ID of the consumer group in the key1=value1 format.
For example, enter the following group ID of the consumer group: group.id=consumer-1.
To configure multiple consumer properties, separate each key-value pair with a comma.
When you configure the consumer property group ID, ensure that you set the Start Position Offset property to Earliest and the Connector Mode property to Batch.
The offset of the next record for the configured consumer group is stored after the successful completion of each mapping run. When you re-run the mapping, it starts reading data from the stored offset point.
If you set the Start Position Offset property to Custom and configure the Custom Start Position Timestamp property, the mapping ignores the recorded offset value saved from the previous run and starts reading data from the value set in the Custom Start Position Timestamp property. This option helps you customize exactly where in the Kafka log the mapping must start reading the data.
If you do not configure the group ID consumer property, the mapping automatically configures a unique consumer group name and does not incrementally load the data. Each mapping re-run reads all existing records in the source topic.