Consider the following rules and guidelines when you configure a Kafka mapping:
•You cannot parameterize the source object in a mapping.
•You cannot configure partitioning in a mapping that reads data from a Kafka source in batch mode.
•You cannot configure Joiner, Lookup, and Sorter transformations in a mapping.
•When you use a Secure Agent group to run a mapping, you cannot configure message recovery.
•When you configure a mapping that reads hierarchical data types from the source, the mapping might fail at runtime.
•When you select JSON as the format type, you cannot read hierarchical data from a Kafka source or write hierarchical data to a Kafka target.
•When you abort a mapping task or terminate the Data Transformation Manager (DTM) process, the number of success rows in the session log and the target results page of the mapping task do not match.
•For a Kafka connection used in an existing mapping that reads Avro data from or writes Avro data to a Kafka topic, do not configure the Schema Registry URL connection property. Otherwise, when you rerun the same mapping rerun, the mapping fails with the following error:
•When you run a mapping that reads from a Kafka source and writes to a target of format JSON type and the target includes the key field, the mapping fails.
•The data preview in the Target transformation displays an additional field, offset, received from the source data. This field is not available in the Kafka target and can't be mapped to the target.
•The header data is encoded in UTF-8 when you read from the source and write to the target.
•You can use the Avro formatting option with the Kafka connector that uses the Confluent schema registry.
•When you read Avro data from or write Avro data to a Kafka topic with Confluent Schema Registry, the mapping runs successfully with the latest registered schema for the selected topic. In the subsequent runs, if the schema is updated, ensure that you refresh the metadata for the mapping or create a new mapping.
•You can configure mappings that connect to Confluent Kafka with schema registry when the schema registry response contains the metadata field.
•You can't use the use.latest.with.metadata client configuration property for the application versioning scheme in Confluent schema registry.
•When you read from or write to a column of Boolean data type in a Kafka topic with Confluent schema registry, ensure that you specify 0 for False and 1 for True.
•When you read Avro data from or write Avro data to a Kafka topic with Confluent schema registry, you cannot view the schema in the formatting options.
•A Kafka mapping might fail to fetch source records during data preview or at runtime.
To fetch all source records, increase the value of the Polling Interval source advanced property.
•You cannot configure partitioning in Realtime mode if the Kafka connection is configured to use the Confluent schema registry.