You need to follow certain rules and guidelines to configure tasks that uses Open Table Connector.
Consider the following rules and guidelines when you configure a task:
•A mapping fails at run time when you specify a simple filter with Starts With, Ends With, or Contains operators to retrieve data. To avoid this issue, configure an advanced filter with a like query to filter query results based on patterns instead of exact textual matches.
•When you configure an advanced data filter for a field of Timestamp data type, the filter ignores the time and considers only the date.
•When you use a simple filter on the date or timestamp data type column to read data, the task fails at run time with the following error:
An error occurred during mapping translation: Invalid expression string for filter condition
As a workaround, use the advance filter to filter date and timestamp data type columns from an Open Table source.
•You can't use uncached lookup in the Lookup transformation.
•When you run a mapping with an Open Table connection to read from or write data to Apache Iceberg tables available in AWS Glue Catalog, the mapping fails if the source and target are pointing to different AWS Glue catalogs available in different regions.
•When you run a mapping with an Upsert target operation and all the condition fields are not mapped in the field mapping, the mapping fails.
•When you configure a mapping with an Apache Iceberg table, the mapping fails in the following scenarios:
- If the source table is managed by the AWS Glue Catalog and the target table is managed by the Hive Metastore catalog.
- If the source table is managed by the Hive Metastore catalog and the target table is managed by the AWS Glue Catalog.
•When you configure a mapping with an Apache Iceberg table managed by Hive Metastore, the mapping fails in the following scenarios:
- If the source table is stored in Amazon S3 and the target table is stored in Microsoft Azure Delta Lake Storage Gen2.
- If the source table is stored in Microsoft Azure Delta Lake Storage Gen2 and the target table is stored in Amazon S3.
•When you read data from an Apache Iceberg table managed by the Hive Metastore catalog in a mapping, the preview job fails.
•When you read a large volume of data from an Apache Iceberg table managed by the Hive Metastore catalog in a mapping, the data preview at the source table fails.
•A mapping fails when you configure an Apache Iceberg table as the source and a Delta Lake table as the target. Similarly, the mapping fails when you configure a Delta Lake table as the source and an Apache Iceberg table as the target.
•The runtime statistics of the Lookup transformation don't show the count of records processed based on the lookup condition. Instead, it shows the count of rows read from the source table.
•When a mapping fails due to rejected or error rows at the target, the runtime statistics for the source and target don't show the correct information.