Troubleshooting a mapping

The performance of a mapping that uses a Microsoft Fabric Data Warehouse connection is not optimal in the first few runs.

When you run a Microsoft Fabric Data Warehouse mapping, the performance is slow in the first few runs compared to subsequent runs. The slowdown occurs because of a third-party limitation related to the statistics that the query engine creates and optimizes in Microsoft Fabric Data Warehouse. Accurate statistics are crucial for optimal query performance in Microsoft Fabric Data Warehouse, and the initial mapping runs are necessary to generate and optimize these statistics.

For more information on this issue and its resolution, see the Microsoft Fabric Data Warehouse performance guidelines in the Microsoft documentation.

Mapping configured to read or write Date and Datetime data types fails.

A mapping in advanced mode fails in the following cases:

• - Data is of the Date data type and the date is less than 1582-10-15.
• - Data is of the Datetime data type and the timestamp is less than 1900-01-01T00:00:00Z.

To resolve this issue, specify the following spark session properties in the mapping task or in the custom properties file for the Secure Agent:

• - spark.sql.legacy.timeParserPolicy=LEGACY
• - spark.sql.parquet.int96RebaseModeInWrite=LEGACY
• - spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY
• - spark.sql.parquet.int96RebaseModeInRead=LEGACY
• - spark.sql.parquet.datetimeRebaseModeInRead=LEGACY