Troubleshooting a mapping in advanced mode

Mapping configured to read or write Date and Int96 data types for Parquet file fails

A mapping configured to read from or write to a Parquet file fails in the following cases:

• - Data is of the Date data type and the date is less than 1582-10-15.
• - Data is of the Int96 data type and the timestamp is less than 1900-01-01T00:00:00Z.

To resolve this issue, specify the following spark session properties in the mapping task or in the custom properties file for the Secure Agent:

• - spark.sql.parquet.int96RebaseModeInWrite=LEGACY
• - spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY
• - spark.sql.parquet.int96RebaseModeInRead=LEGACY
• - spark.sql.parquet.datetimeRebaseModeInRead=LEGACY

Mapping fails when you write data to flat files

When you write data to flat files in Microsoft Fabric OneLake, the mapping fails with the following error: Caused by: java.io.IOException: Failed to rename VersionedFileStatus{VersionedFileStatus

To resolve this issue, perform the following steps:

1. 1Under Spark Session Properties, click Add and select spark.custom.property in the Session Property Name field.
2. 2In the Session Property Value field, specify the following property and set its value to DIRECT:
infaspark.flatfile.writer.committer.name
For example, infaspark.flatfile.writer.committer.name=DIRECT