Data Engineering
Read this section to learn what's changed for Data Engineering products in version 10.5.
Documentation
Effective in version 10.5, instructions for integrating Informatica with a Google Dataproc cluster are in the Data Engineering Integration Guide.
Previously, Dataproc integration instructions were published in How-To Library (H2L) articles. You can view these articles, which cover Dataproc integration with Data Engineering 10.4, on the How-to Library tab in the
Informatica Documentation Portal.
Performance
Effective in version 10.5, the PERCENTILE function on the Spark engine consumes less memory, especially when multiple columns compute the PERCENTILE function on the same data set at different percentiles.
Processing Invalid Values
Effective in version 10.5, the Spark engine writes NULL values to the target when the mapping contains certain invalid values.
The Spark engine writes NULL values in the following situations:
- •The terms argument in PV, FV, PMT, and RATE finance functions passes a 0 value. The value of terms must be an integer greater than 0.
- •The month argument in the MAKE_DATE_TIME function passes an invalid value. The value of month must be from 1 to 12.
Previously, the Spark engine wrote random values.
Note: The Data Integration Service rejects rows with invalid values. If you want rows with invalid values to be rejected and not written to the target, run the mapping in the native environment.