Sorter Transformation in the Hadoop Environment
The Sorter transformation processing in the Hadoop environment depends on the engine that runs the transformation.
Sorter Transformation Support on the Blaze Engine
Some processing rules for the Blaze engine differ from the processing rules for the Data Integration Service.
Mapping Validation
Mapping validation fails in the following situations:
- •The target is configured to maintain row order and the Sorter transformation is not connected directly to a flat file target.
Global Sort
The Blaze engine can perform global sorts when the following conditions are true:
- •The Sorter transformation is connected directly to flat file targets.
- •The target is configured to maintain row order.
- •The sort key is not a binary data type.
If any of the conditions are not true, the Blaze engine performs a local sort.
Data Cache Optimization
If a Sorter transformation is inserted before an Aggregator or Rank transformation to optimize the Aggregator or Rank data cache, the size of the sorter cache is the same size as the data cache for the Aggregator or Rank transformation. To configure the sorter cache, you must configure the size of the data cache for the Aggregator or Rank transformation.
Sorter Transformation Support on the Spark Engine
Mapping validation fails when case sensitivity is disabled.
The Data Integration Service logs a warning and ignores the Sorter transformation in the following situations:
- •There is a type mismatch in between the target and the Sorter transformation sort keys.
- •The transformation contains sort keys that are not connected to the target.
- •The Write transformation is not configured to maintain row order.
- •The transformation is not directly upstream from the Write transformation.
The Data Integration Service treats null values as low even if you configure the transformation to treat null values as high.
Data Cache Optimization
You cannot optimize the sorter cache to store data using variable length.
Sorter Transformation Support on the Hive Engine
Mapping validation fails when case sensitivity is disabled.
The Data Integration Service logs a warning and ignores the Sorter transformation in the following situations:
- •There is a type mismatch between the Sorter transformation and the target.
- •The transformation contains sort keys that are not connected to the target.
- •The Write transformation is not configured to maintain row order.
- •The transformation is not directly upstream from the Write transformation.
The Data Integration Service treats null values as low even if you configure the transformation to treat null values as high.
Data Cache Optimization
You cannot optimize the sorter cache to store data using variable length.