To optimize a Sorter transformation, allocate enough memory to sort the data and specify a different work directory for each partition in the transformation.
Allocating memory
For optimal performance, configure the Sorter cache size with a value less than or equal to the amount of available physical RAM on the Secure Agent machine.
Allocate at least 16 MB of physical memory to sort data using the Sorter transformation. The Sorter cache size is set to 16,777,216 bytes by default. If Data Integration can't allocate enough memory to sort data, the mapping task fails.
If the amount of incoming data is greater than the amount of Sorter cache size, Data Integration temporarily stores data in the Sorter transformation work directory. Data Integration requires disk space of at least twice the amount of incoming data when storing data in the work directory. If the amount of incoming data is significantly greater than the Sorter cache size, Data Integration may require much more than twice the amount of disk space available to the work directory.
Note: The session log contains the input row count and the size of incoming data for the Sorter transformation. For example, the following message appears when Data Integration processes the Sorter transformation:
SORT_40422 End of output from Sorter Transformation [srt_1_Billion_FF]. Processed 999999999 rows (866325637228 input bytes; 868929593344 temp I/O bytes)
In the message, the number of input rows is 999999999 and the size of the input rows is 866325637228.
Specifying a work directory
Data Integration creates temporary files when it sorts data. It stores them in a work directory. You can specify any directory on the Secure Agent machine to use as a work directory.
By default, Data Integration uses the value specified for the $PMTempDir service process variable. When you partition a mapping with a Sorter transformation, you can specify a different work directory for each partition in the pipeline. To increase mapping performance, specify work directories on physically separate disks on the Secure Agent machine.
To specify the work directory for a Sorter transformation, use the Work Directory advanced property in the Sorter transformation.