Override the Maximum Parallelism for a Mapping
By default, the Maximum Parallelism property for each mapping is set to Auto. Each mapping uses the maximum parallelism value defined for the Data Integration Service. You can override the maximum parallelism value to define a maximum value for a particular mapping.
When maximum parallelism is set to different integer values for the Data Integration Service and the mapping, the Data Integration Service uses the minimum value of the two.
You might want to override the Maximum Parallelism property for a mapping for the following reasons:
- You run a complex mapping that results in more threads than the CPU can handle.
- The total number of parallel threads that can run for the complete mapping pipeline is the parallelism value multiplied by the number of pipeline stages. Each partition point adds an additional pipeline stage. A complex mapping with multiple Aggregator or Joiner transformations might have many pipeline stages. A large number of pipeline stages can cause the Data Integration Service to use more threads than the CPU can handle.
- Mapping performance is satisfactory with fewer parallel threads for each pipeline stage.
When a single mapping runs with fewer parallel threads, more threads are available for the Data Integration Service to run additional jobs.
- You want to define a suggested parallelism value for a transformation.
- If you override the maximum parallelism for a mapping, you can define a suggested parallelism value for a specific transformation in the mapping. You might want to define a suggested parallelism value to optimize performance for a transformation that contains many ports or performs complicated calculations.
- You want to define an execution instances value for an Address Validator or Match transformation.
- If you override the maximum parallelism for a mapping, the Data Integration Service considers the execution instances value for an Address Validator or Match transformation in the mapping. You might want to define an execution instances value to optimize performance for the transformation.
Suggested Parallelism for a Transformation
If you override the Maximum Parallelism run-time property for a mapping, you can define the Suggested Parallelism property for a specific transformation in the mapping run-time properties.
The Data Integration Service considers the suggested parallelism value for the number of threads for that transformation pipeline stage as long as the transformation can be partitioned. For example, if you configure the mapping to maintain row order, the Data Integration Service might need to use one thread for the transformation.
If the Maximum Parallelism run-time property for the mapping is set to Auto, you cannot define a suggested parallelism value for any transformations in the mapping. If you set the maximum parallelism value for the mapping to Auto after defining a suggested parallelism value for a transformation, the Data Integration Service ignores the suggested parallelism value.
You might want to define a suggested parallelism value to optimize performance for a transformation that contains many ports or performs complicated calculations.
For example, if a mapping enabled for partitioning processes a small data set, the Data Integration Service might determine that one thread is sufficient to process an Expression transformation pipeline stage. However, if the Expression transformation contains many complicated calculations, the transformation pipeline stage can still take a long time to process. You can enter a suggested parallelism value greater than 1 but less than the maximum parallelism value defined for the mapping or the Data Integration Service. The Data Integration Service uses the suggested parallelism value for the number of threads for the Expression transformation.
You can configure the following values for the Suggested Parallelism property for a transformation when you override the maximum parallelism for the mapping:
Suggested Parallelism Value | Description |
|---|
1 | The Data Integration Service uses one thread to run the transformation. |
Auto | The Data Integration Service considers the maximum parallelism defined for the mapping and for the Data Integration Service. The service uses the lowest value to determine the optimal number of threads that run the transformation. Default for each transformation. |
Greater than 1 | The Data Integration Service considers the suggested parallelism defined for the transformation, the maximum parallelism defined for the mapping, and the maximum parallelism defined for the Data Integration Service. The service uses the lowest value for the number of threads that run the transformation. |
You can define the Suggested Parallelism property in the mapping run-time properties for the following transformations:
- •Aggregator
- •Expression
- •Filter
- •Java
- •Joiner
- •Lookup
- •Normalizer
- •Rank
- •Router
- •Sequence Generator
- •Sorter
- •SQL
- •Union
- •Update Strategy
Execution Instances for Address Validator and Match Transformations
If you override the Maximum Parallelism run-time property for a mapping, the Data Integration Service considers the value of the Execution Instances advanced property defined for an Address Validator or Match transformation.
The Data Integration Service considers the execution instances value for the number of threads for that transformation pipeline stage as long as the transformation can be partitioned. For example, if you configure the mapping to maintain row order, the Data Integration Service might need to use one thread for the transformation.
You can increase the number of execution instances on a Match transformation when you configure the transformation for identity match analysis. You cannot increase the number of execution instances on a Match transformation when you configure the transformation for field match analysis. In field match analysis, the Match transformation uses a single execution instance.
If the Maximum Parallelism run-time property for a mapping is set to Auto, the Data Integration Service ignores the execution instances value defined for an Address Validator or Match transformation.
You can configure the following values for the Execution Instances advanced property for an Address Validator or Match transformation when you override the maximum parallelism for the mapping:
Execution Instances Value | Description |
|---|
1 | The Data Integration Service uses one thread to run the transformation. Default for the Address Validator transformation. |
Auto | The Data Integration Service considers the maximum parallelism defined for the mapping and for the Data Integration Service. The service uses the lowest value to determine the optimal number of threads that run the transformation. Default for the Match transformation in identity match analysis. |
Greater than 1 | The Data Integration Service considers the execution instances defined for the transformation, the maximum parallelism defined for the mapping, and the maximum parallelism defined for the Data Integration Service. The service uses the lowest value for the number of threads that run the transformation. |
Note: The Data Integration Service also considers the Max Address Object Count property on the Content Management Service when it calculates the optimal number of threads for an Address Validator transformation. The Max Address Object Count property determines the maximum number of address validation instances that can run concurrently in a mapping. The Max Address Object Count value must be greater than or equal to the maximum parallelism value on the Data Integration Service.
Overriding the Maximum Parallelism Value
To override the maximum parallelism value, set maximum parallelism in the mapping run-time properties to an integer value greater than 1 and less than the value set for the Data Integration Service.
1. Open the mapping.
2. In the Properties view, click the Run-time tab.
3. Select Native for the Execution Environment.
4. For the Maximum Parallelism property, enter an integer value greater than 1 and less than the value set for the Data Integration Service.
Or you can assign a user-defined parameter to the property, and then define the parameter value in a parameter set or a parameter file.
5. To define a suggested parallelism value for a specific transformation in the mapping, enter an integer value greater than 1 for the transformation in the Suggested Parallelism section.
The following image shows a mapping with an overridden maximum parallelism value and with the default suggested parallelism values for transformations:
6. Save the mapping.
7. To define an execution instances value for an Address Validator or for a Match transformation configured for identity match analysis, complete the following steps:
- a. Open the Address Validator or Match transformation.
- b. In the Advanced view, enter an integer value greater than 1 for the Execution Instances property.
- c. Save the transformation.
