Partitioned Flat File Sources
When a mapping that is enabled for partitioning reads from a flat file source, the Data Integration Service can use multiple threads to read the file source.
The Data Integration Service can create partitions for the following flat file source types:
- •Direct file
- •Indirect file
- •Directory of files
- •Command
- •File or directory of files in Hadoop Distributed File System (HDFS)
When the Data Integration Service uses multiple threads to read a file source, it creates multiple concurrent connections to the source. By default, the Data Integration Service does not preserve row order because it does not read the rows in the file or file list sequentially. To preserve row order when multiple threads read from a single file source, configure concurrent read partitioning.
When the Data Integration Service uses multiple threads to read a direct file, it creates multiple reader threads to read the file concurrently.
When the Data Integration Service uses multiple threads to read an indirect file or a directory of files, it creates multiple reader threads to read the files in the list or directory concurrently. The Data Integration Service might use multiple threads to read a single file. Or, the Data Integration Service might use a single thread to read multiple files in the list or directory.
Concurrent Read Partitioning
To preserve row order when multiple threads read from a single file source, configure the Concurrent Read Partitioning property for a flat file data object to preserve the order.
Configure the Concurrent Read Partitioning property in the Advanced properties for the flat file data object. Find the property in the Runtime: Read section.
Select one of the following options for the Concurrent Read Partitioning property:
- Optimize throughput
The Data Integration Service does not preserve row order when multiple partitions read from a single file source. Use this option if the order in which multiple partitions read from a file source is not important.
Default option.
- Keep relative order
Preserves the sort order of the input rows read by each partition.
The following table shows an example sort order of a file source with 10 rows read by two partitions:
Partition | Rows Read |
|---|
Partition #1 | 1,3,5,8,9 |
Partition #2 | 2,4,6,7,10 |
- Keep absolute order
Preserves the sort order of all input rows read by all partitions. In a pass-through mapping with passive transformations, the order of the rows written to the target is in the same order as the input rows.
The following table shows an example sort order of a file source with 10 rows read by two partitions:
Partition | Rows Read |
|---|
Partition #1 | 1,2,3,4,5 |
Partition #2 | 6,7,8,9,10 |