In a mapping, you can configure a Source transformation to represent a Microsoft Fabric OneLake object.
The following table describes the Microsoft Fabric OneLake source properties that you can configure in a source transformation:
Property
Description
Connection
Name of the source connection. Select a source connection or click New Parameter to define a new parameter for the source connection.
If you want to overwrite the parameter at runtime, select the Allow parameter to be overridden at run time option when you create a parameter. When the task runs, the agent uses the parameters from the file that you specify in the task advanced session properties. Ensure that the parameter file is in the correct format.
When you switch between a non-parameterized and a parameterized Microsoft Fabric OneLake connection, the advanced property values are retained.
Source Type
Select Single Object or Parameter.
Object
Name of the source object.
Ensure that the headers or file data does not contain special characters.
Parameter
Select an existing parameter for the source object or click New Parameter to define a new parameter for the source object. The Parameter property appears only if you select Parameter as the source type.
When you parameterize the source object, specify the complete object path including the file system in the default value of the parameter.
If you want to overwrite the parameter at runtime, select the Allow parameter to be overridden at run time option when you create a parameter. When the task runs, the agent uses the parameters from the file that you specify in the task advanced session properties. Ensure that the parameter file is in the correct format.
Format
The format of the file that you want to read from Microsoft Fabric OneLake.
Default is None. If you select None as the format type, Microsoft Fabric OneLake Connector reads data from Microsoft Fabric OneLake files in binary format.
Note: Ensure that the source file is not empty. You cannot read a JSON file that exceeds 200 MB.
Overrides the lakehouse specified in the connection.
File Name Override
Overrides the source object at run time.
Source Type
The type of source from which you want to read data.
Select the following source types:
- File
- Directory
- Directory - Recursive
Select Directory - Recursive to recursively read from flat and complex files.
Default is File.
Compression Format
Reads compressed data from the source.
Select one of the following options:
- None. To read uncompressed and Parquet files.
- Gzip. To read flat files and Parquet files that use Gzip compression. The compressed files must have the .gz extension.
You cannot preview data for a compressed flat file.
Incremental File Load
Indicates whether you want to incrementally load files when you use a directory as the source for mappings in advanced mode.
When you incrementally load files, the mapping task reads and processes only the files in the directory that have changed since the mapping task last ran.
When you read an Avro, JSON, ORC, Parquet, flat file, or PDF, you can use the ? and * wildcard characters to define one or more characters in a search.
To use wildcard characters for the source file name, select the source type as Directory and enable the Allow Wildcard Characters option in the advanced source properties.
You can use the following wildcard characters:
? (Question mark)
The question mark character (?) allows one occurrence of any character. For example, if you enter the source file name as a?b.txt, the Secure Agent reads data from files with the following names:
- a1b.txt
- a2b.txt
- aab.txt
- acb.txt
* (Asterisk)
The asterisk mark character (*) allows zero or more than one occurrence of any character. If you enter the source file name as a*b.txt, the Secure Agent reads data from files with the following names:
- aab.txt
- a1b.txt
- ab.txt
- abc11b.txt
Rules and guidelines for wildcard characters
Consider the following rules and guidelines when you use wildcard characters:
Mappings
- You cannot use wildcard characters when you read from partition columns.
- When you read a complex file in a mapping, do not use a tilde (~) in the sub-directory name or file name.
- When you use wildcard characters in directory override, the Secure Agent reads data from the folders as well as the files that match the name pattern.
Mappings in advanced mode
- When you read a flat file or complex file and enable wildcard characters, ensure that the path specified in the directory override or file name override matches the file path in the source.
- When you use wildcard characters, ensure that the file name does not start with a special character.
- When you read a flat file, do not use the following special characters in the directory name or sub-directory name in the directory override:
[] {} " ' + ^ % * ? space
Reading files from subdirectories
You can read objects stored in subdirectories in Microsoft Fabric OneLake in mappings.
You can use recursive read for flat files and complex files in mappings. When you create a mapping in advanced mode, you cannot use recursive read for flat files.
To read objects stored in subdirectories, select the source type as Directory - Recursive in the advanced source properties.
Rules and guidelines for reading from subdirectories
Consider the following rules and guidelines when you read objects stored in subdirectories:
Mappings
- When you read from or write to a flat file in Microsoft Fabric OneLake, ensure that the directory or subdirectory name does not contain the percentage (%) character. Else, the mapping fails.
- You cannot use recursive read when you read from partition columns.
- When you read a complex file in a mapping, do not use a tilde (~) in the subdirectory name or file name.
Mappings in advanced mode
- When you read a complex file and enable recursive read, ensure that the path specified in the directory override or file name override matches the file path in the source.
- When you read a flat file, do not use the following special characters in the directory name or sub-directory name in the directory override:
[] {} " ' + ^ % * ? space
Incrementally loading files
You can incrementally load source files from a directory to read and process only the files that have changed since the last time the mapping task ran.
You can incrementally load files only in mappings in advanced mode. Ensure that all of the source files exist in the same Cloud environment. You can incrementally load files when you use ORC, Flat, Parquet, JSON, Discover Structure, and Document file formats.
To incrementally load source files, select Incremental File Load and Directory or Directory-Recursive as the source type in the advanced read options of the Microsoft Fabric OneLake object.
When you incrementally load files from Microsoft Fabric OneLake, the job loads files that have changed from the last load time to five minutes before the job started running. For example, if you run a job at 2:00 p.m, the job loads files changed before 1:55 p.m. The five-minute buffer ensures that the job loads only complete files because uploading objects on Microsoft Fabric OneLake can take a few minutes to complete.
When you configure a mapping task, the Incremental File Load section lists the Source transformations that will incrementally load files and the time that the last job completed loading the files. By default, the next job that runs checks for files modified after the last load time.
You can also override the load time that the mapping uses to look for changed files in the specified source directory. You can reset the incremental file load settings to perform a full load of all the changed files in the directory, or you can configure a time that the mapping uses to look for changed files.
A mapping in advanced mode that incrementally loads a directory that contains complex file format such as JSON fails if there are no new or changed files in the source since the last run.
