A FileName field is a string field that contains the source path of a file. The default precision for a FileName field is 255 characters for a flat file and 1024 characters for a complex file.
You cannot configure the FileName field. You can delete the FileName field if you do not want to read or write the data in the FileName field. You cannot create a folder name with more than 255 characters for a flat file and 1024 characters for a complex file.
FileName is a reserved keyword. Avoid using FileName as the column name in the source data. The name is case sensitive.
The FileName field is applicable to the following file formats:
•Flat file
•Avro
•Parquet
•ORC
Reading source objects path
When you import source objects, the Secure Agent appends a FileName field to the imported source object. The FileName field stores the absolute path of the source file from which the Secure Agent reads the data at run time.
For example, a directory contains a number of files and each file contains multiple records that you want to read. You select the directory as source type in the Microsoft Azure Data Lake Storage Gen2 source advanced properties. When you run the mapping, the Secure Agent reads each record and stores the absolute path of the respective source file in the FileName field.
When you use the FileName field in a source object, the Secure Agent reads file names differently for mappings in advanced mode.
When you import target objects, the Secure Agent appends a FileName field to the imported target object. When you map the FileName field in the target object to an incoming field, the Secure Agent creates the folder structure and the target files based on the FileName field.
The following table describes how the Secure Agent reads file names for mappings:
Format type
Syntax
Example
Complex file
<directory>/<target_file_name>/<target_file_name>=<values of filename field>/part_file
<directory>/<target_file_name>/<target_file_name>=<values of filename field>/part_file
This syntax is applicable only to mappings.
If you create a mapping in advanced mode, the Secure Agent does not create a directory structure.
csv/customer.csv/customer.csv=1
The following table describes the syntax and example for the FileName field scenarios in mappings:
Description
Syntax
Example
When there is no target, a new target file is created.
<target_file_name>=<source_file_name>
customer_tgt.csv=customer_src.csv
When the FileName field of an source is mapped to the FileName field of an existing target, a new target file is created and the existing target is not affected.
<target_file_name>=<source_field_value>
customer_tgt.csv=customer_src.csv
When a source field other than the FileName field is mapped to the FileName field of an existing target, separate files are created for each unique value of the source field.
In the example, n number of files are inserted into the directory where the target file is present. Where, n equals the number of unique values of the source field.
Rules and guidelines for FileName field
Consider the following guidelines when you use the FileName field in mappings:
•FileName is a reserved keyword. Avoid using FileName as the column name in the source data. The name is case sensitive.
•Do not map the source object FileName field to the target object FileName field for a complex file. If you map the FileName field in the target object to an incoming field, the Secure Agent does not create directory structure as expected.
• When you use the FileName field in a target object, the Secure Agent creates folders with different names for null values for mappings in advanced mode:
- For mappings, the target file name is appended with _EMPTY_.
- For mappings in advanced mode, the target file name is appended with _HIVE_DEFAULT_PARTITION_
•When you map a date type incoming field to the FileName field in the target object, the Secure Agent creates a nested folder structure based on the incoming date value for target objects.
•When you map an incoming field to the FileName field in the target object and the target has the Handle Special Characters option enabled or the target file name has special characters, the mapping fails with the following error on a Windows machine:
[ERROR] java.io.IOException: The filename, directory name, or volume label syntax is incorrect
• When you map an incoming field to the FileName field in the target object, the mapping runs successfully for the first time. At subsequent runs, the mapping fails with the following error:
Object not found.
To successfully rerun the mapping, use a dummy target file at design time and override the dummy target file in advanced target properties.
• When you create a target at runtime, the target file name is not generated in the expected format.
This issue occurs if the FileName field is enabled for the target object. To resolve this issue, exclude the FileName field from the incoming fields in target.
•When you write data to an existing target object or create a new target object with a partition directory, the FileName field is not added on the target side. The FileName field is only present at the source side.
To read the FileName field data from the source, use the Expression transformation to rename the FileName field to a different name to avoid any validation failures.