Configuring a file ingestion and replication task

1. 1Define basic task information, such as the task name, project location and runtime environment.
2. 2Configure the source.
3. 3Configure the target.
4. 4Optionally, configure one or more file-processing actions.
5. 5Optionally, set the runtime options.

Before you begin

• •Check that your organization has licenses for File Ingestion and Replication and the FMI packages.
• •The Data Ingestion and Replication application is running on the Secure Agent.
• •Source and target connections exist, based on the sources from where you want to transfer files and the targets to where you want to transfer files.

Defining basic task information

1Start the task wizard in one of the following ways:
• - On the Home page, click the Ingest panel and select File Ingestion and Replication Task.

• - In the navigation bar on the Explore page or the Home page, click New to open the New Asset dialog box. Then, select Data Ingestion and Replication > File Ingestion and Replication and click Create.

Note: If your organization does not have a custom license for application, database, file, or streaming ingestion and replication, the Data Ingestion and Replication category still appears but you cannot select and configure an ingestion and replication task of the unlicensed task type or types.
The Definition page of the file ingestion and replication task wizard appears.
2Configure the following properties:

Property	Description
Task Name	Name of the file ingestion and replication task. The names of file ingestion and replication tasks must be unique within the organization. Task names can contain alphanumeric characters, spaces, and underscores. Names must begin with an alphabetic character or underscore. Task names are not case sensitive.
Location	Project or folder in which the task will reside.
Description	Optional description of the task. Maximum length is 1024 characters.
Runtime Environment	Runtime environment that runs the task. File ingestion and replication tasks can run on a Secure Agent or Cloud Hosted Agent. They cannot run in a serverless runtime environment.

3Click Next.
To edit a file ingestion and replication task, on the Explore page, navigate to the task. In the row that contains the task, from the Actions menu, select Edit.

Configuring the source

1On the Source page, select the source type.
2Select a source connection type and a source connection.
The file ingestion and replication task uses the following source connection types:
• - Local folder
• - Advanced FTP V2
• - Advanced FTPS V2
• - Advanced SFTP V2
• - Amazon S3 V2
• - Cloud Integration Hub
• - Google Cloud Storage V2
• - Hadoop Files V2
• - Microsoft Azure Blob Storage V3
• - Microsoft Azure Data Lake Store Gen2
• - Microsoft Azure Data Lake Store V3
• - Microsoft Fabric OneLake
3Based on the source connection that you select, enter the source options.
Options that appear on the Source tab of the task wizard vary based on the type of source connection that you select.
4Click Next.
The Target tab appears.

Advanced FTP V2 source properties

Advanced FTPS V2 source properties

Advanced SFTP V2 source properties

Amazon S3 V2 source properties

Cloud Integration Hub source properties

File listener source properties

Google Cloud Storage V2 source properties

Hadoop Files V2 source properties

Local folder source properties

Microsoft Azure Blob Storage V3 source properties

Microsoft Azure Data Lake Storage Gen2 source properties

Microsoft Azure Data Lake Store Gen1 V3 source properties

Microsoft Fabric OneLake source properties

Source and target parameters

• •System variables
• •User-defined variables

Using system variables to add source and target parameters

1Click Add Parameter next to the input field, such as Source Directory or Target Directory on the Source or Target tab of the task wizard. .
The Add Parameters window appears.

2Select the required variable from the System Variablescolumn and click . The selected system variable appears on theExpression column. Repeat this procedure to select multiple system variables.
Note: When using a system variable within a task, it should be formatted as ${systemvariablename}.
The following table describes the system variables:

System Variables	Description	Expression
BadFileDir *	Directory for reject files. It cannot include the following special characters: * ? < > " | ,	${$PMBadFileDir}
CacheFileDir *	The location for the cache file.	${$PMCacheDir}
Date **	The current date in ISO (yyyy-MM-dd) format.	${system.date}
Day **	The day of week	${system.day}
ExtProcDir *	Directory for external procedures. It cannot include the following special characters: * ? < > " | ,	${$PMExtProcDir}
Hours **	Hours	${system.hours}
JobId	The id (or job number) of the current job.	${system.jobid}
LookupFileDir *	Directory for lookup files. It cannot include the following special characters: * ? < > " | ,	${$PMLookupFileDir}
Minutes **	Minutes	${system.minutes}
Month **	Numerical month	${system.month}
Name	The name of the current Project.	${system.name}
RootDir *	Root directory accessible by the node. This is the root directory for other service process variables. It cannot include the following special characters: * ? < > " | ,	${$PMRootDir}
RunId	The id when a job is run.	${system.runid}
Seconds **	Seconds	${system.seconds}
SessionLogDir *	Directory for session logs. It cannot include the following special characters: * ? < > " | ,	${$PMSessionLogDir}
SourceFileDir *	Directory for source files. It cannot include the following special characters: * ? < > " | ,	${$PMSourceFileDir}
StorageDir *	Directory for run-time files. Workflow recovery files save to the $PMStorageDir configured in the PowerCenter Integration Service properties. Session recovery files save to the $PMStorageDir configured in the operating system profile. It cannot include the following special characters: * ? < > " | ,	${$PMStorageDir}
TargetFileDir *	Directory for target files. It cannot include the following special characters: * ? < > " | ,	${$PMTargetFileDir}
TempDir *	Directory for temporary files. It cannot include the following special characters: * ? < > " | ,	${$PMTempDir}
Timestamp **	The current date and time in ISO (yyyy-MM-dd HH:mm:ss) format.	${system.timestamp}
WorkflowLogDir *	The location for the workflow log file.	${$PMWorkflowLogDir}
Year **	Year	${system.year}
* Values are fetched from the Data Integration Server. ** Time zone is the Secure Agent time zone.

3Click OK.
The expression appears in the input field.

Using user-defined variables to add source and target parameters

1Click an input field, such as Source Directory or Target Directory on the Source or Target tab of the task wizard and enter the variable. The variable should be formatted as ${systemvariablename}.

2Click OK.
The expression appears in the input field.

Skip duplicate files

Informatica Intelligent Cloud Services (IICS)

The information about skipped duplicate files that is saved in IICS is retained until the file ingestion job is purged. The retention time is based on the Job Log Service (JLS) purge policy. When this information is purged, the file ingestion task transfers the files, including the duplicate files.

To transfer the information about skipped duplicate files from the Secure Agent to IICS, set the agent-dedup-repository property to false in your Secure Agent. The Secure Agent deletes the information about skipped duplicate files after successfully transferring it to IICS. You can configure the agent-dedup-repository property in Administrator. For more information, see the Getting Started guide.

By default, the information about skipped duplicate files is saved in Informatica Intelligent Cloud Services.

Secure Agent

You can save the information about skipped duplicate files in the Secure Agent. The information is retained indefinitely and is subject to disk space availability in the Secure Agent. Currently, there is no retention policy applied on this information in the Secure Agent.

For more information about the log retention policy, see the Knowledge Base article 000209817.

The folder in which this information is stored can be configured and can be shared between agents in a Secure Agent group. You can configure the agent-dedup-repository property in Administrator. For more information, see the Getting Started guide.

To transfer the skipped duplicate files information from IICS to the Secure Agent, set the agent-dedup-repository property to true in your Secure Agent. You can configure the path to store the information about skipped duplicate files in the Secure Agent using the mi-dedup-snapshot-dir property in Administrator. For more information, see the Getting Started guide.

The file ingestion task transfers the information about skipped duplicate files from IICS to the Secure Agent when you run the subsequent file ingestion job. The skipped duplicate files information is deleted from IICS after successfully transferring the information to the Secure Agent.

Configuring the target

1On the Target page, select a connection type.
The file ingestion and replication task supports the following target connection types:
• - Local folder
• - Advanced FTP V2
• - Advanced FTPS V2
• - Advanced SFTP V2
• - Amazon S3 V2
• - Amazon Redshift V2
• - Cloud Integration Hub
• - Databricks
• - Google BigQuery V2
• - Google Cloud Storage V2
• - Hadoop Files V2
• - Microsoft Azure Blob Storage V3
• - Microsoft Azure Data Lake Store Gen2
• - Microsoft Azure Data Lake Store V3
• - Microsoft Azure Synapse SQL
• - Microsoft Fabric OneLake
• - Snowflake Data Cloud
2Select a connection.
3Based on the target connection that you select, enter the target options.
Options that appear on the Target tab of the task wizard vary based on the type of target connection that you select.
4Click Next.
The Schedule tab appears.

Advanced FTP V2 target properties

Advanced FTPS V2 target properties

Advanced SFTP V2 target properties

Amazon Redshift V2 target properties

• •Define Redshift Copy Command Properties. Select this option to define the Amazon Redshift copy command properties.
• •Enter Custom Redshift Copy Command. Select this option to provide a custom Amazon Redshift copy command that the file ingestion and replication task uses.

Amazon S3 V2 target properties

Cloud Integration Hub target properties

Databricks target properties

Google BigQuery V2 target properties

Google Cloud Storage V2 target properties

Hadoop Files V2 target properties

Local folder target properties

Microsoft Azure Blob Storage V3 target properties

Microsoft Azure Data Lake Storage Gen2 target properties

Microsoft Azure Data Lake Store Gen1 V3 target properties

Microsoft Azure Synapse SQL target properties

Microsoft Fabric OneLake target properties

Snowflake Data Cloud target properties

File format and copy options

• •RECORD_DELIMITER = '<character>' | NONE. Single character string that separates records in an input file.
• •FIELD_DELIMITER = '<character>' | NONE. Specifies the single character string that separates records in an input file.
• •FILE_EXTENSION = '<string>' | NONE. Specifies the extension for files unloaded to a stage.
• •SKIP_HEADER = <integer>. Number of lines at the start of the file to skip.
• •DATE_FORMAT = '<string>' | AUTO. Defines the format of date values in the data files or table.
• •TIME_FORMAT = '<string>' | AUTO. Defines the format of time values in the data files or table.
• •TIMESTAMP_FORMAT = <string>' | AUTO. Defines the format of timestamp values in the data files or table.

Example of File format and copy options for loading files to Snowflake

• •Delimit the fields using the pipe character ( |).
• •Files include a single header line that will be skipped.

• •For JSON: on_error='ABORT_STATEMENT'&&file_format = (type = json)&&MATCH_BY_COLUMN_NAME=CASE_INSENSITIVE
• •For AVRO: on_error='ABORT_STATEMENT'&&file_format = (type = avro)&&MATCH_BY_COLUMN_NAME=CASE_INSENSITIVE
• •For ORC: on_error='ABORT_STATEMENT'&&file_format = (type = orc)&&MATCH_BY_COLUMN_NAME=CASE_INSENSITIVE
• •For PARQUET: on_error='ABORT_STATEMENT'&&file_format = (type = parquet)&&MATCH_BY_COLUMN_NAME=CASE_INSENSITIVE

• •The column represented in the data must have the same name as the column in the table. The column names are either case-sensitive (CASE_SENSITIVE ) or case-insensitive (CASE_INSENSITIVE).
• •The column can be in any order.
• •The column in the table must have a data type that is compatible with the values in the column represented in the data. For example, string, number, and Boolean values can be loaded into a variant column.

Rules and guidelines for loading JSON files

For example, see the following data view in a table with variant column:

External stage

Rules and Guidelines for Snowflake Data Cloud file ingestion and replication tasks

Configuring file processing actions

1To add file processing actions, click the plus sign in the Actions tab.
The Action Details window appears.
2Perform the following file processing actions:

Action	Description
Compress	To compress the files, select Compress. Then select one of the following action types: - Zip - Gzip - Tar - Bzip2 You can choose to protect a .zip file by entering a password. If the .zip file contains multiple files, the same password applies to all the compressed files.
Decompress	To decompress compressed files, select Decompress. Then select one of the following action types: - Unzip - Gunzip - Untar - Bunzip2 Use the action type that corresponds to the compression action type that was used to compress the file. For example, for a .zip file, use the Unzip method. Enter the correct password to decompress a password-protected .zip file. A job fails if you decompress zipped files with incorrect passwords. For example: If you have five files and four of them are protected with the same password, but one file has a different password, then the job fails to run when it tries to decompress the file with the different password. When you open compressed files that don't use a password, file ingestion and replication ignores the password that you entered in the action. For example: If you have five files and only four of them are password protected, then the job runs successfully and decompresses all the five files.
Encrypt	To encrypt files by using the PGP encryption method, select Encrypt. Then, select PGP and enter the key ID of the user who decrypts the file. - To add your sign key, select Sign. The key ID and the key passphrase are enabled. - Enter the file suffix. Default is .pgp. You can override the default value. - Enter your private key ID and key passphrase. Do not include spaces in key passphrases. Note: For more information about securing files that file ingestion and replication transfers, see File Ingestion and Replication security.
Decrypt	To decrypt PGP-encrypted files, select Decrypt. Then, select PGP and enter the key passphrase of the user of the target directory. Do not include spaces in key passphrases.
File Operations	To perform operations on the files in the target directory, select File Operations. Then select one of the following action types: - Flatten. To move files from multiple folders to a single folder in the target directory. - Rename Files. To rename files in the target directory. If you choose to rename files as the action, enter a variable to suffix the renamed file.
Virus scan	To scan files for viruses by using the ICAP protocol, select Virus Scan. Then select ICAP and enter the ICAP Server URL or the server where the files are scanned. ICAP sends a response code that indicates whether the malwares are detected in the files. Note: Use the ICAP server of the organization.

3Click Save.
To add another action, click the plus sign. To delete an action, click Delete. To change the order in which the file ingestion and replication task processes files, drag and drop the sequence of actions.
4Click Next. The Runtime Options tab appears.

Configuring runtime options

1On the Runtime Options page, under Schedule Details, select one of the following options:
• - Do not run this task on a schedule. The task does not run according to a defined schedule. You can run the task manually.
• - Run this task on a schedule. Select a schedule by which the task runs.
• - Run this task by file listener. The task runs when the file listener notifies the task of a file event. A file event occurs when files arrive to the monitored folder or the files in the monitored folder are updated or deleted.
You must create a file listener that listens to the folder where files arrive. For more information about creating a file listener, see the section "File listeners" in the Components help.
2Under Failure Management, select the Send a notification on failure option to receive notifications if the task fails and if the task detects infected files. Enter a comma-separated list of email addresses to which to send the notifications.
If the file ingestion and replication task detects an infected file, it copies the file from the source to the quarantine directory. The default directory path is <agent location>/data/quarantine.
3Under Advanced Options, select Allow concurrency to run multiple file ingestion and replication task jobs concurrently.
Warning: Running concurrent jobs might cause unexpected results if the targets include duplicate files.
4Select the number of file batches to run in parallel. Default is 1.
Note: For database connectors that use the Truncate Target Table option, the Parallel Batch value must be 1.
5Select the log level to determine the level of detail in the logs that the job creates. Select one of the following options:
• - Normal. Logs project-level information, such as the name of the user that ran the project, when the project started, any variables that were passed, and when the project stopped. It also logs any errors encountered.
• - Silent. Additionally logs the start and stop times of tasks.
• - Verbose. Additionally logs task-level details, such as the names of the files that were processed.
• - Debug. Additionally logs detailed debugging information, such as message responses from servers.
The default value is Normal.
6Click Save.