Application Ingestion and Replication > Application Ingestion and Replication > Configuring an application ingestion and replication task
  

Configuring an application ingestion and replication task

In Data Integration, use the application ingestion and replication task wizard to configure an application ingestion and replication task.
You can integrate initial load and incremental load tasks that have Snowflake targets with Data Integration taskflows. For more information, see Integrating Application Ingestion and Replication tasks with Data Integration taskflows.
To configure application ingestion and replication tasks, click the Ingest panel on the Home page and then complete the following configuration tasks in the task configuration wizard:
  1. 1Choose a runtime environment, if you haven't set up a default runtime environment.
  2. 2Select a destination connection, or configure a new connection.
  3. 3Select a source connection, or configure a new connection.
  4. 4Specify task details for the source and target.
  5. 5Configure trim transformations (optional).
  6. 6Finalize the task definition by entering a task name, task definition location, runtime environment, and some optional properties. Then save the task..
Click Next or Back to navigate from one page to another. At any point, you can click Save to save the information that you have entered so far under a generated task name to the Default location in Explore. When you finalize the task definition, you can enter a custom task name and location.
After you complete all wizard pages, save the task definition. You can then click Deploy to make the task available as an executable job instance to the Secure Agent.

Before you begin

Before you configure an application ingestion and replication task, complete the following prerequisite tasks in Administrator:

Starting the ingestion and replication task wizard

If the latest task wizard is enabled for your organization, you can start the wizard from the Home page.
Start the wizard in one of the following ways:
Note: File Ingestion and Replication and Streaming Ingestion and Replication use the pre-existing wizard.

Primary cloud data warehouse setup

From the Data Integration Home page, you can configure the primary cloud data warehouse destination where you normally load data.
When you do this, the application ingestion and replication tasks and database ingestion and replication tasks that you create in the new wizard are automatically configured to load data to this destination. You can still change the destination if you need to.
The cloud data warehouse that you choose applies to the organization that you're currently logged into. If you have access to multiple organizations, you can configure a different primary cloud data warehouse for each organization and sub-organization.
The setup steps vary based on whether you've already configured a primary cloud data warehouse. If you've already configured one, you can change or deselect it.

Configuring a primary cloud data warehouse

Configure a primary cloud data warehouse from the Home page.
    1On the Home page, click Yes, let's go in the Do you use a cloud data warehouse as your primary destination? panel.
    2On the Destination page, select your cloud data warehouse type, for example, Snowflake Data Cloud or Databricks Delta, and click Next.
    3On the Connect page, select a connection, or click New and enter the connection properties.
    4Click Connect.

Changing or unselecting a primary cloud data warehouse

If you’ve already configured a primary cloud data warehouse, you can change or unselect it. Change or unselect a primary cloud data warehouse from the Home page.
    1On the Home page, click the cloud data warehouse type in the upper right corner and select Change primary cloud data warehouse.
    2If you want to change your primary cloud data warehouse, select I have a primary cloud data warehouse.
    3To change the cloud data warehouse type, complete the following steps:
    1. aClick Change next to Type.
    2. bOn the Destination page, select the data warehouse type, and then click Next.
    3. cOn the Connect page, select a connection, or click New and enter the connection properties.
    4. dClick Connect.
    4To change the connection, complete the following steps:
    1. aClick Change next to Connection.
    2. bOn the Connect page, select a connection, or click New and enter the connection properties.
    3. cClick Connect.
    5If you no longer wish to use a primary cloud data warehouse, select I don’t have a primary cloud data warehouse, and click Save.

Choose a runtime environment

The first thing you must do after starting the task wizard is to select the runtime environment to use for retrieving the source and target metadata required to define the task. If you previously set a default runtime environment, this step is skipped.
Note: A runtime environment must have previously been configured with one or more Secure Agents in Administrator.
    1In the Choose Runtime Environment dialog box, select the runtime environment you want to use.
    Select a runtime environment that was previously defined in Administrator from the drop-down list.
    Select Set as default if you want to use this runtime environment as the default environment for all tasks you create. Otherwise, leave the check box cleared.
    2Click OK.
    Note: When you finalize the task definition on the Let's Go page, you'll be prompted to enter the runtime environment for running the task. You can use this same runtime environment or select another one.

Configure the destination connection

On the Destination page, select an existing destination connection or add a new one.
This page displays boxes for destination connections that you previously defined from the task wizard or from Administrator.
Note: To add a new connection from the Destination page of the new wizard, you must have previously created at least one connection in Administrator.
Perform one of the following actions:
To manage your connections, go to Administrator.
Tip: As you proceed through the wizard, you can click Save to save your task entries under the generated task name at the top of the page to the Default location. On the last page of the wizard, you'll be able to enter a custom name and location for the task.

Configure the source connection

On the Source page, select an existing source connection or add a new one.
Note: To add a new connection from the Source page, you must have previously created at least one connection in Administrator.
Perform one of the following actions:

Task details: Configure how to replicate data from the source

In Step 1 of Task Details, configure the data source.
Under Source Properties, set the required basic source properties. Under Source Objects or Source Tables, select the source objects or tables from which to replicate data. Then under Advanced Source Properties, set optional advanced source properties as needed. See the property descriptions for your source type:

Configure an Adobe Analytics source

Define source properties for the source that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Type of load operation that you want the application ingestion and replication task to perform. You can select one of the following load types for the task:
  • - Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
  • - Incremental Load: Propagates source data changes to a target continuously or until the job is stopped or ends. The job propagates the changes that have occurred since the last time the job ran or from a specific start point for the first job run.
  • - Initial and Incremental Load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
Path to Report Configuration File
The path to the JSON file that contains the report configurations.
Click Next to proceed to Step 2 of Task Details.

Configure a Google Analytics source

Define source properties for the source that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Type of load operation that you want the application ingestion and replication task to perform. You can select one of the following load types for the task:
  • - Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
  • - Incremental Load: Propagates source data changes to a target continuously or until the job is stopped or ends. The job propagates the changes that have occurred since the last time the job ran or from a specific start point for the first job run.
  • - Initial and Incremental Load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
Account ID
Enter the unique identifier of your Google Analytics service account.
Property ID
Enter the unique identifier of the property whose data you want to replicate.
View ID
Enter the unique identifier of the view whose data you want to replicate.
Path to Report Configuration File
The path to the JSON file that contains the report configurations.
Click Next to proceed to Step 2 of Task Details.

Configure a Marketo source

Define source properties for the source that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Type of load operation that you want the application ingestion and replication task to perform. You can select one of the following load types for the task:
  • - Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
  • - Incremental Load: Propagates source data changes to a target continuously or until the job is stopped or ends. The job propagates the changes that have occurred since the last time the job ran or from a specific start point for the first job run.
  • - Initial and Incremental Load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
Click Next to proceed to Step 2 of Task Details.

Configure a Microsoft Dynamics 365 source

Define source properties for the source that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Type of load operation that you want the application ingestion and replication task to perform. You can select one of the following load types for the task:
  • - Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
  • - Incremental Load: Propagates source data changes to a target continuously or until the job is stopped or ends. The job propagates the changes that have occurred since the last time the job ran or from a specific start point for the first job run.
  • - Initial and Incremental Load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
Click Next to proceed to Step 2 of Task Details.

Configure a NetSuite source

Define source properties for the source that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Type of load operation that you want the application ingestion and replication task to perform. You can select one of the following load types for the task:
  • - Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
  • - Incremental Load: Propagates source data changes to a target continuously or until the job is stopped or ends. The job propagates the changes that have occurred since the last time the job ran or from a specific start point for the first job run.
  • - Initial and Incremental Load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
Click Next to proceed to Step 2 of Task Details.

Configure an Oracle Fusion Cloud source

Define source properties for the source that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Type of load operation that you want the application ingestion and replication task to perform. You can select one of the following load types for the task:
  • - Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
  • - Incremental Load: Propagates source data changes to a target continuously or until the job is stopped or ends. The job propagates the changes that have occurred since the last time the job ran or from a specific start point for the first job run.
  • - Initial and Incremental Load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
Oracle Fusion Replication Approach
Select one of the following replication approaches
  • - Select REST to extract data from various applications of Oracle Fusion such as ERP, SCM, HCM, Sales, and Services, and transfer data to the target.
  • - Select BICC (Business Intelligence Cloud Connector) to extract bulk data from the source to the target.
Oracle Fusion Application
Select the application from which you want to replicate data.
Click Next to proceed to Step 2 of Task Details.

Configure a Salesforce source

Define source properties for the source that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Type of load operation that you want the application ingestion and replication task to perform. You can select one of the following load types for the task:
  • - Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
  • - Incremental Load: Propagates source data changes to a target continuously or until the job is stopped or ends. The job propagates the changes that have occurred since the last time the job ran or from a specific start point for the first job run.
  • - Initial and Incremental Load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
Salesforce API
For initial load tasks and combined initial and incremental load tasks, select the type of Salesforce API that you want to use to retrieve the source data.
Options are:
  • - Standard (REST) API: Replicates source fields of Base64 data type. Informatica recommends that you use the Bulk API 2.0 unless you want to ingest fields of Base64 data type or objects that are not supported by Bulk API 2.0 during initial loading of data. All incremental load activities use only the standard REST API.
  • - Bulk API 2.0: Excludes replication of source fields of Base64 data type. Bulk API 2.0 is the default API for initial load tasks and the initial load of the combined initial and incremental load tasks.
  • - Bulk API: Uses Bulk API 1.0 for primary-key chunking to achieve parallel processing in Salesforce that optimizes the performance and speed of initial and combined initial and incremental load jobs. Use this option to handle large-scale data from Salesforce.
Note: By default, incremental load tasks can capture and replicate change data from source fields of Base64 data type.
Click Next to proceed to Step 2 of Task Details.

Configure a Salesforce Marketing Cloud source

Define source properties for the source that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
MID
Enter the unique Member Identification code assigned to your Salesforce Marketing Cloud account.
Click Next to proceed to Step 2 of Task Details.

Configure an SAP source that uses the SAP OData V2 connector

Define source properties for the source that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Type of load operation that you want the application ingestion and replication task to perform. You can select one of the following load types for the task:
  • - Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
  • - Incremental Load: Propagates source data changes to a target continuously or until the job is stopped or ends. The job propagates the changes that have occurred since the last time the job ran or from a specific start point for the first job run.
  • - Initial and Incremental Load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
OData Service Name
Select the OData service endpoint from where you want to retrieve data.
The list contains a specific SAP service or a list of all available services on the SAP Gateway based on the service type you specified in the SAP OData V2 connection.
Click Next to proceed to Step 2 of Task Details.

Configure an SAP source that uses the SAP ODP Extractor connector

Define source properties for the SAP ECC or SAP S4/HANA source using the SAP ODP Extractor connector that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Type of load operation that you want the application ingestion and replication task to perform. You can select one of the following load types for the task:
  • - Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
  • - Incremental Load: Propagates source data changes to a target continuously or until the job is stopped or ends. The job propagates the changes that have occurred since the last time the job ran or from a specific start point for the first job run.
  • - Initial and Incremental Load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
Context
Select the context containing the source data sources that you want to replicate on the target.
SAP ODP Extractor Connector supports the following ODP providers or contexts for all load types:
Providers/Context
Source SAP System and ODPs
SAP Service Application Programming Interface (S-API)
SAP Data Sources/Extractors without Enterprise Search (ESH)
HANA
SAP HANA Information View
BW
SAP NetWeaver Business Warehouse
ABAP_CDS
ABAP Core Data Services
SAP SLT
SLT Queue
Click Next to proceed to Step 2 of Task Details.

Configure an SAP source that uses the SAP Mass Ingestion connector

Define source properties for the SAP ECC or SAP S4/HANA source using the SAP Mass Ingestion connector that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Type of load operation that you want the application ingestion and replication task to perform. You can select one of the following load types for the task:
  • - Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
  • - Incremental Load: Propagates source data changes to a target continuously or until the job is stopped or ends. The job propagates the changes that have occurred since the last time the job ran or from a specific start point for the first job run.
  • - Initial and Incremental Load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
Schema
For incremental load and combined initial and incremental load jobs, enter the underlying database schema that includes the source tables. Perform the following steps to enter the schema value:
  • - Log in to the SAP application.
  • - Browse to System > Status
  • - Check the Owner value. Enter this value in the Schema field.
Click Next to proceed to Step 2 of Task Details.

Configure a ServiceNow source

Define source properties for the source that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Type of load operation that you want the application ingestion and replication task to perform. You can select one of the following load types for the task:
  • - Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
  • - Incremental Load: Propagates source data changes to a target continuously or until the job is stopped or ends. The job propagates the changes that have occurred since the last time the job ran or from a specific start point for the first job run.
  • - Initial and Incremental Load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
Click Next to proceed to Step 2 of Task Details.

Configure a Workday source

Define source properties for the source that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Type of load operation that you want the application ingestion and replication task to perform. You can select one of the following load types for the task:
  • - Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
  • - Incremental Load: Propagates source data changes to a target continuously or until the job is stopped or ends. The job propagates the changes that have occurred since the last time the job ran or from a specific start point for the first job run.
  • - Initial and Incremental Load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
Workday API
select the type of web service that you want to use to read source data. Options are:
  • - SOAP: Uses SOAP APIs to extract Workday data.
  • - RaaS: Uses Workday Report-as-a-Service (RaaS) to extract source data from custom objects and fields through custom reports. You can use Workday RaaS only in initial load jobs.
If you choose to use the SOAP API, perform the following steps:
  1. 1From the Product list, select Human Capital Management.
  2. 2From the Services list, select the Human Capital Management (HCM) services from which you want to ingest data to your target.
  3. You can select multiple services from the Services list.
  4. 3From the Output Type list, select the format in which you want the data to be stored on the target.
  5. The ingestion jobs extract the source data in an XML structure. Based on the format that you select, the job writes the extracted data to the target as a single object in either JSON or XML format.
If you choose to use the RaaS API, perform the following steps:
  1. 1In the Number of Reports field, select the number of reports you want to extract from the source.
  2. 2If you choose to extract a single report, in the Report Name or URL field, enter the name or URL of the custom report you want to read from the source.
  3. 3If you choose to extract multiple reports, in the Report Configuration File field, enter the path to the CSV file that you created for the list of custom reports that you want to read from the source.
Click Next to proceed to Step 2 of Task Details.

Configure a Zendesk source

Define source properties for the source that you selected on the Source page.

Source Properties

Configure the basic source properties:
Property
Description
Load Type
Type of load operation that you want the application ingestion and replication task to perform. You can select one of the following load types for the task:
  • - Initial Load: Loads data read at a specific point in time from the source application to the target in a batch operation. You can perform an initial load to materialize a target to which incremental change data will be sent.
  • - Incremental Load: Propagates source data changes to a target continuously or until the job is stopped or ends. The job propagates the changes that have occurred since the last time the job ran or from a specific start point for the first job run.
  • - Initial and Incremental Load: Performs an initial load of point-in-time data to the target and then automatically switches to propagating incremental data changes made to the same source objects on a continuous basis.
Click Next to proceed to Step 2 of Task Details.

Task details: Configure how to replicate data to the target

Configure the data target in Step 2 of Task Details.
    bulletUnder Target Properties, set the required basic target properties. Then toggle on Show Advanced Options at the top of the page to set optional advanced target properties as needed. See the property descriptions for your target type:

Configure an Amazon Redshift target

Define target properties for the Amazon Redshift destination.

Target Properties

Define the following required Amazon Redshift target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source objects.
Schema
Select the target schema in which Application Ingestion and Replication creates the target tables.
Bucket
Specifies the name of the Amazon S3 bucket that stores, organizes, and controls access to the data objects that you load to Amazon Redshift.
Data Directory or Task Target Directory
Specifies the subdirectory where Application Ingestion and Replication stores output files for jobs associated with the task. This field is called Data Directory for an initial load job or Task Target Directory for an incremental load or combined initial and incremental load job.
Click Next to proceed, or click Save.

Configure an Amazon S3 target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Amazon S3 target properties:
Property
Description
Open Table Format
The Open Table format to replicate data to Amazon S3.
You can select from the following options:
  • - Apache Iceberg. Replicates data to the Amazon S3 cloud storage as Apache Iceberg tables. You can access these tables directly from Amazon S3 using the AWS Glue Catalog.
  • - None. Does not use an Open Table format to replicate data.
The default value is None.
Namespace
The name of the database in AWS Glue Catalog where you want to store and manage your Apache Iceberg tables when you use the Apache Iceberg Open Table format.
Output Format
Select the format of the output file. Options are:
  • - CSV
  • - AVRO
  • - PARQUET
The default value is CSV.
Note: Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
Warehouse Base Directory
The root directory in Amazon S3 to store the target files and tables when you use the Apache Iceberg Open Table format.
Add Headers to CSV File
If CSV is selected as the output format, select this check box to add a header with source column names to the output CSV file.
Parquet Compression Type
If the PARQUET output format is selected, you can select a compression type that is supported by Parquet. Options are:
  • - None
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Format
If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are:
  • - Avro-Flat. This Avro schema format lists all Avro fields in one record.
  • - Avro-Generic. This Avro schema format lists all columns from a source table in a single array of Avro fields.
  • - Avro-Nested. This Avro schema format organizes each type of information in a separate record.
The default value is Avro-Flat.
Avro Serialization Format
If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are:
  • - None
  • - Binary
  • - JSON
The default value is Binary.
Avro Schema Directory
If AVRO is selected as the output format, specify the local directory where Application Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern:
schemaname_tablename.txt
Note: If this directory is not specified, no Avro schema definition file is produced.
File Compression Type
Select a file compression type for output files in CSV or AVRO output format. Options are:
  • - None
  • - Deflate
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Encryption Type
Select the encryption type for the Amazon S3 files when you write the files to the target. Options are:
  • - None
  • - Client Side Encryption
  • - Client Side Encryption with KMS
  • - Server Side Encryption
  • - Server Side Encryption with KMS
The default is None, which means no encryption is used.
Avro Compression Type
If AVRO is selected as the output format, select an Avro compression type. Options are:
  • - None
  • - Bzip2
  • - Deflate
  • - Snappy
The default value is None, which means no compression is used.
Deflate Compression Level
If Deflate is selected in the Avro Compression Type field, specify a compression level from 0 to 9. The default value is 0.
Add Directory Tags
For incremental load and combined initial and incremental load tasks, select this check box to add the "dt=" prefix to the names of apply cycle directories to be compatible with the naming convention for Hive partitioning. This check box is cleared by default.
Task Target Directory
For incremental load and combined initial and incremental load tasks, the root directory for the other directories that hold output data files, schema files, and CDC cycle contents and completed files. You can use it to specify a custom root directory for the task. If you enable the Connection Directory as Parent option, you can still optionally specify a task target directory to use with the parent directory specified in the connection properties.
This field is required if the {TaskTargetDirectory} placeholder is specified in patterns for any of the following directory fields.
Data Directory
For initial load tasks, define a directory structure for the directories where Application Ingestion and Replication stores output data files and optionally stores the schema.
The default directory pattern is {TableName)_{Timestamp}.
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter a folder name or use variables to create a folder name.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the initial load job starts to transfer data to the target.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
Note: If you manually enter the directory expression, ensure that you enclose placeholders with curly brackets { }. Placeholder values are not case sensitive.
For example:
myDir1/{SchemaName}/{TableName}
myDir1/myDir2/{SchemaName}/{YYYY}/{MM}/{TableName}_{Timestamp}
myDir1/{toLower(SchemaName)}/{TableName}_{Timestamp}
For incremental load and combined initial and incremental load tasks, define a custom path to the subdirectory that contains the cdc-data data files.
The default directory pattern is {TaskTargetDirectory}/data/{TableName}/data
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter {TaskTargetDirectory} for a task-specific base directory on the target to use instead of the S3 folder path specified in the connection properties.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the CDC cycle started.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For Amazon S3 and Microsoft Azure Data Lake Storage Gen2 targets, Application Ingestion and Replication uses the directory specified in the target connection properties as the root for the data directory path when Connection Directory as Parent is selected.
For Google Cloud Storage targets, Application Ingestion and Replication uses the Bucket name that you specify in the target properties for the ingestion task.
For Microsoft Fabric OneLake targets, the parent directory is the path specified in the Lakehouse Path field in the Microsoft Fabric OneLake connection properties.
For Amazon S3 targets with Open Table format, the data directory field is not applicable. Enabling the Connection Directory as Parent includes the connection directory before the warehouse base path. If disabled, files are saved directly under the warehouse base directory.
Connection Directory as Parent
If you use the Open Table format, select this check box to use the directory value specified in the target connection properties as the parent directory. This path appends to the file path on S3 while creating the file. This check box is selected by default.
For example, if the S3 directory set in the connection is myFolderOnS3/F1 and the Warehouse Base Directory is /myFold, files are saved to myFolderOnS3/F1/myFold/<files>. However, if you do not select the Connection Directory as Parent option, files are saved directly to /myFold/<files>.
If you do not use the Open Table format, selecting this check box uses the directory value specified in the target connection properties as the parent directory for the custom directory paths specified in the task target properties. For initial load tasks, the parent directory is used in the Data Directory and Schema Directory. For incremental load and combined initial and incremental load tasks, the parent directory is used in the Data Directory, Schema Directory, Cycle Completion Directory, and Cycle Contents Directory.
This check box is selected by default. If you clear it, for initial loads, define the full path to the output files in the Data Directory field. For incremental loads, optionally specify a root directory for the task in the Task Target Directory.
Schema Directory
Specify a custom directory to store the schema file if you want to store it in a directory other than the default directory. For initial loads, previously used values if available are shown in a list for your convenience. This field is optional.
For initial loads, the schema is stored in the data directory by default. For incremental loads and combined initial and incremental loads, the default directory for the schema file is {TaskTargetDirectory}/data/{TableName}/schema
You can use the same placeholders as for the Data Directory field. If you manually enter placeholders, ensure that you enclose them with curly brackets { }. If you include the toUpper or toLower function, put the placeholder name in parentheses and enclose both the function and placeholder in curly brackets, for example: {toLower(SchemaName)}
Note: Schema is written only to output data files in CSV format. Data files in Parquet and Avro formats contain their own embedded schema.
Cycle Completion Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle completed file. Default is {TaskTargetDirectory}/cycle/completed.
Cycle Contents Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle contents files. Default is {TaskTargetDirectory}/cycle/contents.
Use Cycle Partitioning for Data Directory
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under each data directory.
If this option is not selected, individual data files are written to the same directory without a timestamp, unless you define an alternative directory structure.
Use Cycle Partitioning for Summary Directories
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under the summary contents and completed subdirectories.
List Individual Files in Contents
For incremental load and combined initial and incremental load tasks, lists individual data files under the contents subdirectory.
If Use Cycle Partitioning for Summary Directories is cleared, this option is selected by default. All of the individual files are listed in the contents subdirectory unless you can configure custom subdirectories by using the placeholders, such as for timestamp or date.
If Use Cycle Partitioning for Data Directory is selected, you can still optionally select this check box to list individual files and group them by CDC cycle.
Click Next to proceed, or click Save.

Configure a Databricks target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Databricks target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source objects.
Schema
Select the target schema in which Application Ingestion and Replication creates the target tables.
Apply Mode
For incremental load and combined initial and incremental load jobs, indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
  • - Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
  • - Soft Deletes. Apply source delete operations to the target as soft deletes. A soft delete marks the deleted row as deleted without actually removing it from the database. For example, a delete on the source results in a change record on the target with "D" displayed in the INFA_OPERATION_TYPE column.
  • After enabling Soft Deletes, any update in the source table during normal or backlog mode results in the deletion of the matching record, insertion of the updated record, and marking of the INFA_OPERATION_TYPE operation as NULL in the target table. Similarly, inserting a record in the source table during backlog mode results in marking the INFA_OPERATION_TYPE operation as E in the target table record.
    Consider using soft deletes if you have a long-running business process that needs the soft-deleted data to finish processing, to restore data after an accidental delete operation, or to track deleted values for audit purposes.
  • - Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, transaction ID, and before image. Consider using Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
  • After enabling the Audit apply mode, any update in the source table during backlog or normal mode results in marking the INFA_OPERATION_TYPE operation as E in the target table record. Similarly, inserting a record in the source table during backlog mode results in marking the INFA_OPERATION_TYPE operation as E in the target table record.
Default is Standard.
Data Directory or Task Target Directory
Specifies the subdirectory where Application Ingestion and Replication stores output files for jobs associated with the task. This field is called Data Directory for an initial load job or Task Target Directory for an incremental load or combined initial and incremental load job.
Click Next to proceed, or click Save.

Configure a Google BigQuery target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Google BigQuery target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source objects.
Schema
Select the target schema in which Application Ingestion and Replication creates the target tables.
Apply Mode
For incremental load and combined initial and incremental load jobs, indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
  • - Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
  • - Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, time, owner, transaction ID, generated ascending sequence number, and before image. Consider using Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
  • - Soft Deletes. Apply source delete operations to the target as soft deletes. A soft delete marks the deleted row as deleted without actually removing it from the database. For example, a delete on the source results in a change record on the target with "D" displayed in the INFA_OPERATION_TYPE column.
  • Consider using soft deletes if you have a long-running business process that needs the soft-deleted data to finish processing, to restore data after an accidental delete operation, or to track deleted values for audit purposes.
    Note: If you use Soft Deletes mode, you must not perform an update on the primary key in a source table. Otherwise, data corruption can occur on the target.
The default value is Standard.
Bucket
Specifies the name of an existing bucket container that stores, organizes, and controls access to the data objects that you load to Google Cloud Storage.
Data Directory or Task Target Directory
Specifies the subdirectory where Application Ingestion and Replication stores output files for jobs associated with the task. This field is called Data Directory for an initial load job or Task Target Directory for an incremental load or combined initial and incremental load job.
Click Next to proceed, or click Save.

Configure a Google Cloud Storage target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Google Cloud Storage target properties:
Property
Description
Output Format
Select the format of the output file. Options are:
  • - CSV
  • - AVRO
  • - PARQUET
The default value is CSV.
Note: Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
Add Headers to CSV File
If CSV is selected as the output format, select this check box to add a header with source column names to the output CSV file.
Parquet Compression Type
If the PARQUET output format is selected, you can select a compression type that is supported by Parquet. Options are:
  • - None
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Format
If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are:
  • - Avro-Flat. This Avro schema format lists all Avro fields in one record.
  • - Avro-Generic. This Avro schema format lists all columns from a source table in a single array of Avro fields.
  • - Avro-Nested. This Avro schema format organizes each type of information in a separate record.
The default value is Avro-Flat.
Avro Serialization Format
If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are:
  • - None
  • - Binary
  • - JSON
The default value is Binary.
Avro Schema Directory
If AVRO is selected as the output format, specify the local directory where Application Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern:
schemaname_tablename.txt
Note: If this directory is not specified, no Avro schema definition file is produced.
File Compression Type
Select a file compression type for output files in CSV or AVRO output format. Options are:
  • - None
  • - Deflate
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Compression Type
If AVRO is selected as the output format, select an Avro compression type. Options are:
  • - None
  • - Bzip2
  • - Deflate
  • - Snappy
The default value is None, which means no compression is used.
Deflate Compression Level
If Deflate is selected in the Avro Compression Type field, specify a compression level from 0 to 9. The default value is 0.
Add Directory Tags
For incremental load and combined initial and incremental load tasks, select this check box to add the "dt=" prefix to the names of apply cycle directories to be compatible with the naming convention for Hive partitioning. This check box is cleared by default.
Bucket
Specifies the name of an existing bucket container that stores, organizes, and controls access to the data objects that you load to Google Cloud Storage.
Task Target Directory
For incremental load and combined initial and incremental load tasks, the root directory for the other directories that hold output data files, schema files, and CDC cycle contents and completed files. You can use it to specify a custom root directory for the task. If you enable the Connection Directory as Parent option, you can still optionally specify a task target directory to use with the parent directory specified in the connection properties.
This field is required if the {TaskTargetDirectory} placeholder is specified in patterns for any of the following directory fields.
Data Directory
For initial load tasks, define a directory structure for the directories where Application Ingestion and Replication stores output data files and optionally stores the schema.
The default directory pattern is {TableName)_{Timestamp}.
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter a folder name or use variables to create a folder name.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the initial load job starts to transfer data to the target.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
Note: If you manually enter the directory expression, ensure that you enclose placeholders with curly brackets { }. Placeholder values are not case sensitive.
For example:
myDir1/{SchemaName}/{TableName}
myDir1/myDir2/{SchemaName}/{YYYY}/{MM}/{TableName}_{Timestamp}
myDir1/{toLower(SchemaName)}/{TableName}_{Timestamp}
For incremental load and combined initial and incremental load tasks, define a custom path to the subdirectory that contains the cdc-data data files.
The default directory pattern is {TaskTargetDirectory}/data/{TableName}/data
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter {TaskTargetDirectory} for a task-specific base directory on the target to use instead of the S3 folder path specified in the connection properties.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the CDC cycle started.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For Amazon S3 and Microsoft Azure Data Lake Storage Gen2 targets, Application Ingestion and Replication uses the directory specified in the target connection properties as the root for the data directory path when Connection Directory as Parent is selected.
For Google Cloud Storage targets, Application Ingestion and Replication uses the Bucket name that you specify in the target properties for the ingestion task.
For Microsoft Fabric OneLake targets, the parent directory is the path specified in the Lakehouse Path field in the Microsoft Fabric OneLake connection properties.
For Amazon S3 targets with Open Table format, the data directory field is not applicable. Enabling the Connection Directory as Parent includes the connection directory before the warehouse base path. If disabled, files are saved directly under the warehouse base directory.
Schema Directory
Specify a custom directory to store the schema file if you want to store it in a directory other than the default directory. For initial loads, previously used values if available are shown in a list for your convenience. This field is optional.
For initial loads, the schema is stored in the data directory by default. For incremental loads and combined initial and incremental loads, the default directory for the schema file is {TaskTargetDirectory}/data/{TableName}/schema
You can use the same placeholders as for the Data Directory field. If you manually enter placeholders, ensure that you enclose them with curly brackets { }. If you include the toUpper or toLower function, put the placeholder name in parentheses and enclose both the function and placeholder in curly brackets, for example: {toLower(SchemaName)}
Note: Schema is written only to output data files in CSV format. Data files in Parquet and Avro formats contain their own embedded schema.
Cycle Completion Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle completed file. Default is {TaskTargetDirectory}/cycle/completed.
Cycle Contents Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle contents files. Default is {TaskTargetDirectory}/cycle/contents.
Use Cycle Partitioning for Data Directory
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under each data directory.
If this option is not selected, individual data files are written to the same directory without a timestamp, unless you define an alternative directory structure.
Use Cycle Partitioning for Summary Directories
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under the summary contents and completed subdirectories.
List Individual Files in Contents
For incremental load and combined initial and incremental load tasks, lists individual data files under the contents subdirectory.
If Use Cycle Partitioning for Summary Directories is cleared, this option is selected by default. All of the individual files are listed in the contents subdirectory unless you can configure custom subdirectories by using the placeholders, such as for timestamp or date.
If Use Cycle Partitioning for Data Directory is selected, you can still optionally select this check box to list individual files and group them by CDC cycle.
Click Next to proceed, or click Save.

Configure a Kafka target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Kafka target properties:
Property
Description
Use Table Name as Topic Name
Indicates whether Application Ingestion and Replication writes messages that contain source data to separate topics, one for each source object, or writes all messages to a single topic.
Select this check box to write messages to separate table-specific topics. The topic names match the source table names, unless you add the source schema name, a prefix, or a suffix in the Include Schema Name, Table Prefix, or Table Suffix properties.
By default, this check box is cleared.
Include Schema Name
When Use Table Name as Topic Name is selected, this check box appears and is selected by default. This setting adds the source schema name in the table-specific topic names. The topic names then have the format schemaname_tablename.
If you do not want to include the schema name, clear this check box.
Table Prefix
When Use Table Name as Topic Name is selected, this property appears so that you can optionally enter a prefix to add to the table-specific topic names. For example, if you specify myprefix_, the topic names have the format myprefix_tablename. If you omit the underscore (_) after the prefix, the prefix is prepended to the table name.
Table Suffix
When Use Table Name as Topic Name is selected, this property appears so that you can optionally enter a suffix to add to the table-specific topic names. For example, if you specify _mysuffix, the topic names have the format tablename_mysuffix. If you omit the underscore (_) before the suffix, the suffix is appended to the table name.
Output Format
Select the format of the output file. Options are:
  • - CSV
  • - AVRO
  • - JSON
The default value is CSV.
Note: Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
If your Kafka target uses Confluent Schema Registry to store schemas for incremental load jobs, you must select AVRO as the format.
JSON Format
If JSON is selected as the output format, select the level of detail of the output. Options are:
  • - Concise. This format records only the most relevant data in the output, such as the operation type and the column names and values.
  • - Verbose. This format records detailed information, such as the table name and column types.
Avro Format
If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are:
  • - Avro-Flat. This Avro schema format lists all Avro fields in one record.
  • - Avro-Generic. This Avro schema format lists all columns from a source table in a single array of Avro fields.
  • - Avro-Nested. This Avro schema format organizes each type of information in a separate record.
The default value is Avro-Flat.
Avro Serialization Format
If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are:
  • - Binary
  • - JSON
  • - None
The default value is Binary.
If you have a Confluent Kafka target that uses Confluent Schema Registry to store schemas, select None. Otherwise, Confluent Schema Registry does not register the schema. Do not select None if you are not using Confluent Scheme Registry.
Avro Schema Directory
If AVRO is selected as the output format, specify the local directory where Database Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern:
schemaname_tablename.txt
Note: If this directory is not specified, no Avro schema definition file is produced.
If a source schema change is expected to alter the target, the Avro schema definition file is regenerated with a unique name that includes a timestamp, in the following format:
schemaname_tablename_YYYYMMDDhhmmss.txt
This unique naming pattern ensures that older schema definition files are preserved for audit purposes.
Avro Compression Type
If AVRO is selected as the output format, select an Avro compression type. Options are:
  • - None
  • - Bzip2
  • - Deflate
  • - Snappy
The default value is None, which means no compression is used.
Click Next to proceed, or click Save.

Configure a Microsoft Azure Data Lake Storage Gen2 target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Microsoft Azure Data Lake Storage Gen2 target properties:
Property
Description
Output Format
Select the format of the output file. Options are:
  • - CSV
  • - AVRO
  • - PARQUET
The default value is CSV.
Note: Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
Add Headers to CSV File
If CSV is selected as the output format, select this check box to add a header with source column names to the output CSV file.
Parquet Compression Type
If the PARQUET output format is selected, you can select a compression type that is supported by Parquet. Options are:
  • - None
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Format
If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are:
  • - Avro-Flat. This Avro schema format lists all Avro fields in one record.
  • - Avro-Generic. This Avro schema format lists all columns from a source table in a single array of Avro fields.
  • - Avro-Nested. This Avro schema format organizes each type of information in a separate record.
The default value is Avro-Flat.
Avro Serialization Format
If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are:
  • - None
  • - Binary
  • - JSON
The default value is Binary.
Avro Schema Directory
If AVRO is selected as the output format, specify the local directory where Application Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern:
schemaname_tablename.txt
Note: If this directory is not specified, no Avro schema definition file is produced.
File Compression Type
Select a file compression type for output files in CSV or AVRO output format. Options are:
  • - None
  • - Deflate
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Compression Type
If AVRO is selected as the output format, select an Avro compression type. Options are:
  • - None
  • - Bzip2
  • - Deflate
  • - Snappy
The default value is None, which means no compression is used.
Deflate Compression Level
If Deflate is selected in the Avro Compression Type field, specify a compression level from 0 to 9. The default value is 0.
Add Directory Tags
For incremental load and combined initial and incremental load tasks, select this check box to add the "dt=" prefix to the names of apply cycle directories to be compatible with the naming convention for Hive partitioning. This check box is cleared by default.
Task Target Directory
For incremental load and combined initial and incremental load tasks, the root directory for the other directories that hold output data files, schema files, and CDC cycle contents and completed files. You can use it to specify a custom root directory for the task. If you enable the Connection Directory as Parent option, you can still optionally specify a task target directory to use with the parent directory specified in the connection properties.
This field is required if the {TaskTargetDirectory} placeholder is specified in patterns for any of the following directory fields.
Data Directory
For initial load tasks, define a directory structure for the directories where Application Ingestion and Replication stores output data files and optionally stores the schema.
The default directory pattern is {TableName)_{Timestamp}.
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter a folder name or use variables to create a folder name.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the initial load job starts to transfer data to the target.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
Note: If you manually enter the directory expression, ensure that you enclose placeholders with curly brackets { }. Placeholder values are not case sensitive.
For example:
myDir1/{SchemaName}/{TableName}
myDir1/myDir2/{SchemaName}/{YYYY}/{MM}/{TableName}_{Timestamp}
myDir1/{toLower(SchemaName)}/{TableName}_{Timestamp}
For incremental load and combined initial and incremental load tasks, define a custom path to the subdirectory that contains the cdc-data data files.
The default directory pattern is {TaskTargetDirectory}/data/{TableName}/data
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter {TaskTargetDirectory} for a task-specific base directory on the target to use instead of the S3 folder path specified in the connection properties.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the CDC cycle started.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For Amazon S3 and Microsoft Azure Data Lake Storage Gen2 targets, Application Ingestion and Replication uses the directory specified in the target connection properties as the root for the data directory path when Connection Directory as Parent is selected.
For Google Cloud Storage targets, Application Ingestion and Replication uses the Bucket name that you specify in the target properties for the ingestion task.
For Microsoft Fabric OneLake targets, the parent directory is the path specified in the Lakehouse Path field in the Microsoft Fabric OneLake connection properties.
For Amazon S3 targets with Open Table format, the data directory field is not applicable. Enabling the Connection Directory as Parent includes the connection directory before the warehouse base path. If disabled, files are saved directly under the warehouse base directory.
Connection Directory as Parent
Select this check box to use the directory value that is specified in the target connection properties as the parent directory for the custom directory paths specified in the task target properties.
For initial load tasks, the parent directory is used in the Data Directory and Schema Directory.
For incremental load and combined initial and incremental load tasks, the parent directory is used in the Data Directory, Schema Directory, Cycle Completion Directory, and Cycle Contents Directory.
This check box is selected by default. If you clear it, for initial loads, define the full path to the output files in the Data Directory field. For incremental loads, optionally specify a root directory for the task in the Task Target Directory.
Schema Directory
Specify a custom directory to store the schema file if you want to store it in a directory other than the default directory. For initial loads, previously used values if available are shown in a list for your convenience. This field is optional.
For initial loads, the schema is stored in the data directory by default. For incremental loads and combined initial and incremental loads, the default directory for the schema file is {TaskTargetDirectory}/data/{TableName}/schema
You can use the same placeholders as for the Data Directory field. If you manually enter placeholders, ensure that you enclose them with curly brackets { }. If you include the toUpper or toLower function, put the placeholder name in parentheses and enclose both the function and placeholder in curly brackets, for example: {toLower(SchemaName)}
Note: Schema is written only to output data files in CSV format. Data files in Parquet and Avro formats contain their own embedded schema.
Cycle Completion Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle completed file. Default is {TaskTargetDirectory}/cycle/completed.
Cycle Contents Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle contents files. Default is {TaskTargetDirectory}/cycle/contents.
Use Cycle Partitioning for Data Directory
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under each data directory.
If this option is not selected, individual data files are written to the same directory without a timestamp, unless you define an alternative directory structure.
Use Cycle Partitioning for Summary Directories
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under the summary contents and completed subdirectories.
List Individual Files in Contents
For incremental load and combined initial and incremental load tasks, lists individual data files under the contents subdirectory.
If Use Cycle Partitioning for Summary Directories is cleared, this option is selected by default. All of the individual files are listed in the contents subdirectory unless you can configure custom subdirectories by using the placeholders, such as for timestamp or date.
If Use Cycle Partitioning for Data Directory is selected, you can still optionally select this check box to list individual files and group them by CDC cycle.
Click Next to proceed, or click Save.

Configure a Microsoft Azure Synapse Analytics target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Microsoft Azure Synapse Analytics target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source objects.
Schema
Select the target schema in which Application Ingestion and Replication creates the target tables. The schema name that is specified in the connection properties is displayed by default.
This field is case sensitive. Therefore, ensure that you entered the schema name in the connection properties in the correct case.
Click Next to proceed, or click Save.

Configure an Microsoft Fabric Data Warehouse target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Microsoft Fabric Data Warehouse target properties:
Property
Description
Schema Path
The schema path where the task creates the target tables.
Apply Mode
For incremental load and combined initial and incremental load jobs, indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
The default value is Standard.
Note: This field does not appear if you selected Query-based as the CDC method on the Source page of the task wizard.
Staging Lakehouse Name
The name of the staging lakehouse used as an intermediate storage area for data before it is loaded into Azure Fabric Warehouse.
Click Next to proceed, or click Save.

Configure a Microsoft Fabric OneLake target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Microsoft Fabric OneLake target properties:
Property
Description
Open Mirroring
Select this checkbox to use Microsoft Fabric open mirroring. Open mirroring creates a synchronized, near real-time copy of your source data in Microsoft Fabric OneLake. For more information, see Using open mirroring to replicate data to Microsoft Fabric OneLake.
Landing Zone URL
If you selected Open Mirroring, enter the unique Microsoft Fabric OneLake path where mirrored data files are stored in Parquet format before processing in Microsoft Fabric.
Note: The rest of the fields in this table don't apply when you enable open mirroring.
Output Format
Select the format of the output file. Options are:
  • - CSV
  • - AVRO
  • - PARQUET
The default value is CSV. Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
Note: Open mirroring uses Parquet output format by default.
Add Headers to CSV File
If CSV is selected as the output format, select this check box to add a header with source column names to the output CSV file.
Parquet Compression Type
If the PARQUET output format is selected, you can select a compression type that is supported by Parquet. Options are:
  • - None
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Format
If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are:
  • - Avro-Flat. This Avro schema format lists all Avro fields in one record.
  • - Avro-Generic. This Avro schema format lists all columns from a source table in a single array of Avro fields.
  • - Avro-Nested. This Avro schema format organizes each type of information in a separate record.
The default value is Avro-Flat.
Avro Serialization Format
If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are:
  • - None
  • - Binary
  • - JSON
The default value is Binary.
Avro Schema Directory
If AVRO is selected as the output format, specify the local directory where Application Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern:
schemaname_tablename.txt
Note: If this directory is not specified, no Avro schema definition file is produced.
File Compression Type
Select a file compression type for output files in CSV or AVRO output format. Options are:
  • - None
  • - Deflate
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Compression Type
If AVRO is selected as the output format, select an Avro compression type. Options are:
  • - None
  • - Bzip2
  • - Deflate
  • - Snappy
The default value is None, which means no compression is used.
Deflate Compression Level
If Deflate is selected in the Avro Compression Type field, specify a compression level from 0 to 9. The default value is 0.
Add Directory Tags
For incremental load and combined initial and incremental load tasks, select this check box to add the "dt=" prefix to the names of apply cycle directories to be compatible with the naming convention for Hive partitioning. This check box is cleared by default.
Task Target Directory
For incremental load and combined initial and incremental load tasks, the root directory for the other directories that hold output data files, schema files, and CDC cycle contents and completed files. You can use it to specify a custom root directory for the task.
This field is required if the {TaskTargetDirectory} placeholder is specified in patterns for any of the following directory fields.
Data Directory
For initial load tasks, define a directory structure for the directories where Application Ingestion and Replication stores output data files and optionally stores the schema.
The default directory pattern is {TableName)_{Timestamp}.
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter a folder name or use variables to create a folder name.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the initial load job starts to transfer data to the target.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
Note: If you manually enter the directory expression, ensure that you enclose placeholders with curly brackets { }. Placeholder values are not case sensitive.
For example:
myDir1/{SchemaName}/{TableName}
myDir1/myDir2/{SchemaName}/{YYYY}/{MM}/{TableName}_{Timestamp}
myDir1/{toLower(SchemaName)}/{TableName}_{Timestamp}
For incremental load and combined initial and incremental load tasks, define a custom path to the subdirectory that contains the cdc-data data files.
The default directory pattern is {TaskTargetDirectory}/data/{TableName}/data
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter {TaskTargetDirectory} for a task-specific base directory on the target to use instead of the S3 folder path specified in the connection properties.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the CDC cycle started.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For Amazon S3 and Microsoft Azure Data Lake Storage Gen2 targets, Application Ingestion and Replication uses the directory specified in the target connection properties as the root for the data directory path when Connection Directory as Parent is selected.
For Google Cloud Storage targets, Application Ingestion and Replication uses the Bucket name that you specify in the target properties for the ingestion task.
For Microsoft Fabric OneLake targets, the parent directory is the path specified in the Lakehouse Path field in the Microsoft Fabric OneLake connection properties.
For Amazon S3 targets with Open Table format, the data directory field is not applicable. Enabling the Connection Directory as Parent includes the connection directory before the warehouse base path. If disabled, files are saved directly under the warehouse base directory.
Schema Directory
Specify a custom directory to store the schema file if you want to store it in a directory other than the default directory. For initial loads, previously used values if available are shown in a list for your convenience. This field is optional.
For initial loads, the schema is stored in the data directory by default. For incremental loads and combined initial and incremental loads, the default directory for the schema file is {TaskTargetDirectory}/data/{TableName}/schema
You can use the same placeholders as for the Data Directory field. If you manually enter placeholders, ensure that you enclose them with curly brackets { }. If you include the toUpper or toLower function, put the placeholder name in parentheses and enclose both the function and placeholder in curly brackets, for example: {toLower(SchemaName)}
Note: Schema is written only to output data files in CSV format. Data files in Parquet and Avro formats contain their own embedded schema.
Cycle Completion Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle completed file. Default is {TaskTargetDirectory}/cycle/completed.
Cycle Contents Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle contents files. Default is {TaskTargetDirectory}/cycle/contents.
Use Cycle Partitioning for Data Directory
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under each data directory.
If this option is not selected, individual data files are written to the same directory without a timestamp, unless you define an alternative directory structure.
Use Cycle Partitioning for Summary Directories
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under the summary contents and completed subdirectories.
List Individual Files in Contents
For incremental load and combined initial and incremental load tasks, lists individual data files under the contents subdirectory.
If Use Cycle Partitioning for Summary Directories is cleared, this option is selected by default. All of the individual files are listed in the contents subdirectory unless you can configure custom subdirectories by using the placeholders, such as for timestamp or date.
If Use Cycle Partitioning for Data Directory is selected, you can still optionally select this check box to list individual files and group them by CDC cycle.
Click Next to proceed, or click Save.

Configure a Microsoft SQL Server target

Define Microsoft SQL Server target properties.

Target Properties

Define the following required Microsoft SQL Server target properties:
Property
Description
Target Creation
The Create Target Tables option generates the target tables based on the source tables.
Note: After the target table is created, Application Ingestion and Replication intelligently handles the target tables on subsequent job runs. Application Ingestion and Replication might truncate or re-create the target tables depending on specific circumstances.
Schema
Select the target schema in which Application Ingestion and Replication creates the target tables. The schema name that is specified in the connection properties is displayed by default.
This field is case sensitive. Therefore, ensure that you entered the schema name in the connection properties in the correct case.
Click Next to proceed, or click Save.

Configure an Open Table target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Open Table target properties:
Property
Description
Schema Path
The schema path where the task creates the target tables.
Catalog Name with Database Name
The catalog and database name where the task organizes and stores metadata for the target tables.
Enter the values in the following format: <catalogname>/<databasename>
Table Location
The storage path in Amazon S3 where the task creates the Open Table data files.
For example: s3://s3format/cdc
Table Properties
Configuration settings you can define for the Open Table target.
Enter the values as comma-separated key-value pairs in the following format: 'k1'='v1','k2'='v2'.
Include the required property: 'format'='parquet'
Table Type
The table type that defines how the Open Table format is stored and managed.
Enter the following value: iceberg
Apply Mode
For incremental load and combined initial and incremental load jobs, indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
The default value is Standard.
Note: This field does not appear if you selected Query-based as the CDC method on the Source page of the task wizard.
Enable Case Transformation
By default, target table names and column names are generated in the same case as the corresponding source names, unless cluster-level or session-level properties on the target override this case-sensitive behavior. If you want to control the case of letters in the target names, select this check box.
Click Next to proceed, or click Save.

Configure an Oracle Cloud Object Storage target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Oracle Cloud Object Storage target properties:
Property
Description
Output Format
Select the format of the output file. Options are:
  • - CSV
  • - AVRO
  • - PARQUET
The default value is CSV.
Note: Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
Add Headers to CSV File
If CSV is selected as the output format, select this check box to add a header with source column names to the output CSV file.
Parquet Compression Type
If the PARQUET output format is selected, you can select a compression type that is supported by Parquet. Options are:
  • - None
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Format
If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are:
  • - Avro-Flat. This Avro schema format lists all Avro fields in one record.
  • - Avro-Generic. This Avro schema format lists all columns from a source table in a single array of Avro fields.
  • - Avro-Nested. This Avro schema format organizes each type of information in a separate record.
The default value is Avro-Flat.
Avro Serialization Format
If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are:
  • - None
  • - Binary
  • - JSON
The default value is Binary.
Avro Schema Directory
If AVRO is selected as the output format, specify the local directory where Application Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern:
schemaname_tablename.txt
Note: If this directory is not specified, no Avro schema definition file is produced.
File Compression Type
Select a file compression type for output files in CSV or AVRO output format. Options are:
  • - None
  • - Deflate
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Compression Type
If AVRO is selected as the output format, select an Avro compression type. Options are:
  • - None
  • - Bzip2
  • - Deflate
  • - Snappy
The default value is None, which means no compression is used.
Deflate Compression Level
If Deflate is selected in the Avro Compression Type field, specify a compression level from 0 to 9. The default value is 0.
Add Directory Tags
For incremental load and combined initial and incremental load tasks, select this check box to add the "dt=" prefix to the names of apply cycle directories to be compatible with the naming convention for Hive partitioning. This check box is cleared by default.
Task Target Directory
For incremental load and combined initial and incremental load tasks, the root directory for the other directories that hold output data files, schema files, and CDC cycle contents and completed files. You can use it to specify a custom root directory for the task. If you enable the Connection Directory as Parent option, you can still optionally specify a task target directory to use with the parent directory specified in the connection properties.
This field is required if the {TaskTargetDirectory} placeholder is specified in patterns for any of the following directory fields.
Connection Directory as Parent
Select this check box to use the directory value that is specified in the target connection properties as the parent directory for the custom directory paths specified in the task target properties.
For initial load tasks, the parent directory is used in the Data Directory and Schema Directory.
For incremental load and combined initial and incremental load tasks, the parent directory is used in the Data Directory, Schema Directory, Cycle Completion Directory, and Cycle Contents Directory.
This check box is selected by default. If you clear it, for initial loads, define the full path to the output files in the Data Directory field. For incremental loads, optionally specify a root directory for the task in the Task Target Directory.
Data Directory
For initial load tasks, define a directory structure for the directories where Application Ingestion and Replication stores output data files and optionally stores the schema.
The default directory pattern is {TableName)_{Timestamp}.
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter a folder name or use variables to create a folder name.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the initial load job starts to transfer data to the target.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
Note: If you manually enter the directory expression, ensure that you enclose placeholders with curly brackets { }. Placeholder values are not case sensitive.
For example:
myDir1/{SchemaName}/{TableName}
myDir1/myDir2/{SchemaName}/{YYYY}/{MM}/{TableName}_{Timestamp}
myDir1/{toLower(SchemaName)}/{TableName}_{Timestamp}
For incremental load and combined initial and incremental load tasks, define a custom path to the subdirectory that contains the cdc-data data files.
The default directory pattern is {TaskTargetDirectory}/data/{TableName}/data
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter {TaskTargetDirectory} for a task-specific base directory on the target to use instead of the S3 folder path specified in the connection properties.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the CDC cycle started.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For Amazon S3 and Microsoft Azure Data Lake Storage Gen2 targets, Application Ingestion and Replication uses the directory specified in the target connection properties as the root for the data directory path when Connection Directory as Parent is selected.
For Google Cloud Storage targets, Application Ingestion and Replication uses the Bucket name that you specify in the target properties for the ingestion task.
For Microsoft Fabric OneLake targets, the parent directory is the path specified in the Lakehouse Path field in the Microsoft Fabric OneLake connection properties.
For Amazon S3 targets with Open Table format, the data directory field is not applicable. Enabling the Connection Directory as Parent includes the connection directory before the warehouse base path. If disabled, files are saved directly under the warehouse base directory.
Schema Directory
Specify a custom directory to store the schema file if you want to store it in a directory other than the default directory. For initial loads, previously used values if available are shown in a list for your convenience. This field is optional.
For initial loads, the schema is stored in the data directory by default. For incremental loads and combined initial and incremental loads, the default directory for the schema file is {TaskTargetDirectory}/data/{TableName}/schema
You can use the same placeholders as for the Data Directory field. If you manually enter placeholders, ensure that you enclose them with curly brackets { }. If you include the toUpper or toLower function, put the placeholder name in parentheses and enclose both the function and placeholder in curly brackets, for example: {toLower(SchemaName)}
Note: Schema is written only to output data files in CSV format. Data files in Parquet and Avro formats contain their own embedded schema.
Cycle Completion Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle completed file. Default is {TaskTargetDirectory}/cycle/completed.
Cycle Contents Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle contents files. Default is {TaskTargetDirectory}/cycle/contents.
Use Cycle Partitioning for Data Directory
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under each data directory.
If this option is not selected, individual data files are written to the same directory without a timestamp, unless you define an alternative directory structure.
Use Cycle Partitioning for Summary Directories
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under the summary contents and completed subdirectories.
List Individual Files in Contents
For incremental load and combined initial and incremental load tasks, lists individual data files under the contents subdirectory.
If Use Cycle Partitioning for Summary Directories is cleared, this option is selected by default. All of the individual files are listed in the contents subdirectory unless you can configure custom subdirectories by using the placeholders, such as for timestamp or date.
If Use Cycle Partitioning for Data Directory is selected, you can still optionally select this check box to list individual files and group them by CDC cycle.
Click Next to proceed, or click Save.

Configure an Oracle target

Define target properties for the Oracle destination.

Target Properties

Define the following required Oracle target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source objects.
Schema
Select the target schema in which Application Ingestion and Replication creates the target tables.
Apply Mode
For incremental load and combined initial and incremental load jobs, indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
  • - Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
  • - Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, time, owner, transaction ID, generated ascending sequence number, and before image. Consider using Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
Note: The Audit apply mode applies for an SAP source with SAP Mass Ingestion connector.
The default value is Standard.
Click Next to proceed, or click Save.

Configure a PostgreSQL target

Define target properties for the PostgreSQL destination.

Target Properties

Define the following required PostgreSQL target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source objects.
Schema
Select the target schema in which Application Ingestion and Replication creates the target tables.
Click Next to proceed, or click Save.

Configure a Salesforce Data 360 target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Salesforce Data 360 target properties:
Property
Description
Schema Path
The schema path where the task creates the target tables.
Data Space
Name of the data space in Salesforce Data 360 where you want to load data.
If you do not enter a data space, the default data space is used.
Category
The data category assigned to data from source streams when it is loaded into Salesforce Data 360.
Options are:
  • - Profile. Applies to customer attributes such as name, email, or address.
  • - Engagement. Applies to behavioral and interaction data such as website visits, email opens, clicks, and other activity logs.
  • - Other. Applies to data that doesn’t fit into Profile or Engagement, such as transactional records, order histories, CRM system logs, and other system or contextual data.
Event Time Field
Name of the event date and time field for engagement records, if you selected the Engagement category. It defines exactly when an interaction occurred, such as a purchase, click, or login, and helps identify the most recent or relevant event.
Recorded Modified Field
The name of the recorded modified field that captures the date and time when a record was last updated if you selected the Profile or Other category.
Apply Mode
For incremental load and combined initial and incremental load jobs, indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
The default value is Standard.
Note: This field does not appear if you selected Query-based as the CDC method on the Source page of the task wizard.
Click Next to proceed, or click Save.

Configure a Snowflake Data Cloud target

Define target properties for the Snowflake destination.

Target Properties

Define the following required Snowflake target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source objects.
Schema
Select the target schema in which Application Ingestion and Replication creates the target tables.
Stage
The name of internal staging area that holds the data read from the source before the data is written to the target tables. This name must not include spaces. If the staging area does not exist, it will be automatically created.
Note: This field is not available if you selected the Superpipe option in the Advanced Target Properties.
Apply Mode
For incremental load and combined initial and incremental load jobs, indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
  • - Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
  • - Soft Deletes. Apply source delete operations to the target as soft deletes. A soft delete marks the deleted row as deleted without actually removing it from the database. For example, a delete on the source results in a change record on the target with "D" displayed in the INFA_OPERATION_TYPE column.
  • After enabling Soft Deletes, any update in the source table during normal or backlog mode results in the deletion of the matching record, insertion of the updated record, and marking of the INFA_OPERATION_TYPE operation as NULL in the target table. Similarly, inserting a record in the source table during backlog mode results in marking the INFA_OPERATION_TYPE operation as E in the target table record.
    Consider using soft deletes if you have a long-running business process that needs the soft-deleted data to finish processing, to restore data after an accidental delete operation, or to track deleted values for audit purposes.
  • - Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, transaction ID, and before image. Consider using Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
  • After enabling the Audit apply mode, any update in the source table during backlog or normal mode results in marking the INFA_OPERATION_TYPE operation as E in the target table record. Similarly, inserting a record in the source table during backlog mode results in marking the INFA_OPERATION_TYPE operation as E in the target table record.
    Note: The Audit apply mode applies for SAP source with SAP Mass Ingestion connector.
Default is Standard.
Click Next to proceed, or click Save.

Transform the data

You can apply trim transformations to selected tables and columns to remove spaces to the left or right of character column values. You can also define row-level filter rules to filter out data rows for source tables based on column conditions you define before the data is applied to the target.
Note: If you edit row-level filters in the task for a deployed job, you must Redeploy the job afterwards for the updated filters to take effect.
    1On the Transform Data page, select the tables and columns to which you want to assign a transformation.
    Note: You can apply trim transformations and row-level filters to the same tables and columns.
    2To add a trim transformation, click Add Transformation.
    The How do you want to transform your data? dialog box appears.
    3Click the + (Add a new row) icon to add a row. Then, in the Transformation Type list, select one of the following options:
    Click the Save icon to add the entry.
    4Click Next to go to the Summary tab where you can review your transformation settings.
    5If the settings are correct on the Summary tab, click Save to save them and return to the initial Transform Data page.
    6To add another transformation type for a different table or set of tables, repeat steps 1 through 5.
    Tip: You can remove a transformation assignment on the Transform Data page. Select the table with the unwanted transformation and click Clear All.
    7To add row-level filters to the selected tables and columns, click the down arrow next to Add Transformation and select Add Row Filter.
    The Add Row Filter option is available only only for application ingestion and replication tasks that have an SAP source (with an Oracle or HANA database) and use the SAP Mass Ingestion connector or that have a Salesforce source and use the Salesforce Mass Ingestion connector. The tasks can use any load type.
    Select the Add Row Filter option to add row-level filters to tables and columns.
    The How do you want to filter your data? dialog box appears.
    8Select the table and filter type to apply the filter conditions.
    1. aFrom the set of tables you previously selected, select the table that you want to assign a filter to.
    2. bSelect one of the following filter types:
    3. The default option is Basic.
    9To add a Basic filter, complete the following substeps:
    1. aClick the + (Add a new row) icon to add a row.
    2. bUnder Column Name, select a column.
    3. Columns with unsupported data types for row filtering are marked as "Not supported."
    4. cUnder Operator, select an operator type to use with the value.
    5. dUnder Value, select or enter a value, depending on the column type. Then click the Save icon on the right end of the row to save the condition.
    6. The following table describes the values that are valid for each column data type supported for filtering:
      Column data type
      Description
      INTEGER
      Enter a numeric value. You can use "+" and "-" only once before the number. The value must be between -2147483648 and 2147483647.
      LONG
      Enter a numeric value. You can use "+" and "-" only once before the number. The value must be between -9,223,372,036,854,775,808 and 9,223,372,036,854,775,807.
      BIGINT
      Enter a numeric value. You can use "+" and "-" only once before the number. Maximum length is 50 digits.
      BIGDEC
      Enter a numeric value. You can use "+" and "-" only once before the number. A decimal is allowed. Maximum length is 50 digits.
      STRING
      Enter text.
      DATE
      Use the date picker to select the date.
      TIME
      Enter the value in the format HH:MM:SS.MS, with milliseconds being optional and up to a maximum length is 9 digits.
      For example, 13:14:15.123456789
      DATETIME
      Use the date picker to select the date and time.
      OFFSET_DATETIME
      Use the date picker to select the date, time, and time zone.
      Note: Application Ingestion and Replication does not support BOOLEAN, BINARY, BLOB, CLOB, and graphic column data types.
    7. eClick Validate to test the syntax of the specified condition.
    8. fTo add another Basic condition, repeat steps a through e.
    9. The AND operator is used to combine the multiple conditions.
    10. gClick Save to validate and save the changes.
    11. hWhen done defining Basic filter conditions, click OK to return to the Transform Data page.
    10To define an Advanced filter that consists of multiple conditions combined with the AND or OR operator, manually enter the conditions in the box.
    Note: If you entered a Basic filter conditions for a column and then switched to the Advanced filter, the Basic condition is displayed so that you can add to it to make a more complex filter.
    1. aUnder Column Name, select a column and click the > arrow.
    2. The column name appears in the Filter Condition box.
      Note: For combined load tasks, do not include columns that you expect will be updated during CDC processing. If the column is updated, it might become ineligible for replication and cause unpredictable results. In this case, you'd need to Resync the job.
    3. bIn the Filter Condition box, type one or more conditions for the selected column. Manually enter conditions using the supported syntax and the appropriate operators, which can vary based on the column data type. You can also nest conditions using parentheses. See Syntax for row-level filtering. When done, click the Save icon on the right end of the row to save the advanced filter.
    4. The following table describes the values that are valid for each column data type supported for filtering:
      Column data type
      Description
      INTEGER
      Enter a numeric value. You can use "+" and "-" only once before the number. The value must be between -2147483648 and 2147483647.
      LONG
      Enter a numeric value. You can use "+" and "-" only once before the number. The value must be between -9,223,372,036,854,775,808 and 9,223,372,036,854,775,807.
      BIGINT
      Enter a number value. You can use "+" and "-" only once before the number. Maximum length is 50 digits.
      BIGDEC
      Enter a numeric value. You can use "+" and "-" only once before the number. A decimal is allowed. Maximum length is 50 digits.
      STRING
      Enter an input attribute in single quotes (').
      DATE
      Enter the value in the format YYYY-MM-DD. Enter the input attribute in single quotes (').
      TIME
      Enter the value in the format HH:MM:SS.MS, with milliseconds (MS) being optional and up to a maximum length is 9 digits. Enter the input attribute in single quotes (').
      For example, 13:14:15.123456789
      DATETIME
      Enter the date and time in the following format:
      YYYY-MM-DDTHH:MM:SS:MS
      For example, 2024-12-31T03:04:05.123456789
      Enter the input attribute in single quotes (').
      OFFSET_DATETIME
      Enter the date, time, and time zone in the following format:
      YYYY-MM-DDTHH:MM:SS.MS+05:00
      For example, 2024-03-15T10:03:04.123456789+05:00
      Enter the input attribute in single quotes (').
      Notes:
    5. cClick Validate to test the syntax of the specified conditions.
    6. Note: Switching from Advanced to Basic filter type after creating or editing an Advanced filter condition, deletes all changes to the filter condition, even if you saved it.
    7. dClick Save to validate and save the changes and then click OK to return to the Transform Data page.
    8. Note: Do not modify any column included in the filter after the task has been deployed. If you do so, the row-level filtering might not work properly.
    The Filters column on the Transform Data page shows the applied filters as hyperlinks. Clicking the link opens the selected filter in edit mode. Tables with an advanced filter display Advanced next to their filter conditions in the Filters column.
    On the Transform Data page, clicking the Clear All button at the top right hand corner removes all filters, including trim transformations and row-level filters from the selected tables.
    11When done, click Next.

Syntax for row-level filtering

If you create Advanced row-level filters when you define an application ingestion and replication task, ensure that you enter the filter conditions using the correct syntax. Otherwise, filter validation is likely to fail.
Operators
In an Advanced filter, you can use the following operators within a condition, depending on the column data type:
Operator
Description
=
Equals
!=
Does not equal
>
Greater than
>=
Greater than or equal to
<
Less than
<=
Less than or equal to
IS NULL
Contains a null
IS NOT NULL
Cannot contain a null
BETWEEN x AND y
Greater than or equal to x and less than or equal to y
NOT BETWEEN %s AND %s
Not greater than or equal to x and less than or equal to y
LIKE
A comparative operative for string columns only.
Example: LIKE '%06%7__' . This condition matches against the following values: 06789, A06X789, AB06XY789", "06X789, and A06789. However, it does not match against these values: A06789Z, A0678, A6789, "".
NOT LIKE
A comparative operative for string columns only.
IN
True if the operand is equal to one of a list of expressions
NOT IN
True if the operand is NOT equal to one of a list of expressions
+ - / *
Numeric computation operators for addition, subtraction, division, and multiplication
Syntax rules
In Advanced filters, use with the following syntax rules:

Finalize the task definition

Almost done! On the Let's Go! page, complete a few more properties. Then you can Save and Deploy the task.
    1Under General Properties, set the following properties:
    Property
    Description
    Task Name
    Enter a name that you want to use to identify theapplication ingestion and replication task, if you do not want to use the generated name. Using a descriptive name will make finding the task easier later.
    Task names can contain Latin alphanumeric characters, spaces, periods (.), commas (,), underscores (_), plus signs (+), and hyphens (-). Task names cannot include other special characters. Task names are not case sensitive. Maximum length is 50 characters.
    Note: If you include spaces in the task name, after you deploy the task, the spaces do not appear in the corresponding job name.
    Location
    The project or project\folder in Explore that will contain the task definition. If you do not specify a project, the "Default" project is used.
    Runtime Environment
    Select the runtime environment that you want to use to run the task. By default, the runtime environment that you initially entered when you began defining the task is displayed. You can use this runtime environment or select another one.
    Tip: To refresh the list of runtime environments, click Refresh.
    The runtime environment can be a Secure Agent group that consists of one or more Secure Agents. A Secure Agent is a lightweight program that runs tasks and enables secure communication.
    Alternatively, for application ingestion and replication initial load jobs that have selected source types, you can use a serverless runtime environment hosted on Microsoft Azure.
    Note: You cannot choose a serverless runtime environment if a local runtime environment was previously selected.
    The Cloud Hosted Agent is not supported.
    Select Set as default to use the specified runtime environment as your default environment for all tasks you create. Otherwise, leave this check box cleared.
    Description
    Optionally, enter a description you want to use for the task.
    Maximum length is 4,000 characters.
    Schedule
    If you want to run an initial load task based on a schedule instead of manually starting it, select Run this task based on a schedule. Then select a schedule that was previously defined in Administrator.
    The default option is Do not run this task based on a schedule.
    Note: This field is not available for incremental load and combined initial and incremental load tasks.
    To view and edit the schedule options, go to Administrator. If you edit the schedule, the changes will apply to all jobs that use the schedule. If you edit the schedule after deploying the task, you do not need to redeploy the task.
    If the schedule criteria for running the job is met but the previous job run is still active, Application Ingestion and Replication skips the new job run.
    Execute in Taskflow
    Select this check box to make the task available in Data Integration to add to a taskflow as an event source.You can then include transformations in the taskflow to transform the ingested data. Available for initial load and incremental load tasks with Snowflake targets that don't use the Superpipe option.
    2To display advanced properties, toggle on Show Advanced Options.
    3Optionally, edit the value in the Number of Rows in Output File value to specify the maximum number of rows that the application ingestion and replication task writes to an output file.
    Note: The Number of Rows in Output File field is not displayed for jobs that have an Apache Kafka target or if you use the Superpipe option for the Snowflake target.
    Valid values are 1 through 100000000. The default value for Amazon S3, Microsoft Azure Data Lake Storage Gen2, and Oracle Cloud Infrastructure (OCI) Object Storage targets is 1000 rows. For the other targets, the default value is 100000 rows.
    Note: For incremental load and combined initial and incremental load operations, change data is flushed to the target either when the specified number of rows is reached or when the flush latency period expires and the job is not in the middle of processing a transaction. The flush latency period is the time that the job waits for more change data before flushing data to the target. The latency period is set to 10 seconds and cannot be changed.
    4For initial load jobs only, optionally clear the File Extension Based on File Type check box if you want the output data files for Amazon S3, Google Cloud Storage, Microsoft Azure Data Lake Storage, or Microsoft Fabric OneLake targets to have the .dat extension. This check box is selected by default, which causes the output files to have file-name extensions based on their file types.
    Note: For incremental load jobs with these target types, this option is not available. Application Ingestion and Replication always uses output file-name extensions based on file type.
    5Optionally, configure an apply cycle. An apply cycle is a cycle of applying change data that starts with fetching the intermediate data from the source and ends with the commit of the data to the target. For continuous replication, the source processes the data in multiple low-latency apply cycles.
    For application ingestion and replication incremental load tasks that have Amazon S3, Google Cloud Storage, Microsoft Azure Data Lake Storage Gen2, or Microsoft Fabric OneLake targets, you can configure the following apply cycle options:
    Option
    Description
    Apply Cycle Interval
    Specifies the amount of time that must elapse before an application ingestion and replication job ends an apply cycle. You can specify days, hours, minutes, and seconds or specify values for a subset of these time fields leaving the other fields blank.
    The default value is 15 minutes.
    Apply Cycle Change Limit
    Specifies the number of records that must be processed before an application ingestion and replication job ends an apply cycle. When this record limit is reached, the ingestion job ends the apply cycle and writes the change data to the target.
    The default value is 10000 records.
    Note: During startup, jobs might reach this limit more frequently than the apply cycle interval if they need to catch up on processing a backlog of older data.
    Low Activity Flush Interval
    Specifies the amount of time, in hours, minutes, or both, that must elapse during a period of no change activity on the source before an application ingestion and replication job ends an apply cycle. When this time limit is reached, the ingestion job ends the apply cycle and writes the change data to the target.
    If you do not specify a value for this option, an application ingestion and replication job ends apply cycles only after either the Apply Cycle Change Limit or Apply Cycle Interval limit is reached.
    No default value is provided.
    6For incremental load jobs that have an Apache Kafka target, configure the following Checkpoint Options:
    Option
    Description
    Checkpoint All Rows
    Indicates whether an application ingestion and replication job performs checkpoint processing for every message that is sent to the Kafka target.
    Note: If this check box is selected, the Checkpoint Every Commit, Checkpoint Row Count, and Checkpoint Frequency (secs) options are ignored.
    Checkpoint Every Commit
    Indicates whether an application ingestion and replication job performs checkpoint processing for every commit that occurs on the source.
    Checkpoint Row Count
    Specifies the maximum number of messages that an application ingestion and replication job sends to the target before adding a checkpoint. If you set this option to 0, the job does not perform checkpoint processing based on the number of messages. If you set this option to 1, the job adds a checkpoint for each message.
    Checkpoint Frequency (secs)
    Specifies the maximum number of seconds that must elapse before an application ingestion and replication job adds a checkpoint. If you set this option to 0, an application ingestion and replication does not perform checkpoint processing based on elapsed time.
    7Under Schema Drift Options, if the detection of schema drift is supported for your source and target combination, specify the schema drift option to use for each of the supported types of DDL operations.
    Note: The Schema Drift Options section appears only for incremental load and combined initial and incremental load tasks. Additionally, this section appears only for the sources that support automatic detection of schema changes.
    The following table describes the schema drift options that you can set for a DDL operation type:
    Option
    Description
    Ignore
    Do not replicate DDL changes that occur on the source database to the target.
    Replicate
    Allow the application ingestion and replication job to replicate the DDL changes to the target.
    The types of supported DDL operations are:
    • - Add Column
    • - Modify Column
    • - Drop Column
    • - Rename Column
    Application ingestion and replication jobs doesn't support modifying or renaming columns for Google BigQuery target, and adding columns for Oracle targets.
    Stop Job
    Stop the application ingestion and replication job.
    Stop Table
    Stop processing the source object on which the DDL change occurred.
    Note: When one or more objects are excluded from replication because of the Stop Object schema drift option, the status of the job changes to Running with Warning. The application ingestion and replication job cannot retrieve the data changes that occurred on the source object after the job stops processing the changes. This action leads to data loss on the target. To avoid data loss, you must re-synchronize the source and target objects that the job stopped processing before you resume the application ingestion and replication job.
    8Under Custom Properties, you can specify one or more custom properties that Informatica provides to meet your special requirements. To add a property, in the Create Property field, enter the property name and value. Then click Add Property.
    Specify these properties only at the direction of Informatica Global Customer Support. Usually, these properties address unique environments or special processing needs. You can specify multiple properties, if necessary. A property name can contain only alphanumeric characters and the following special characters: periods (.), hyphens (-), and underscores (_)
    9Click Save to save the task.
    10Click Deploy to deploy a job instance for the task, or click View to view or edit the task.
    You can run a job that has the status of Deployed from the My Jobs page.