Database Ingestion and Replication > Configuring a database ingestion and replication task > Task Details: Configure how to replicate data to the target
  

Task Details: Configure how to replicate data to the target

Configure the data target in Step 2 of Task Details.
    bulletUnder Target Properties, set the required basic target properties. Then toggle on Show Advanced Options at the top of the page to set optional advanced target properties as needed. See the property descriptions for your target type:

Configure an Amazon Redshift target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Amazon Redshift target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source tables.
Note:
After the target table is created, Database Ingestion and Replication intelligently handles the target tables on subsequent job runs. Database Ingestion and Replication might truncate or re-create the target tables depending on the specific circumstances.
Schema
Select the target schema in which Database Ingestion and Replication creates the target tables.
Apply Mode
For incremental load and combined initial and incremental load jobs, this property indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
  • - Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
  • - Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, time, owner, transaction ID, generated ascending sequence number, and before image.
    • Consider using the Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
  • - Soft Deletes. Apply source delete operations to the target as soft deletes. A soft delete marks the deleted row as deleted without actually removing it from the database. For example, a delete on the source results in a change record on the target with "D" displayed in the INFA_OPERATION_TYPE column.
    • Consider using soft deletes if you have a long-running business process that needs the soft-deleted data to finish processing, to restore data after an accidental delete operation, or to track deleted values for audit purposes.
    Note:
    If you use Soft Deletes mode, you must not perform an update on the primary key in a source table. Otherwise, data corruption can occur on the target.
The default value is Standard.
Note:
This field does not appear if you selected Query-based as the CDC method on the Source page of the task wizard.
Bucket
The name of the Amazon S3 bucket that stores, organizes, and controls access to the data objects that you load to Amazon Redshift.
Data Directory or Task Target Directory
The subdirectory where Database Ingestion and Replication stores output files for jobs associated with the task. This field is called Data Directory for an initial load job or Task Target Directory for an incremental load or combined initial and incremental load job.
Click Next to proceed, or click Save.

Configure an Amazon S3 target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Amazon S3 target properties:
Property
Description
Open Table Format
Controls whether to use Open Table format to replicate data to Amazon S3 storage.
Select one of the following options:
  • - Apache Iceberg. Replicate data to the Amazon S3 cloud storage as Apache Iceberg tables. You can access these tables directly from Amazon S3 using the AWS Glue Catalog.
  • - None. Do not use Open Table format.
The default value is None.
Namespace
If you use the Apache Iceberg Open Table format, enter the name of the database in AWS Glue Catalog where you want to store and manage your Apache Iceberg tables.
Output Format
Select the format of the output file. Options are:
  • - CSV
  • - AVRO
  • - PARQUET
The default value is CSV if you do not use an Open Table format. If you select the Open table format, Parquet is selected by default for Apache Iceberg.
Note:
Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
Warehouse Base Directory
If you use the Apache Iceberg Open Table format, enter the root directory in Amazon S3 where you want to store the target files and tables.
Add Headers to CSV File
If CSV is selected as the output format, select this check box to add a header with source column names to the output CSV file.
Avro Format
If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are:
  • - Avro-Flat. This Avro schema format lists all Avro fields in one record.
  • - Avro-Generic. This Avro schema format lists all columns from a source table in a single array of Avro fields.
  • - Avro-Nested. This Avro schema format organizes each type of information in a separate record.
The default value is Avro-Flat.
Avro Serialization Format
If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are:
  • - None
  • - Binary
  • - JSON
The default value is Binary.
Avro Schema Directory
If AVRO is selected as the output format, specify the local directory where Database Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern:
schemaname_tablename.txt
Note:
If this directory is not specified, no Avro schema definition file is produced.
File Compression Type
Controls whether to use a file compression type for output files in CSV or AVRO output format. Options are:
  • - None
  • - Deflate
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Encryption Type
Select the encryption type for the output files that are written to the Amazon S3 target. Options are:
  • - None
  • - Client Side Encryption
  • - Client Side Encryption with KMS
  • - Server Side Encryption
  • - Server Side Encryption with KMS
Note:
If you use Client Side Encryption or Client Side Encryption with KMS, the maximum size of the file that is encrypted and decrypted is 64 GB.
The default is None, which means no encryption is used.
Avro Compression Type
If AVRO is selected as the output format, you can select an Avro compression type. Options are:
  • - None
  • - Bzip2
  • - Deflate
  • - Snappy
The default value is None, which means no compression is used.
Parquet Compression Type
If the PARQUET output format is selected, you can select a compression type that is supported by Parquet. Options are:
  • - None
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Deflate Compression Level
If Deflate is selected in the Avro Compression Type field, specify a compression level from 0 to 9. The default value is 0.
Add Directory Tags
For incremental load and combined initial and incremental load tasks, select this check box to add the "dt=" prefix to the names of apply cycle directories to be compatible with the naming convention for Hive partitioning. This check box is cleared by default.
Task Target Directory
For incremental load and combined initial and incremental load tasks, the root directory for the other directories that hold output data files, schema files, and CDC cycle contents and completed files. You can use it to specify a custom root directory for the task. If you enable the Connection Directory as Parent option, you can still optionally specify a task target directory to use with the parent directory specified in the connection properties.
This field is required if the {TaskTargetDirectory} placeholder is specified in patterns for any of the following directory fields.
Connection Directory as Parent
Select this check box to use the directory value that is specified in the target connection properties as the parent directory for the custom directory paths specified in the task target properties.
For initial load tasks, the parent directory is used in the Data Directory and Schema Directory.
For incremental load and combined initial and incremental load tasks, the parent directory is used in the Data Directory, Schema Directory, Cycle Completion Directory, and Cycle Contents Directory.
This check box is selected by default. If you clear it, for initial loads, define the full path to the output files in the Data Directory field. For incremental loads, optionally specify a root directory for the task in the Task Target Directory.
Data Directory
For initial load tasks, define a directory structure for the directories where Database Ingestion and Replication stores output data files and optionally stores the schema.
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter a folder name or use variables to create a folder name.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the initial load job starts to transfer data to the target.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For example:
myDir1/{SchemaName}/{TableName} myDir1/myDir2/{SchemaName}/{YYYY}/{MM}/{TableName}_{Timestamp} myDir1/{toLower(SchemaName)}/{TableName}_{Timestamp}
Note:
If you manually enter the directory expression, ensure that you enclose placeholders with curly brackets { }. Placeholder values are not case sensitive.
For incremental load and combined initial and incremental load tasks, define a custom path to the subdirectory that contains the cdc-data data files.
The default directory pattern is {TaskTargetDirectory}/data/{TableName}/data
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter {TaskTargetDirectory} for a task-specific base directory on the target to use instead of the S3 folder path specified in the connection properties.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the CDC cycle started.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For Amazon S3, Flat File, Microsoft Azure Data Lake Storage Gen2, and Oracle Cloud Object Store targets, Database Ingestion and Replication uses the directory specified in the target connection properties as the root for the data directory path when Connection Directory as Parent is selected.
For Google Cloud Storage targets, Database Ingestion and Replication uses the Bucket name that you specify in the target properties for the ingestion task.
For Microsoft Fabric OneLake targets, the parent directory is the path specified in the Lakehouse Path field in the Microsoft Fabric OneLake connection properties.
For Amazon S3 with Open Table format, enabling 'Connection Directory as Parent' appends the connection directory to the Warehouse Base Directory; if disabled, files are saved directly under the Warehouse Base Directory.
For Amazon S3 targets with Open Table format, the data directory field is not applicable. Enabling the Connection Directory as Parent includes the connection directory before the warehouse base path. If disabled, files are saved directly under the warehouse base directory.
Schema Directory
Specify a custom directory for storing the schema file if you want to store it in a directory other than the default directory. For initial loads, previously used values if available are shown in a list for your convenience. This field is optional.
For initial loads, the schema is stored in the data directory by default. For incremental loads and combined initial and incremental loads, the schema file is stored in the following directory by default: {TaskTargetDirectory}/data/{TableName}/schema.
You can use the same placeholders as for the Data Directory field. If you manually enter placeholders, ensure that you enclose them with curly brackets { }. If you include the toUpper or toLower function, put the placeholder name in parentheses and enclose both the function and placeholder in curly brackets, for example: {toLower(SchemaName)}
Note:
Schema is written only to output data files in CSV format. Data files in Parquet and Avro formats contain their own embedded schema.
Cycle Completion Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle completed file.
Default is {TaskTargetDirectory}/cycle/completed.
Cycle Contents Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle contents files.
Default is {TaskTargetDirectory}/cycle/contents.
Use Cycle Partitioning for Data Directory
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under each data directory.
If this option is not selected, individual data files are written to the same directory without a timestamp, unless you define an alternative directory structure.
Use Cycle Partitioning for Summary Directories
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under the summary contents and completed subdirectories.
List Individual Files in Contents
For incremental load and combined initial and incremental load tasks, lists individual data files under the contents subdirectory.
If Use Cycle Partitioning for Summary Directories is cleared, this option is selected by default. All of the individual files are listed in the contents subdirectory unless you can configure custom subdirectories by using the placeholders, such as for timestamp or date.
If Use Cycle Partitioning for Data Directory is selected, you can still optionally select this check box to list individual files and group them by CDC cycle.
Click Next to proceed, or click Save.

Configure a Databricks target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Databricks target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source tables.
Note:
After the target table is created, Database Ingestion and Replication intelligently handles the target tables on subsequent job runs. Database Ingestion and Replication might truncate or re-create the target tables depending on the specific circumstances.
Schema
Select the target schema in which Database Ingestion and Replication creates the target tables.
Apply Mode
For incremental load and combined initial and incremental load jobs, this property indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
  • - Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
  • - Soft Deletes. Apply source delete operations to the target as soft deletes. A soft delete marks the deleted row as deleted without actually removing it from the database. For example, a delete on the source results in a change record on the target with "D" displayed in the INFA_OPERATION_TYPE column.
    • Consider using soft deletes if you have a long-running business process that needs the soft-deleted data to finish processing, to restore data after an accidental delete operation, or to track deleted values for audit purposes.
    Note:
    If you use Soft Deletes mode, you must not perform an update on the primary key in a source table. Otherwise, data corruption can occur on the target.
  • - Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, time, owner, transaction ID, generated ascending sequence number, and before image.
    • Consider using the Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
The default value is Standard.
Note:
This field does not appear if you selected Query-based as the CDC method on the Source page of the task wizard.
Data Directory or Task Target Directory
The subdirectory where Database Ingestion and Replication stores output files for jobs associated with the task. This field is called Data Directory for an initial load job or Task Target Directory for an incremental load or combined initial and incremental load job.
Click Next to proceed, or click Save.

Configure a Flat File target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Flat File target properties:
Note:
For flat file targets, these properties apply to initial load jobs only.
Property
Description
Output Format
Select the format of the output file. Options are:
  • - CSV
  • - AVRO
The default value is CSV.
Note:
Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
Add Headers to CSV File
If CSV is selected as the output format, select this check box to add a header with source column names to the output CSV file.
Avro Format
If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are:
  • - Avro-Flat. This Avro schema format lists all Avro fields in one record.
  • - Avro-Generic. This Avro schema format lists all columns from a source table in a single array of Avro fields.
  • - Avro-Nested. This Avro schema format organizes each type of information in a separate record.
The default value is Avro-Flat.
Avro Serialization Format
If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are:
  • - None
  • - Binary
  • - JSON
The default value is Binary.
Avro Schema Directory
If AVRO is selected as the output format, specify the local directory where Database Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern:
schemaname_tablename.txt
Note:
If this directory is not specified, no Avro schema definition file is produced.
File Compression Type
Controls whether to use a file compression type for output files in CSV or AVRO output format. Options are:
  • - None
  • - Deflate
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Compression Type
If AVRO is selected as the output format, you can select an Avro compression type. Options are:
  • - None
  • - Bzip2
  • - Deflate
  • - Snappy
The default value is None, which means no compression is used.
Deflate Compression Level
If Deflate is selected in the Avro Compression Type field, specify a compression level from 0 to 9. The default value is 0.
Data Directory
For initial load tasks, define a directory structure for the directories where Database Ingestion and Replication stores output data files and optionally stores the schema. To define directory pattern, you can use the following types of entries:
  • - The placeholders {SchemaName}, {TableName), {Timestamp}, {YY}, {YYYY}, {MM}, and {DD}, where {YY}, {YYYY}, {MM}, and {DD} are for date elements. The {Timestamp} values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the initial load job starts to transfer data to the target.
  • - Specific directory names.
  • - The toUpper() and toLower() functions, which force the values for an associated (placeholder) to uppercase or lowercase.
Note:
Placeholder values are not case sensitive.
Examples:
myDir1/{SchemaName}/{TableName}
myDir1/myDir2/{SchemaName}/{YYYY}/{MM}/{TableName}_{Timestamp}
myDir1/{toLower(SchemaName)}/{TableName}_{Timestamp}
The default directory pattern is {TableName)_{Timestamp}.
Note:
For Flat File targets, Database Ingestion and Replication uses the directory specified in the target connection properties as the root for the data directory path when Connection Directory as Parent is selected.
Connection Directory as Parent
For initial load tasks, select this check box to use the directory value that is specified in the target connection properties as the parent directory for the custom directory paths specified in the task target properties. The parent directory is used in the Data Directory and Schema Directory.
Schema Directory
For initial load tasks, you can specify a custom directory in which to store the schema file if you want to store it in a directory other than the default directory. This field is optional.
The schema is stored in the data directory by default. For incremental loads, the default directory for the schema file is {TaskTargetDirectory}/data/{TableName}/schema.
You can use the same placeholders as for the Data Directory field. Ensure the placeholders are enclosed in curly brackets { }.
Click Next to proceed, or click Save.

Configure a Google BigQuery target

When you define a database ingestion and replication task that has a Google BigQuery target, you must enter some target properties on the Target tab of the task wizard.

Target Properties

Define the following required Google BigQuery target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source tables.
Note:
After the target table is created, Database Ingestion and Replication intelligently handles the target tables on subsequent job runs. Database Ingestion and Replication might truncate or re-create the target tables depending on the specific circumstances.
Schema
Select the target schema in which Database Ingestion and Replication creates the target tables.
Apply Mode
For incremental load and combined initial and incremental load jobs, this property indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
  • - Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
  • - Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, time, owner, transaction ID, generated ascending sequence number, and before image.
    • Consider using the Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
  • - Soft Deletes. Apply source delete operations to the target as soft deletes. A soft delete marks the deleted row as deleted without actually removing it from the database. For example, a delete on the source results in a change record on the target with "D" displayed in the INFA_OPERATION_TYPE column.
    • Consider using soft deletes if you have a long-running business process that needs the soft-deleted data to finish processing, to restore data after an accidental delete operation, or to track deleted values for audit purposes.
    Note:
    If you use Soft Deletes mode, you must not perform an update on the primary key in a source table. Otherwise, data corruption can occur on the target.
The default value is Standard.
Note:
This field does not appear if you selected Query-based as the CDC method on the Source page of the task wizard.
Bucket
The name of an existing bucket container that stores, organizes, and controls access to the data objects that you load to Google Cloud Storage.
Data Directory or Task Target Directory
The subdirectory where Database Ingestion and Replication stores output files for jobs associated with the task. This field is called Data Directory for an initial load job or Task Target Directory for an incremental load or combined initial and incremental load job.
Click Next to proceed, or click Save.

Configure a Google Cloud Storage target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Google Cloud Storage target properties:
Property
Description
Output Format
The format of the output files. Options are:
  • - CSV
  • - AVRO
  • - PARQUET
The default value is CSV.
Note:
Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
Add Headers to CSV File
If CSV is selected as the output format, select this check box to add a header with source column names to the output CSV file.
Avro Format
If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are:
  • - Avro-Flat. This Avro schema format lists all Avro fields in one record.
  • - Avro-Generic. This Avro schema format lists all columns from a source table in a single array of Avro fields.
  • - Avro-Nested. This Avro schema format organizes each type of information in a separate record.
The default value is Avro-Flat.
Avro Serialization Format
If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are:
  • - None
  • - Binary
  • - JSON
The default value is Binary.
Avro Schema Directory
If AVRO is selected as the output format, specify the local directory where Database Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern:
schemaname_tablename.txt
Note:
If this directory is not specified, no Avro schema definition file is produced.
File Compression Type
Controls whether to use a file compression type for output files in CSV or AVRO output format. Options are:
  • - None
  • - Deflate
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Compression Type
If AVRO is selected as the output format, you can select an Avro compression type. Options are:
  • - None
  • - Bzip2
  • - Deflate
  • - Snappy
The default value is None, which means no compression is used.
Parquet Compression Type
If the PARQUET output format is selected, you can select a compression type that is supported by Parquet. Options are:
  • - None
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Deflate Compression Level
If Deflate is selected in the Avro Compression Type field, specify a compression level from 0 to 9. The default value is 0.
Add Directory Tags
For incremental load and combined initial and incremental load tasks, select this check box to add the "dt=" prefix to the names of apply cycle directories to be compatible with the naming convention for Hive partitioning. This check box is cleared by default.
Bucket
The name of an existing bucket container that stores, organizes, and controls access to the data objects that you load to Google Cloud Storage.
Task Target Directory
For incremental load and combined initial and incremental load tasks, the root directory for the other directories that hold output data files, schema files, and CDC cycle contents and completed files. You can use it to specify a custom root directory for the task. If you enable the Connection Directory as Parent option, you can still optionally specify a task target directory to use with the parent directory specified in the connection properties.
This field is required if the {TaskTargetDirectory} placeholder is specified in patterns for any of the following directory fields.
Data Directory
For initial load tasks, define a directory structure for the directories where Database Ingestion and Replication stores output data files and optionally stores the schema.
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter a folder name or use variables to create a folder name.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the initial load job starts to transfer data to the target.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For example:
myDir1/{SchemaName}/{TableName} myDir1/myDir2/{SchemaName}/{YYYY}/{MM}/{TableName}_{Timestamp} myDir1/{toLower(SchemaName)}/{TableName}_{Timestamp}
Note:
If you manually enter the directory expression, ensure that you enclose placeholders with curly brackets { }. Placeholder values are not case sensitive.
For incremental load and combined initial and incremental load tasks, define a custom path to the subdirectory that contains the cdc-data data files.
The default directory pattern is {TaskTargetDirectory}/data/{TableName}/data
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter {TaskTargetDirectory} for a task-specific base directory on the target to use instead of the S3 folder path specified in the connection properties.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the CDC cycle started.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For Amazon S3, Flat File, Microsoft Azure Data Lake Storage Gen2, and Oracle Cloud Object Store targets, Database Ingestion and Replication uses the directory specified in the target connection properties as the root for the data directory path when Connection Directory as Parent is selected.
For Google Cloud Storage targets, Database Ingestion and Replication uses the Bucket name that you specify in the target properties for the ingestion task.
For Microsoft Fabric OneLake targets, the parent directory is the path specified in the Lakehouse Path field in the Microsoft Fabric OneLake connection properties.
For Amazon S3 with Open Table format, enabling 'Connection Directory as Parent' appends the connection directory to the Warehouse Base Directory; if disabled, files are saved directly under the Warehouse Base Directory.
For Amazon S3 targets with Open Table format, the data directory field is not applicable. Enabling the Connection Directory as Parent includes the connection directory before the warehouse base path. If disabled, files are saved directly under the warehouse base directory.
Schema Directory
Specify a custom directory for storing the schema file if you want to store it in a directory other than the default directory. For initial loads, previously used values if available are shown in a list for your convenience. This field is optional.
For initial loads, the schema is stored in the data directory by default. For incremental loads and combined initial and incremental loads, the schema file is stored in the following directory by default: {TaskTargetDirectory}/data/{TableName}/schema.
You can use the same placeholders as for the Data Directory field. If you manually enter placeholders, ensure that you enclose them with curly brackets { }. If you include the toUpper or toLower function, put the placeholder name in parentheses and enclose both the function and placeholder in curly brackets, for example: {toLower(SchemaName)}
Note:
Schema is written only to output data files in CSV format. Data files in Parquet and Avro formats contain their own embedded schema.
Cycle Completion Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle completed file.
Default is {TaskTargetDirectory}/cycle/completed.
Cycle Contents Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle contents files.
Default is {TaskTargetDirectory}/cycle/contents.
Use Cycle Partitioning for Data Directory
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under each data directory.
If this option is not selected, individual data files are written to the same directory without a timestamp, unless you define an alternative directory structure.
Use Cycle Partitioning for Summary Directories
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under the summary contents and completed subdirectories.
List Individual Files in Contents
For incremental load and combined initial and incremental load tasks, lists individual data files under the contents subdirectory.
If Use Cycle Partitioning for Summary Directories is cleared, this option is selected by default. All of the individual files are listed in the contents subdirectory unless you can configure custom subdirectories by using the placeholders, such as for timestamp or date.
If Use Cycle Partitioning for Data Directory is selected, you can still optionally select this check box to list individual files and group them by CDC cycle.
Click Next to proceed, or click Save.

Configure a Kafka target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Kafka target properties:
Note:
These properties apply to incremental load operations only.
Property
Description
Use Table Name as Topic Name
Controls whether Database Ingestion and Replication writes messages that contain source data to separate topics, one for each source table, or writes all messages to a single topic.
Select this check box to write messages to separate table-specific topics. The topic names match the source table names, unless you add the source schema name, a prefix, or a suffix in the Include Schema Name, Table Prefix, or Table Suffix properties.
By default, this check box is cleared. With the default setting, you must specify the name of the single topic to which all messages are written in the Topic Name property.
Include Schema Name
When Use Table Name as Topic Name is selected, this check box appears and is selected by default. This setting adds the source schema name in the table-specific topic names. The topic names then have the format schemaname_tablename.
If you do not want to include the schema name, clear this check box.
Table Prefix
When Use Table Name as Topic Name is selected, this property appears so that you can optionally enter a prefix to add to the table-specific topic names. For example, if you specify myprefix_, the topic names have the format myprefix_tablename. If you omit the underscore (_) after the prefix, the prefix is prepended to the table name.
Table Suffix
When Use Table Name as Topic Name is selected, this property appears so that you can optionally enter a suffix to add to the table-specific topic names. For example, if you specify _mysuffix, the topic names have the format tablename_mysuffix. If you omit the underscore (_) before the suffix, the suffix is appended to the table name.
Topic Name
When Use table name as topic name is not selected, you must enter the name of the single Kafka topic to which all messages that contain source data will be written.
Output Format
Select the format of the output file. Options are:
  • - CSV
  • - AVRO
  • - JSON
The default value is CSV.
Note:
Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
If your Kafka target uses Confluent Schema Registry to store schemas for incremental load jobs, you must select AVRO as the format.
JSON Format
If JSON is selected as the output format, select the level of detail of the output. Options are:
  • - Concise. This format records only the most relevant data in the output, such as the operation type and the column names and values.
  • - Verbose. This format records detailed information, such as the table name and column types.
Avro Format
If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are:
  • - Avro-Flat. This Avro schema format lists all Avro fields in one record.
  • - Avro-Generic. This Avro schema format lists all columns from a source table in a single array of Avro fields.
  • - Avro-Nested. This Avro schema format organizes each type of information in a separate record.
The default value is Avro-Flat.
Avro Serialization Format
If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are:
  • - Binary
  • - JSON
  • - None
The default value is Binary.
If you have a Confluent Kafka target that uses Confluent Schema Registry to store schemas, select None. Otherwise, Confluent Schema Registry does not register the schema. Do not select None if you are not using Confluent Scheme Registry.
Avro Schema Directory
If AVRO is selected as the output format, specify the local directory where Database Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern:
schemaname_tablename.txt
Note:
If this directory is not specified, no Avro schema definition file is produced.
If a source schema change is expected to alter the target, the Avro schema definition file is regenerated with a unique name that includes a timestamp, in the following format:
schemaname_tablename_YYYYMMDDhhmmss.txt
This unique naming pattern ensures that older schema definition files are preserved for audit purposes.
Avro Compression Type
If AVRO is selected as the output format, you can select an Avro compression type. Options are:
  • - None
  • - Bzip2
  • - Deflate
  • - Snappy
The default value is None, which means no compression is used.
Deflate Compression Level
If Deflate is selected in the Avro Compression Type field, specify a compression level from 0 to 9. The default value is 0.
Click Next to proceed, or click Save.

Configure a Microsoft Azure Data Lake Storage target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Microsoft Azure Data Lake Storage target properties:
Property
Description
Output Format
The format of the output files. Options are:
  • - CSV
  • - AVRO
  • - PARQUET
The default value is CSV.
Note:
Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
Add Headers to CSV File
If CSV is selected as the output format, select this check box to add a header with source column names to the output CSV file.
Avro Format
If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are:
  • - Avro-Flat. This Avro schema format lists all Avro fields in one record.
  • - Avro-Generic. This Avro schema format lists all columns from a source table in a single array of Avro fields.
  • - Avro-Nested. This Avro schema format organizes each type of information in a separate record.
The default value is Avro-Flat.
Avro Serialization Format
If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are:
  • - None
  • - Binary
  • - JSON
The default value is Binary.
Avro Schema Directory
If AVRO is selected as the output format, specify the local directory where Database Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern:
schemaname_tablename.txt
Note:
If this directory is not specified, no Avro schema definition file is produced.
File Compression Type
Controls whether to use a file compression type for output files in CSV or AVRO output format. Options are:
  • - None
  • - Deflate
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Compression Type
If AVRO is selected as the output format, you can select an Avro compression type. Options are:
  • - None
  • - Bzip2
  • - Deflate
  • - Snappy
The default value is None, which means no compression is used.
Parquet Compression Type
If the PARQUET output format is selected, you can select a compression type that is supported by Parquet. Options are:
  • - None
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Deflate Compression Level
If Deflate is selected in the Avro Compression Type field, specify a compression level from 0 to 9. The default value is 0.
Add Directory Tags
For incremental load and combined initial and incremental load tasks, select this check box to add the "dt=" prefix to the names of apply cycle directories to be compatible with the naming convention for Hive partitioning. This check box is cleared by default.
Task Target Directory
For incremental load and combined initial and incremental load tasks, the root directory for the other directories that hold output data files, schema files, and CDC cycle contents and completed files. You can use it to specify a custom root directory for the task. If you enable the Connection Directory as Parent option, you can still optionally specify a task target directory to use with the parent directory specified in the connection properties.
This field is required if the {TaskTargetDirectory} placeholder is specified in patterns for any of the following directory fields.
Connection Directory as Parent
Select this check box to use the directory value that is specified in the target connection properties as the parent directory for the custom directory paths specified in the task target properties.
For initial load tasks, the parent directory is used in the Data Directory and Schema Directory.
For incremental load and combined initial and incremental load tasks, the parent directory is used in the Data Directory, Schema Directory, Cycle Completion Directory, and Cycle Contents Directory.
This check box is selected by default. If you clear it, for initial loads, define the full path to the output files in the Data Directory field. For incremental loads, optionally specify a root directory for the task in the Task Target Directory.
Data Directory
For initial load tasks, define a directory structure for the directories where Database Ingestion and Replication stores output data files and optionally stores the schema.
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter a folder name or use variables to create a folder name.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the initial load job starts to transfer data to the target.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For example:
myDir1/{SchemaName}/{TableName} myDir1/myDir2/{SchemaName}/{YYYY}/{MM}/{TableName}_{Timestamp} myDir1/{toLower(SchemaName)}/{TableName}_{Timestamp}
Note:
If you manually enter the directory expression, ensure that you enclose placeholders with curly brackets { }. Placeholder values are not case sensitive.
For incremental load and combined initial and incremental load tasks, define a custom path to the subdirectory that contains the cdc-data data files.
The default directory pattern is {TaskTargetDirectory}/data/{TableName}/data
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter {TaskTargetDirectory} for a task-specific base directory on the target to use instead of the S3 folder path specified in the connection properties.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the CDC cycle started.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For Amazon S3, Flat File, Microsoft Azure Data Lake Storage Gen2, and Oracle Cloud Object Store targets, Database Ingestion and Replication uses the directory specified in the target connection properties as the root for the data directory path when Connection Directory as Parent is selected.
For Google Cloud Storage targets, Database Ingestion and Replication uses the Bucket name that you specify in the target properties for the ingestion task.
For Microsoft Fabric OneLake targets, the parent directory is the path specified in the Lakehouse Path field in the Microsoft Fabric OneLake connection properties.
For Amazon S3 with Open Table format, enabling 'Connection Directory as Parent' appends the connection directory to the Warehouse Base Directory; if disabled, files are saved directly under the Warehouse Base Directory.
For Amazon S3 targets with Open Table format, the data directory field is not applicable. Enabling the Connection Directory as Parent includes the connection directory before the warehouse base path. If disabled, files are saved directly under the warehouse base directory.
Schema Directory
Specify a custom directory for storing the schema file if you want to store it in a directory other than the default directory. For initial loads, previously used values if available are shown in a list for your convenience. This field is optional.
For initial loads, the schema is stored in the data directory by default. For incremental loads and combined initial and incremental loads, the schema file is stored in the following directory by default: {TaskTargetDirectory}/data/{TableName}/schema.
You can use the same placeholders as for the Data Directory field. If you manually enter placeholders, ensure that you enclose them with curly brackets { }. If you include the toUpper or toLower function, put the placeholder name in parentheses and enclose both the function and placeholder in curly brackets, for example: {toLower(SchemaName)}
Note:
Schema is written only to output data files in CSV format. Data files in Parquet and Avro formats contain their own embedded schema.
Cycle Completion Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle completed file.
Default is {TaskTargetDirectory}/cycle/completed.
Cycle Contents Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle contents files.
Default is {TaskTargetDirectory}/cycle/contents.
Use Cycle Partitioning for Data Directory
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under each data directory.
If this option is not selected, individual data files are written to the same directory without a timestamp, unless you define an alternative directory structure.
Use Cycle Partitioning for Summary Directories
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under the summary contents and completed subdirectories.
List Individual Files in Contents
For incremental load and combined initial and incremental load tasks, lists individual data files under the contents subdirectory.
If Use Cycle Partitioning for Summary Directories is cleared, this option is selected by default. All of the individual files are listed in the contents subdirectory unless you can configure custom subdirectories by using the placeholders, such as for timestamp or date.
If Use Cycle Partitioning for Data Directory is selected, you can still optionally select this check box to list individual files and group them by CDC cycle.
Click Next to proceed, or click Save.

Configure a Microsoft Azure Synapse Analytics target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Microsoft Azure Synapse Analytics target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source tables.
Note:
After the target table is created, Database Ingestion and Replication intelligently handles the target tables on subsequent job runs. Database Ingestion and Replication might truncate or re-create the target tables depending on the specific circumstances.
Schema
Select the target schema in which Database Ingestion and Replication creates the target tables. The schema name that is specified in the connection properties is displayed by default. Because this field is case sensitive, ensure that you entered the schema name in the connection properties in the correct case.
Click Next to proceed, or click Save.

Configure a Microsoft Fabric Data Warehouse target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Microsoft Fabric Data Warehouse target properties:
Property
Description
Schema Path
Select the schema path where database ingestion and replication creates the target tables.
Apply Mode
For incremental load and combined initial and incremental load jobs, this property indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
  • - Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
  • - Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, time, owner, transaction ID, generated ascending sequence number, and before image.
    • Consider using the Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
The default value is Standard.
Note:
This field does not appear if you selected Query-based as the CDC method on the Source page of the task wizard.
Staging Lakehouse Name
The name of the intermediate storage location in the data lake where data is temporarily stored before it is loaded into Microsoft Fabric Data Warehouse.
Click Next to proceed, or click Save.

Configure a Microsoft Fabric OneLake target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Microsoft Fabric OneLake target properties:
Property
Description
Open Mirroring
Select this checkbox to use Microsoft Fabric open mirroring. Open mirroring creates a synchronized, near real-time copy of your source data in Microsoft Fabric OneLake in the Parquet format. For more information, see Using open mirroring to replicate data to Microsoft Fabric OneLake.
Note:
Make sure to select Parquet as the output format before you enable Open Mirroring and continue with the open mirroring configurations.
Landing Zone URL
If you selected Open Mirroring, enter the unique Microsoft Fabric OneLake path where mirrored data files are stored in Parquet format before processing in Microsoft Fabric.
Note:
The rest of the fields in this table don't apply when you enable open mirroring.
Output Format
Select the format of the output file. Options are:
  • - CSV
  • - AVRO
  • - PARQUET
The default value is CSV. Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
Note:
Open mirroring uses the Parquet output format.
Add Headers to CSV File
If CSV is selected as the output format, select this check box to add a header with source column names to the output CSV file.
Avro Format
If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are:
  • - Avro-Flat. This Avro schema format lists all Avro fields in one record.
  • - Avro-Generic. This Avro schema format lists all columns from a source table in a single array of Avro fields.
  • - Avro-Nested. This Avro schema format organizes each type of information in a separate record.
The default value is Avro-Flat.
Avro Serialization Format
If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are:
  • - None
  • - Binary
  • - JSON
The default value is Binary.
Avro Schema Directory
If AVRO is selected as the output format, specify the local directory where Database Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern:
schemaname_tablename.txt
Note:
If this directory is not specified, no Avro schema definition file is produced.
File Compression Type
Controls whether to use a file compression type for output files in CSV or AVRO output format. Options are:
  • - None
  • - Deflate
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Compression Type
If AVRO is selected as the output format, you can select an Avro compression type. Options are:
  • - None
  • - Bzip2
  • - Deflate
  • - Snappy
The default value is None, which means no compression is used.
Parquet Compression Type
If the PARQUET output format is selected, you can select a compression type that is supported by Parquet. Options are:
  • - None
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Deflate Compression Level
If Deflate is selected in the Avro Compression Type field, specify a compression level from 0 to 9. The default value is 0.
Add Directory Tags
For incremental load and combined initial and incremental load tasks, select this check box to add the "dt=" prefix to the names of apply cycle directories to be compatible with the naming convention for Hive partitioning. This check box is cleared by default.
Task Target Directory
For incremental load and combined initial and incremental load tasks, the root directory for the other directories that hold output data files, schema files, and CDC cycle contents and completed files. You can use it to specify a custom root directory for the task.
This field is required if the {TaskTargetDirectory} placeholder is specified in patterns for any of the following directory fields.
Data Directory
For initial load tasks, define a directory structure for the directories where Database Ingestion and Replication stores output data files and optionally stores the schema.
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter a folder name or use variables to create a folder name.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the initial load job starts to transfer data to the target.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For example:
myDir1/{SchemaName}/{TableName} myDir1/myDir2/{SchemaName}/{YYYY}/{MM}/{TableName}_{Timestamp} myDir1/{toLower(SchemaName)}/{TableName}_{Timestamp}
Note:
If you manually enter the directory expression, ensure that you enclose placeholders with curly brackets { }. Placeholder values are not case sensitive.
For incremental load and combined initial and incremental load tasks, define a custom path to the subdirectory that contains the cdc-data data files.
The default directory pattern is {TaskTargetDirectory}/data/{TableName}/data
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter {TaskTargetDirectory} for a task-specific base directory on the target to use instead of the S3 folder path specified in the connection properties.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the CDC cycle started.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For Amazon S3, Flat File, Microsoft Azure Data Lake Storage Gen2, and Oracle Cloud Object Store targets, Database Ingestion and Replication uses the directory specified in the target connection properties as the root for the data directory path when Connection Directory as Parent is selected.
For Google Cloud Storage targets, Database Ingestion and Replication uses the Bucket name that you specify in the target properties for the ingestion task.
For Microsoft Fabric OneLake targets, the parent directory is the path specified in the Lakehouse Path field in the Microsoft Fabric OneLake connection properties.
For Amazon S3 with Open Table format, enabling 'Connection Directory as Parent' appends the connection directory to the Warehouse Base Directory; if disabled, files are saved directly under the Warehouse Base Directory.
For Amazon S3 targets with Open Table format, the data directory field is not applicable. Enabling the Connection Directory as Parent includes the connection directory before the warehouse base path. If disabled, files are saved directly under the warehouse base directory.
Schema Directory
Specify a custom directory for storing the schema file if you want to store it in a directory other than the default directory. For initial loads, previously used values if available are shown in a list for your convenience. This field is optional.
For initial loads, the schema is stored in the data directory by default. For incremental loads and combined initial and incremental loads, the schema file is stored in the following directory by default: {TaskTargetDirectory}/data/{TableName}/schema.
You can use the same placeholders as for the Data Directory field. If you manually enter placeholders, ensure that you enclose them with curly brackets { }. If you include the toUpper or toLower function, put the placeholder name in parentheses and enclose both the function and placeholder in curly brackets, for example: {toLower(SchemaName)}
Note:
Schema is written only to output data files in CSV format. Data files in Parquet and Avro formats contain their own embedded schema.
Cycle Completion Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle completed file.
Default is {TaskTargetDirectory}/cycle/completed.
Cycle Contents Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle contents files.
Default is {TaskTargetDirectory}/cycle/contents.
Use Cycle Partitioning for Data Directory
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under each data directory.
If this option is not selected, individual data files are written to the same directory without a timestamp, unless you define an alternative directory structure.
Use Cycle Partitioning for Summary Directories
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under the summary contents and completed subdirectories.
List Individual Files in Contents
For incremental load and combined initial and incremental load tasks, lists individual data files under the contents subdirectory.
If Use Cycle Partitioning for Summary Directories is cleared, this option is selected by default. All of the individual files are listed in the contents subdirectory unless you can configure custom subdirectories by using the placeholders, such as for timestamp or date.
If Use Cycle Partitioning for Data Directory is selected, you can still optionally select this check box to list individual files and group them by CDC cycle.
Click Next to proceed, or click Save.

Configure a Microsoft SQL Server target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Microsoft SQL Server target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source tables.
Note:
After the target table is created, Database Ingestion and Replication intelligently handles the target tables on subsequent job runs. Database Ingestion and Replication might truncate or re-create the target tables depending on the specific circumstances.
Schema
Select the target schema in which Database Ingestion and Replication creates the target tables.
Apply Mode
For incremental load and combined initial and incremental load jobs, this property indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
  • - Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
  • - Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, time, owner, transaction ID, generated ascending sequence number, and before image.
    • Consider using the Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
  • - Soft Deletes. Apply source delete operations to the target as soft deletes. A soft delete marks the deleted row as deleted without actually removing it from the database. For example, a delete on the source results in a change record on the target with "D" displayed in the INFA_OPERATION_TYPE column.
    • Consider using soft deletes if you have a long-running business process that needs the soft-deleted data to finish processing, to restore data after an accidental delete operation, or to track deleted values for audit purposes.
    Note:
    If you use Soft Deletes mode, you must not perform an update on the primary key in a source table. Otherwise, data corruption can occur on the target.
The default value is Standard.
Note:
This field does not appear if you selected Query-based as the CDC method on the Source page of the task wizard.
Click Next to proceed, or click Save.

Configure an Open Table target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Open Table target properties:
Property
Description
Schema Path
The schema path in Open Table where Database Ingestion and Replication creates the target tables.
Catalog Name with Database Name
The catalog and database name where the task organizes and stores metadata for the target tables.
Enter the values in the following format: <catalogname>/<databasename>
Table Location
The storage path in Amazon S3 where the task creates the Open Table data files.
For example: s3://s3format/cdc
Table Properties
Configuration settings that you can define for the Open Table target.
Enter the values as comma-separated key-value pairs in the following format: '<parameter name>=<parameter value>'='<parameter value>','<parameter name>=<parameter value>'='<parameter value>'.
For the Open Table target, the task loads data in Parquet format. So, you must include the following property: 'format'='parquet'
Table Type
The table type that defines how the Open Table format is stored and managed.
You must set the value to iceberg.
Apply Mode
For incremental load and combined initial and incremental load jobs, this property indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
  • - Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
  • - Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, time, owner, transaction ID, generated ascending sequence number, and before image.
    • Consider using the Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
The default value is Standard.
Note:
This field does not appear if you selected Query-based as the CDC method on the Source page of the task wizard.
Enable Case Transformation
By default, target table names and column names are generated in the same case as the corresponding source names, unless cluster-level or session-level properties on the target override this case-sensitive behavior. If you want to control the case of letters in the target names, select this check box.
Click Next to proceed, or click Save.

Configure an Oracle target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Oracle target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source tables.
Note:
After the target table is created, Database Ingestion and Replication intelligently handles the target tables on subsequent job runs. Database Ingestion and Replication might truncate or re-create the target tables depending on the specific circumstances.
Schema
Select the target schema in which Database Ingestion and Replication creates the target tables.
Apply Mode
For incremental load and combined initial and incremental load jobs, this property indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
  • - Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
  • - Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, time, owner, transaction ID, and generated ascending sequence number. Consider using Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
The default value is Standard.
Note:
This field does not appear if you selected Query-based as the CDC method on the Source page of the task wizard.
Click Next to proceed, or click Save.

Configure an Oracle Cloud Object Storage target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Oracle Cloud Object Storage target properties:
Property
Description
Output Format
The format of the output files. Options are:
  • - CSV
  • - AVRO
  • - PARQUET
The default value is CSV.
Note:
Output files in CSV format use double-quotation marks ("") as the delimiter for each field.
Add Headers to CSV File
If CSV is selected as the output format, select this check box to add a header with source column names to the output CSV file.
Avro Format
If you selected AVRO as the output format, select the format of the Avro schema that will be created for each source table. Options are:
  • - Avro-Flat. This Avro schema format lists all Avro fields in one record.
  • - Avro-Generic. This Avro schema format lists all columns from a source table in a single array of Avro fields.
  • - Avro-Nested. This Avro schema format organizes each type of information in a separate record.
The default value is Avro-Flat.
Avro Serialization Format
If AVRO is selected as the output format, select the serialization format of the Avro output file. Options are:
  • - None
  • - Binary
  • - JSON
The default value is Binary.
Avro Schema Directory
If AVRO is selected as the output format, specify the local directory where Database Ingestion and Replication stores Avro schema definitions for each source table. Schema definition files have the following naming pattern:
schemaname_tablename.txt
Note:
If this directory is not specified, no Avro schema definition file is produced.
File Compression Type
Controls whether to use a file compression type for output files in CSV or AVRO output format. Options are:
  • - None
  • - Deflate
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Avro Compression Type
If AVRO is selected as the output format, you can select an Avro compression type. Options are:
  • - None
  • - Bzip2
  • - Deflate
  • - Snappy
The default value is None, which means no compression is used.
Parquet Compression Type
If the PARQUET output format is selected, you can select a compression type that is supported by Parquet. Options are:
  • - None
  • - Gzip
  • - Snappy
The default value is None, which means no compression is used.
Deflate Compression Level
If Deflate is selected in the Avro Compression Type field, specify a compression level from 0 to 9. The default value is 0.
Add Directory Tags
For incremental load and combined initial and incremental load tasks, select this check box to add the "dt=" prefix to the names of apply cycle directories to be compatible with the naming convention for Hive partitioning. This check box is cleared by default.
Task Target Directory
For incremental load and combined initial and incremental load tasks, the root directory for the other directories that hold output data files, schema files, and CDC cycle contents and completed files. You can use it to specify a custom root directory for the task. If you enable the Connection Directory as Parent option, you can still optionally specify a task target directory to use with the parent directory specified in the connection properties.
This field is required if the {TaskTargetDirectory} placeholder is specified in patterns for any of the following directory fields.
Connection Directory as Parent
Select this check box to use the directory value that is specified in the target connection properties as the parent directory for the custom directory paths specified in the task target properties.
For initial load tasks, the parent directory is used in the Data Directory and Schema Directory.
For incremental load and combined initial and incremental load tasks, the parent directory is used in the Data Directory, Schema Directory, Cycle Completion Directory, and Cycle Contents Directory.
This check box is selected by default. If you clear it, for initial loads, define the full path to the output files in the Data Directory field. For incremental loads, optionally specify a root directory for the task in the Task Target Directory.
Data Directory
For initial load tasks, define a directory structure for the directories where Database Ingestion and Replication stores output data files and optionally stores the schema.
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter a folder name or use variables to create a folder name.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the initial load job starts to transfer data to the target.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For example:
myDir1/{SchemaName}/{TableName} myDir1/myDir2/{SchemaName}/{YYYY}/{MM}/{TableName}_{Timestamp} myDir1/{toLower(SchemaName)}/{TableName}_{Timestamp}
Note:
If you manually enter the directory expression, ensure that you enclose placeholders with curly brackets { }. Placeholder values are not case sensitive.
For incremental load and combined initial and incremental load tasks, define a custom path to the subdirectory that contains the cdc-data data files.
The default directory pattern is {TaskTargetDirectory}/data/{TableName}/data
To customize the directory pattern, click the Edit icon to select from the following listed path types and values:
  • - Folder Path. Enter {TaskTargetDirectory} for a task-specific base directory on the target to use instead of the S3 folder path specified in the connection properties.
  • - Timestamp values. Select data elements Timestamp, yy, yyyy, mm, or dd. The Timestamp values are in the format yyyymmdd_hhmissms. The generated dates and times in the directory paths indicate when the CDC cycle started.
  • - Schema Name. Select SchemaName, toUpper(SchemaName), or toLower(SchemaName).
  • - Table Name. Select TableName, toUpper(TableName), and toLower(TableName).
For Amazon S3, Flat File, Microsoft Azure Data Lake Storage Gen2, and Oracle Cloud Object Store targets, Database Ingestion and Replication uses the directory specified in the target connection properties as the root for the data directory path when Connection Directory as Parent is selected.
For Google Cloud Storage targets, Database Ingestion and Replication uses the Bucket name that you specify in the target properties for the ingestion task.
For Microsoft Fabric OneLake targets, the parent directory is the path specified in the Lakehouse Path field in the Microsoft Fabric OneLake connection properties.
For Amazon S3 with Open Table format, enabling 'Connection Directory as Parent' appends the connection directory to the Warehouse Base Directory; if disabled, files are saved directly under the Warehouse Base Directory.
For Amazon S3 targets with Open Table format, the data directory field is not applicable. Enabling the Connection Directory as Parent includes the connection directory before the warehouse base path. If disabled, files are saved directly under the warehouse base directory.
Schema Directory
Specify a custom directory for storing the schema file if you want to store it in a directory other than the default directory. For initial loads, previously used values if available are shown in a list for your convenience. This field is optional.
For initial loads, the schema is stored in the data directory by default. For incremental loads and combined initial and incremental loads, the schema file is stored in the following directory by default: {TaskTargetDirectory}/data/{TableName}/schema.
You can use the same placeholders as for the Data Directory field. If you manually enter placeholders, ensure that you enclose them with curly brackets { }. If you include the toUpper or toLower function, put the placeholder name in parentheses and enclose both the function and placeholder in curly brackets, for example: {toLower(SchemaName)}
Note:
Schema is written only to output data files in CSV format. Data files in Parquet and Avro formats contain their own embedded schema.
Cycle Completion Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle completed file.
Default is {TaskTargetDirectory}/cycle/completed.
Cycle Contents Directory
For incremental load and combined initial and incremental load tasks, the path to the directory that contains the cycle contents files.
Default is {TaskTargetDirectory}/cycle/contents.
Use Cycle Partitioning for Data Directory
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under each data directory.
If this option is not selected, individual data files are written to the same directory without a timestamp, unless you define an alternative directory structure.
Use Cycle Partitioning for Summary Directories
For incremental load and combined initial and incremental load tasks, select this option to create a timestamp subdirectory for each CDC cycle, under the summary contents and completed subdirectories.
List Individual Files in Contents
For incremental load and combined initial and incremental load tasks, lists individual data files under the contents subdirectory.
If Use Cycle Partitioning for Summary Directories is cleared, this option is selected by default. All of the individual files are listed in the contents subdirectory unless you can configure custom subdirectories by using the placeholders, such as for timestamp or date.
If Use Cycle Partitioning for Data Directory is selected, you can still optionally select this check box to list individual files and group them by CDC cycle.
Click Next to proceed, or click Save.

Configure a PostgreSQL target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required PostgreSQL target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source tables.
Note:
After the target table is created, Database Ingestion and Replication intelligently handles the target tables on subsequent job runs. Database Ingestion and Replication might truncate or re-create the target tables depending on the specific circumstances.
Schema
Select the target schema in which Database Ingestion and Replication creates the target tables.
Apply Mode
For incremental load and combined initial and incremental load jobs, this property indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
  • - Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
  • - Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, time, owner, transaction ID, generated ascending sequence number, and before image.
    • Consider using the Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
  • - Soft Deletes. Apply source delete operations to the target as soft deletes. A soft delete marks the deleted row as deleted without actually removing it from the database. For example, a delete on the source results in a change record on the target with "D" displayed in the INFA_OPERATION_TYPE column.
    • Consider using soft deletes if you have a long-running business process that needs the soft-deleted data to finish processing, to restore data after an accidental delete operation, or to track deleted values for audit purposes.
    Note:
    If you use Soft Deletes mode, you must not perform an update on the primary key in a source table. Otherwise, data corruption can occur on the target.
The default value is Standard.
Note:
This field does not appear if you selected Query-based as the CDC method on the Source page of the task wizard.
Click Next to proceed, or click Save.

Configure a Salesforce Data 360 target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Salesforce Data 360 target properties:
Property
Description
Schema Path
The schema path where Database Ingestion and Replication creates the target tables.
Data Space
Name of the data space in Salesforce Data 360 where you want to load data.
If you do not enter a data space, the default data space is used.
Category
The data category assigned to data from source streams when it is loaded into Salesforce Data 360.
Options are:
  • - Profile. Applies to customer attributes such as name, email, or address.
  • - Engagement. Applies to behavioral and interaction data such as website visits, email opens, clicks, and other activity logs.
  • - Other. Applies to data that doesn’t fit into Profile or Engagement, such as transactional records, order histories, CRM system logs, and other system or contextual data.
Event Time Field
Name of the event date and time field for engagement records if you selected the Engagement category. It defines exactly when an interaction occurred, such as a purchase, click, or login, and helps identify the most recent or relevant event.
Recorded Modified Field
The name of the recorded modified field that captures the date and time when a record was last updated if you selected the Profile or Other category.
Apply Mode
For incremental load and combined initial and incremental load jobs, this property indicates how source DML changes, including inserts, updates, and deletes, are applied to the target.
You can use only the Standard apply mode. This mode accumulates the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
Note:
This field does not appear if you selected Query-based as the CDC method on the Source page of the task wizard.
Click Next to proceed, or click Save.

Configure a Snowflake Data Cloud target

Define target properties for the destination that you selected on the Destination page.

Target Properties

Define the following required Snowflake target properties:
Property
Description
Target Creation
The only available option is Create Target Tables, which generates the target tables based on the source tables.
Note:
After the target table is created, Database Ingestion and Replication intelligently handles the target tables on subsequent job runs. Database Ingestion and Replication might truncate or re-create the target tables depending on the specific circumstances.
Schema
Select the target schema in which Database Ingestion and Replication creates the target tables.
Stage
The name of internal staging area that holds the data read from the source before the data is written to the target tables. This name must not include spaces. If the staging area does not exist, it will be automatically created.
Note:
This field is not available if you selected the Superpipe option in the Advanced Target Properties.
Apply Mode
For incremental load and combined initial and incremental load jobs, this property indicates how source DML changes, including inserts, updates, and deletes, are applied to the target. Options are:
  • - Standard. Accumulate the changes in a single apply cycle and intelligently merge them into fewer SQL statements before applying them to the target. For example, if an update followed by a delete occurs on the source row, no row is applied to the target. If multiple updates occur on the same column or field, only the last update is applied to the target. If multiple updates occur on different columns or fields, the updates are merged into a single update record before being applied to the target.
  • - Soft Deletes. Apply source delete operations to the target as soft deletes. A soft delete marks the deleted row as deleted without actually removing it from the database. For example, a delete on the source results in a change record on the target with "D" displayed in the INFA_OPERATION_TYPE column.
    • Consider using soft deletes if you have a long-running business process that needs the soft-deleted data to finish processing, to restore data after an accidental delete operation, or to track deleted values for audit purposes.
    Note:
    If you use Soft Deletes mode, you must not perform an update on the primary key in a source table. Otherwise, data corruption can occur on the target.
  • - Audit. Apply an audit trail of every DML operation made on the source tables to the target. A row for each DML change on a source table is written to the generated target table along with the audit columns you select under the Advanced section. The audit columns contain metadata about the change, such as the DML operation type, time, owner, transaction ID, generated ascending sequence number, and before image.
    • Consider using the Audit apply mode when you want to use the audit history to perform downstream computations or processing on the data before writing it to the target database or when you want to examine metadata about the captured changes.
The default value is Standard.
Note:
This field does not appear if you selected Query-based as the CDC method on the Source page of the task wizard.
Click Next to proceed, or click Save.