Changed behavior

The November 2024 release of Data Ingestion and Replication includes the following changed behaviors.

New minimum value for Merge Frequency for the Superpipe option in Snowflake targets

For incremental load and combined initial and incremental load tasks that have a Snowflake Data Cloud target and use the Superpipe option, the minimum interval at which the change data rows are merged and applied to the Snowflake tables has been decreased from 300 seconds to 60 seconds. Use the Merge Frequency field on the Target page of the task wizard to set the interval. Valid values are 60 through 604800.

Incremental filter for the DATE data type columns for Oracle Fusion sources

For application ingestion and replication incremental load jobs and combined load jobs with an Oracle Fusion source run successfully when the datastore contains columns with the DATE data type marked as either Incremental Filter or Is Effective Start Date.

Previously, only tasks with the TIMESTAMP data type columns marked as Incremental Filter ran successfully.

Updated file name format for Microsoft Azure Data Lake Storage Gen2 targets

In a streaming ingestion and replication job with a Microsoft Azure Data Lake Storage Gen2 target using the rollover strategy, file names in the target directory now include milliseconds in the timestamp in the following format:

Previously, the file names didn't include the milliseconds.

Database Ingestion agent service property for job load types

When you configure the Database Ingestion service for a Secure Agent in Administrator, you can set a new property to specify the job load types that the agent service can run. Under Sytem Configuration Details, select Database Ingestion as the service and DBMI_AGENT_CONFIG as the type. Then set the supportedLoadTypes property to one or more of the following values, using a comma (,) as the separator:

Default is INITIAL,INCREMENTAL,INCREMENTAL_STAGING, which indicates all load types.