The November 2024 release of Data Ingestion and Replication includes the following changed behaviors.
New minimum value for Merge Frequency for the Superpipe option in Snowflake targets
For incremental load and combined initial and incremental load tasks that have a Snowflake Data Cloud target and use the Superpipe option, the minimum interval at which the change data rows are merged and applied to the Snowflake tables has been decreased from 300 seconds to 60 seconds. Use the Merge Frequency field on the Target page of the task wizard to set the interval. Valid values are 60 through 604800.
Incremental filter for the DATE data type columns for Oracle Fusion sources
For application ingestion and replication incremental load jobs and combined load jobs with an Oracle Fusion source run successfully when the datastore contains columns with the DATE data type marked as either Incremental Filter or Is Effective Start Date.
Previously, only tasks with the TIMESTAMP data type columns marked as Incremental Filter ran successfully.
Updated file name format for Microsoft Azure Data Lake Storage Gen2 targets
In a streaming ingestion and replication job with a Microsoft Azure Data Lake Storage Gen2 target using the rollover strategy, file names in the target directory now include milliseconds in the timestamp in the following format:
Previously, the file names didn't include the milliseconds.
Database Ingestion agent service property for job load types
When you configure the Database Ingestion service for a Secure Agent in Administrator, you can set a new property to specify the job load types that the agent service can run. Under Sytem Configuration Details, select Database Ingestion as the service and DBMI_AGENT_CONFIG as the type. Then set the supportedLoadTypes property to one or more of the following values, using a comma (,) as the separator:
•INITIAL. Initial load jobs or the initial load phase of combined initial and incremental load jobs.
•INCREMENTAL. Incremental load jobs or the incremental phase of combined initial and incremental load jobs, which write to your target..
•INCREMENTAL_STAGING. CDC staging tasks of incremental load or combined load jobs.
Default is INITIAL,INCREMENTAL,INCREMENTAL_STAGING, which indicates all load types.