Changed behavior

For application ingestion tasks that have a Salesforce source, you can use the Chunk Size field in the Source page to specify the number of records to be run in parallel. Chunk size can improve memory utilization, reduce the need for a larger JVM requirement, and run more parallel partitions.

Previously, the partitions were entered in the Number of Partitions field. This field is no longer available.

When you configure an application ingestion task with any source and a Snowflake target with Soft Deletes apply mode, any update in the source table during normal or backlog mode now results in the deletion of the matching record, insertion of the updated record, and marking of the INFA_OPERATION_TYPE operation as NULL in the target table. Similarly, inserting a record in the source table during backlog mode results in deletion of the matching record, insertion of the updated record, and marking of the INFA_OPERATION_TYPE operation as NULL in the target table.

Previously, updating or inserting records in the source table resulted in updating the existing record and marking of the INFA_OPERATION_TYPE operation as D and NULL in the new record in the target table.

When you make a schema change to a data object in an application ingestion incremental load job that have an SAP S4 HANA source, the SAP Mass Ingestion connector now generates an alert message. When the alert appears, the processing of the data object stops, and you must create a new task to perform an initial load and then an incremental load with the changed data object to apply the schema changes, depending on the job status.

Previously, no alert messages were generated, and the schema changes were ignored.

When you define a combined initial and incremental load database ingestion task, the Initial Start Point for Incremental Load is no longer available under the Advanced properties on the Source page of the task wizard. This field is not displayed for the combined initial and incremental load jobs because for this load type, initial loading is not performed until the incremental processing of change data reaches the end of the current transaction log. Consequently, the user input for the Initial Start Point for Incremental Load field is irrelevant.

You can now customize the Initial Start Point for Incremental Load field for incremental load jobs only.

If you enable the Soft Deletes apply mode for application ingestion and database ingestion tasks that have Databricks Delta targets, any source updates or inserts are now applied to the target with NULL shown in the INFA_OPERATION_TYPE column and in any other "Add Operation..." metadata columns you selected on the Target page. For deletes, "D" is shown in the INFA_OPERATION_TYPE column and non-null values are written to the other metadata columns you selected. The soft deletes behavior for Databricks Delta is now consistent with the behavior for Snowflake.

Previously, for the unload phase of a combined job, all DML changes caused the INFA_OPERATION_TYPE column and all selected metadata columns to be populated with values. For the incremental CDC phase, inserts and updates caused all metadata fields except INFA_OPERATION_TYPE to be populated, and deletes caused all metadata fields including INFA_OPERATION_TYPE to be populated.

If you want to use Windows authentication to connect to SQL Server, select Windows Authentication v2 in the Authentication Mode property when you define SQL Server connection properties. If you previously selected Windows Authentication (Deprecated) for SQL Server connections, switch to Windows Authentication v2 to avoid potential errors.