Data Engineering
Read this section to learn what's new for Data Engineering in version 10.5.2.
Databricks
Read this section to learn about new features that support Databricks functionality.
Delta Lake Schema Evolution
Effective in version 10.5.2, you can configure dynamic mappings to apply Databricks source schema changes to Delta Lake targets. You can choose from among three different strategies to update Delta Lake target tables when a source schema changes. Mappings succeed with merge enabled in the mapping and schema enforcement disabled in the Databricks workspace.
For more information, see the following documentation:
Use Databricks and Custom Parameters to Configure Ephemeral Clusters
Effective in version 10.5.2, you can use custom parameters and Databricks parameters in a JSON file to configure ephemeral clusters when you use a cluster creation workflow. You can use the JSON file, which is easily edited, instead of specifying parameter values in the Create Cluster task properties.
For more information, see cluster workflow documentation in the Data Engineering Integration User Guide.
Unmanaged Databricks Table Targets
Effective in version 10.5.2, you can configure mappings to write to unmanaged Databricks table targets.
For more information, see the Data Engineering Integration User Guide
Data Transformation
Effective in version 10.5.2, you can configure the ExcelToDataXml document processor to add elements to the output when a source table contains empty cells in the middle or at the end of table rows.
For more information, see the Data Transformation User Guide.
Kudu Dynamic Mapping
Effective in version 10.5.2, you can enter a comma-separated list of column names that you want to be primary keys when you create a Kudu object using the dynamic mapping flow option.
For more information, see the Data Engineering Streaming User Guide.