The November 2023 release of INFACore includes the following new features and enhancements for the INFACore Python SDK.
Unity catalog support with Databricks integration
When you integrate INFACore with Databricks enabled for unity catalog, you can access and work with the metadata and data assets stored in the unity catalog directly from your Databricks Notebook.
You can specify a unity catalog name in a Databricks Delta connection to read from or write to tables in unity catalogs in Databricks using INFACore from any integrated development environment. You can specify the catalog name in a Databricks Delta connection either through the SQL Warehouse JDBC URL or the Catalog Name field.
Reuse user-defined functions
You can reuse the transformation logic from user-defined functions (UDFs) configured in your Informatica Intelligent Cloud Services organization. In this release, you can leverage mapplets as UDFs in INFACore. Get the list of UDFs from your organization and transform the data using INFACore. You can also chain UDFs to generate a data pipeline dynamically and run the pipeline.
Bring your own data pipelines
You can reuse data pipelines such as parameterized mapping templates created in your Informatica Intelligent Cloud Services organization. You can pass native data frames and other parameter values in these mapping templates in INFACore and run multiple transformations using a single line of code. You can combine custom pipelines with native DataFrame functions, allowing you to mix and match and achieve the best solution for your use case.
INFACore integration with the Microsoft Fabric ecosystem
You can use INFACore within the Microsoft Fabric ecosystem to connect to various endpoints to leverage INFACore functionality. For use cases that require intermediate staging of data, you have the option to configure a Microsoft Fabric environment to stage the data temporarily using the Microsoft Fabric OneLake connection. INFACore creates the Microsoft Fabric pyspark DataFrame for these use cases.