You can connect to various endpoints from Databricks Notebook to read from or write data. You also have the option to configure Databricks as an intermediate storage which helps you use INFACore seamlessly in Databricks Notebook. When you integrate INFACore with Databricks enabled for unity catalog, you can access and work with the metadata and data assets stored in the unity catalog directly from your Databricks Notebook.
For use cases that require staging of data, you can use the Databricks environment to stage the data temporarily. INFACore creates the Databricks native DataFrame for these use cases.
When you configure a Databricks environment for intermediate staging, you can use an agent installed remotely in a Cloud computing service or a virtual machine to stage the data in the Databricks environment.
You need to specify the compute engine type as Remote, provide the runtime environment name, and the Databricks Delta connection name in the Python SDK. INFACore uses the configured runtime environment to stage the data in Databricks using the specified Databricks Delta connection.