Step 2. Specify the remote agent engine details for Databricks Notebook
After installing the SDK and logging in to INFACore from Databricks Notebook, configure Databricks as an intermediate storage.
Perform the following tasks in the INFACore SDK Python code to configure Databricks as an intermediate storage:
1Enter the compute engine type as Remote and the runtime environment name.
2Enter the ecosystem type and the configured Databricks Delta connection name.
Here is an example Python code snippet that demonstrates how to use INFACore with the remote agent details in a Databricks Notebook environment:
user_config = {
"login": {
"username": "infacore",
"password": "<password>",
"env_type": "test",
"url": "https://dmr-us.informaticacloud.com"
},
"compute_engine": {
"type": "remote",
"runtime_env_name": "<abc01.infa.com>",
"install": False
},
"ecosystem" : {
"type": "Databricks Delta",
"staging" :{
"connection_name" : "Databricks_AWS"
}
}
}
You can perform various operations or access functionalities that INFACore offers within your Databricks Notebook.
When you run the code, INFACore establishes a connection with Databricks Delta using the specified runtime environment. It uses the connection to stage the data in Databricks when you read from or write to the configured Databricks Delta endpoint.
For information on available INFACore functionalities and how to use specific methods and parameters, see the SDK Reference for Python documentation.