After you complete the prerequisites, install the INFACore SDK in the configured development environment. You need to specify the remote Secure Agent details and the Databricks ecosystem for intermediate storage.
After you configure these details, you can use Databricks Connect to connect to a remote Databricks cluster from your development environment to run INFACore jobs. You can perform various operations or access functionalities that INFACore offers within your development environment.
For information on available INFACore functionalities and how to use specific methods and parameters, see the INFACore SDK Reference for Python.
Step 1. Install INFACore in your development environment
Go to Informatica Marketplace and download and install the Informatica-infacore Python SDK package in your development environment, for example in VSCode, PyCharm, and JupyterLab.
For more information about the INFACore Python SDK installation steps, see the INFACore Installation and the SDK Reference for Python documentation.
Step 2. Specify the remote agent engine details and the intermediate staging type
In your development environment, set the remote agent runtime details. Also, set Databricks as an intermediate storage with Databricks Delta as the connection. INFACore uses the connection to stage the data in the development environment when you read from or write data to an endpoint.
Perform the following tasks in the INFACore SDK Python code in your development environment to set the remote agent and use Databricks as an intermediate storage. :
1Enter the compute engine type as Remote and the runtime environment name.
2Enter the ecosystem type and the configured Databricks Delta connection name.
Here is an example Python code snippet that demonstrates how to use INFACore with the remote agent details in a Databricks environment:
user_config = {
"login": {
"username": "infacore",
"password": "<password>",
"env_type": "test",
"url": "https://dmr-us.informaticacloud.com"
},
"compute_engine": {
"type": "remote",
"runtime_env_name": "<abc01.infa.com>",
"install": False
},
"ecosystem" : {
"type": "Databricks Delta",
"staging" :{
"connection_name" : "Databricks_AWS"
}
}
}
When you run the code, INFACore establishes a connection with Databricks Delta using the specified runtime environment. It uses the connection to stage the data in Databricks when you read from or write to the configured endpoint.