Job cluster
Configure the Spark parameters for job cluster to use Azure and AWS staging based on where the cluster is deployed.
You also need to enable the Secure Agent properties for runtime processing on the job cluster.
You can use job cluster only on the Linux operating system.
Configure Spark parameters
Before you connect to the job cluster, you must configure the Spark parameters on AWS and Azure.
Configuration on AWS
Add the following Spark configuration parameters for the job cluster and restart the cluster:
- •spark.hadoop.fs.s3a.access.key <value>
- •spark.hadoop.fs.s3a.secret.key <value>
- •spark.hadoop.fs.s3a.endpoint <value>
Ensure that the access and secret key configured has access to the buckets where you store the data for Databricks tables.
Configuration on Azure
Add the following Spark configuration parameters for the job cluster and restart the cluster:
- •fs.azure.account.oauth2.client.id.<storage-account-name>.dfs.core.windows.net <value>
- •fs.azure.account.auth.type.<storage-account-name>.dfs.core.windows.net <value>
- •fs.azure.account.oauth2.client.secret.<storage-account-name>.dfs.core.windows.net <Value>
- •fs.azure.account.oauth.provider.type.<storage-account-name>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
- •fs.azure.account.oauth2.client.endpoint.<storage-account-name>.dfs.core.windows.net https://login.microsoftonline.com/<Tenant ID>/oauth2/token
Ensure that the client ID and client secret configured has access to the file systems where you store the data for Databricks tables.
Configure Secure Agent properties
To connect to job cluster, enable the Secure Agent properties for runtime.
- 1In Administrator, select the Secure Agent listed on the Runtime Environments tab.
- 2Click Edit.
- 3In the System Configuration Details section, select Data Integration Server as the Service and DTM as the Type.
- 4Edit the JVMOption field and set the value to -DUseDatabricksSql=false.
