Let's configure the Open Table connection properties to connect to AWS Glue Catalog, Hive Metastore, Polaris REST catalog, or Nessie catalog.
Before you begin
Before you get started, you will need to create the minimal IAM policies to interact with Apache Iceberg or Delta Lake tables managed by AWS Glue Catalog and install the Hive JDBC driver for Hive metastore. You also need to configure the authentication-specific prerequisites to connect to Amazon S3 or Microsoft Azure Data Lake Storage Gen2 storage.
Permanent IAM Credentials authentication for Amazon S3 requires the access key and secret key values of the IAM user. EC2 Role to Assume Role authentication for Amazon S3 requires the ARN of the IAM role that the EC2 role assumes to generate temporary security credentials.
To configure Service Principal authentication for Microsoft Azure Data Lake Storage Gen2, you need the Azure account name, client secret, client ID, and tenant ID for your application registered in the Azure Active Directory.
Check out Prerequisites to learn more about how to configure policies and role to access Apache Iceberg or Delta Lake tables.
Open Table formats with associated catalog and storage types
You can choose the Open Table format that you want to use and its associated catalog type and storage type to interact with data.
The following table summarizes the Open Table formats that you can use, their catalog types, storage types, and the authentication options available for each storage type:
Open Table format
Catalog type
Catalog authentication type
Storage type
Storage authentication type
Apache Iceberg
AWS Glue Catalog*
None
Amazon S3*
- AWS Permanent IAM Credentials authentication
- EC2 Role to Assume Role
Hive Metastore
None
Amazon S3
AWS Permanent IAM Credentials authentication
Hive Metastore
None
Microsoft Azure Data Lake Storage Gen2
Azure Service Principal authentication
REST Catalog*
OAuth 2.0 Credentials
Amazon S3*
- AWS Permanent IAM Credentials authentication
- Credential Vending
REST Catalog
OAuth 2.0 Credentials
Microsoft Azure Data Lake Storage Gen2
- Azure Service Principal authentication
- Credential Vending
Nessie Catalog*
OAuth 2.0 Credentials
MinIO*
MinIO Permanent IAM Credentials authentication
Delta Lake
AWS Glue Catalog
None
Amazon S3
AWS Permanent IAM Credentials authentication
*Open Table formats with catalog and storage types that apply to both mappings and mappings in advanced mode.
The other catalog and storage types apply only to mappings in advanced mode.
Connection details
The following table describes the Open Table connection properties:
Property
Description
Connection Name
Name of the connection.
Each connection name must be unique within the organization. Connection names can contain alphanumeric characters, spaces, and the following special characters: _ . + -,
Maximum length is 255 characters.
Description
Description of the connection. Maximum length is 4000 characters.
Use Secret Vault
Stores sensitive credentials for this connection in the secrets manager that is configured for your organization.
This property appears only if secrets manager is set up for your organization.
When you enable the secret vault in the connection, you can select the credentials that the Secure Agent retrieves from the secrets manager. If you don't enable this option, the credentials are stored in the repository or on a local Secure Agent, depending on how your organization is configured.
Note:
If you’re using this connection to apply data access policies through pushdown or proxy services, you cannot use the Secret Vault configuration option.
The Open Table format that you want to use to read from or write data to a catalog.
Select Apache Iceberg or Delta Lake from the list.
Catalog types
You can select AWS Glue Catalog, Hive Metastore, REST Catalog, or Nessie catalog as the catalog type to manage the metadata of the Open Table format that you selected.
Select the catalog type that your Open Table format uses and then configure the catalog specific parameters.
AWS Glue Catalog
If the Apache Iceberg or Delta Lake Open Table format uses AWS Glue Catalog as the catalog type, configure the properties specific to AWS Glue Catalog.
The following table describes the property to configure AWS Glue Catalog:
For example, jdbc:athena://Region=us-west1;OutputLocation=s3://working/dir.
Catalog Authentication Type
The authentication method to connect to the catalog.
Select one of the following options:
- None. Connects to AWS Glue Catalog or Hive Metastore without any authentication credentials.
- OAuth 2.0 Client Credentials. Connects to a REST catalog or Nessie catalog using a Client ID and Client Secret to obtain an access token from the authorization server.
Hive Metastore
If the Apache Iceberg Open Table format uses Hive Metastore as the catalog type, configure the properties specific to Hive Metastore.
The following table describes the properties to configure Hive Metastore:
Property
Description
Hive Metastore URI
The Hive thrift server URL to connect to Hive Metastore.
Hive JDBC URL
The JDBC URL to connect to Hive4 server.
Hive User Name
The user name of your Hive account to connect to Hive Metastore.
Hive Password
The password of your Hive account to connect to Hive Metastore.
Catalog Authentication Type
The authentication method to connect to the catalog.
Select one of the following options:
- None. Connects to AWS Glue Catalog or Hive Metastore without any authentication credentials.
- OAuth 2.0 Client Credentials. Connects to a REST catalog or Nessie catalog using a Client ID and Client Secret to obtain an access token from the authorization server.
REST Catalog
If the Apache Iceberg Open Table format uses REST catalog as the catalog type, configure the properties specific to REST Catalog.
The following table describes the properties to configure REST catalog:
Property
Description
REST Catalog Type
The type of REST catalog that you want to connect to.
Select Polaris Catalog.
Catalog Endpoint URL
The endpoint URL of the REST catalog.
For example, http://35.98.240.22:8181.
Catalog Authentication Type
The authentication method to connect to the catalog.
Select one of the following options:
- None. Connects to an AWS Glue Catalog or a Hive Metastore without any authentication credentials.
- OAuth 2.0 Client Credentials. Connects to a REST catalog or Nessie catalog using a client ID and client secret to obtain an access token from the OAuth 2.0 authorization server.
Access Token URL
The URL provided by the OAuth 2.0 authorization server to obtain an access token.
For example, http://35.98.240.22:8181/api/catalog/v1/oauth/tokens.
Client ID
The client ID of OAuth 2.0 authorization server that is registered with the catalog.
Client Secret
The client secret of OAuth 2.0 authorization server that is registered with the catalog.
Scope
The scope parameters that define the permissions an access token grants to the catalog.
For example, PRINCIPAL_ROLE:ALL.
Nessie Catalog
If the Apache Iceberg Open Table format uses Nessie catalog as the catalog type, configure the properties specific to Nessie Catalog.
The following table describes the properties to configure Nessie catalog:
Property
Description
Nessie URI
The base URI of the Nessie REST API endpoint.
For example, http://3.8.140.58:18120/api/v2.
Catalog Authentication Type
The authentication method to connect to the catalog.
Select one of the following options:
- None. Connects to an AWS Glue Catalog or a Hive Metastore without any authentication credentials.
- OAuth 2.0 Client Credentials. Connects to a REST catalog or Nessie catalog using a client ID and client secret to obtain an access token from the OAuth 2.0 authorization server.
Access Token URL
The URL provided by the OAuth 2.0 authorization server to obtain an access token.
For example, https://in.okta.com/oauth2/default/v1/token.
Client ID
The client ID of OAuth 2.0 authorization server that is registered with the catalog.
Client Secret
The client secret of OAuth 2.0 authorization server that is registered with the catalog.
Scope
The scope parameters that define the permissions an access token grants to the catalog.
For example, nessieopentablepyn.access.
Storage types
You can choose Amazon S3 , Microsoft Azure Data Lake Storage Gen2, or MinIO as the storage type to store the Open Table format tables.
Select the storage type and configure the storage specific authentication parameters.
Amazon S3
If you use AWS Glue Catalog, Hive Metastore, or REST Catalog as the catalog type, configure the properties specific to Amazon S3 storage.
AWS Permanent IAM Credentials authentication
You can use AWS Permanent IAM Credentials authentication for Amazon S3 storage when you connect to an AWS Glue Catalog, Hive Metastore, or REST Catalog.
The following table describes the properties to configure AWS Permanent IAM Credentials authentication:
Property
Description
Access Key
The key to access the AWS Glue Catalog.
Secret Key
The secret key to access the AWS Glue Catalog. The secret key is associated with the access key and uniquely identifies the account.
AWS EC2 Role to Assume Role authentication
You can use AWS EC2 Role to Assume Role authentication for Amazon S3 storage only when you read Apache Iceberg tables from AWS Glue Catalog.
The following table describes the properties to configure AWS EC2 Role to Assume Role authentication:
Property
Description
IAM Role ARN
The ARN of the IAM role assumed by the EC2 role to generate the temporary session credentials.
External ID
A unique, user-defined string value that the IAM role requires the EC2 role to provide when calling the sts:AssumeRole API.
Credential Vending
You can use credential vending for Amazon S3 storage when you connect to a REST Catalog.
Credential vending determines whether the storage for the REST catalog requires authentication. When you select credential vending, the REST catalog automatically generates temporary session credentials and handles storage access. You do not need to provide the storage credentials separately.
Microsoft Azure Data Lake Storage Gen2
If you use Hive Metastore or REST Catalog as the catalog type, configure the properties specific to Microsoft Azure Data Lake Storage Gen2.
Azure Service Principal authentication
The following table describes the properties to configure Service Principal authentication:
Property
Description
Azure Account Name
The name of the Microsoft Azure Data Lake Storage Gen2 account to stage the files.
Azure Client ID
The client ID of your application.
Enter the application ID or client ID for your application registered in the Azure Active Directory.
Azure Client Secret
The client secret for your application.
Azure Tenant ID
The directory ID or tenant ID for your application.
Credential Vending
You can use credential vending for Microsoft Azure Data Lake Storage Gen2 when you connect to a REST Catalog.
Credential vending determines whether the storage for the REST catalog requires authentication. When you select credential vending, the REST catalog automatically generates temporary session credentials and handles storage access. You do not need to provide the storage credentials separately.
MinIO
If you use Nessie catalog as the catalog type, configure the properties specific to MinIO.
Select MinIO Permanent IAM Credentials authentication as the authentication type to access Open Table formats in MinIO.
MinIO Permanent IAM Credentials authentication
The following table describes the properties to configure MinIO Permanent IAM Credentials authentication:
Property
Description
Endpoint URL
The HTTP or HTTPS URL of the MinIO storage endpoint.
For example, http://3.8.140.58:9000.
Access Key
Access key ID that uniquely identifies the IAM user credentials to access MinIO.
Secret Key
Secret access key that authenticates the access key ID to securely access MinIO.
Advanced settings
The following table describes the advanced connection properties:
Property
Description
Warehouse Location
The path to the warehouse in MinIO where the Nessie catalog stores the Iceberg tables.
For example, s3://warehouse/.
Additional Properties
Additional properties to configure the MinIO storage.
The default properties are client.region=us-east-1 and s3.path-style-access=true.
Do not edit or delete the s3.path-style-access=true property.
You can edit the client.region property and set the value to the MinIO region for your catalog.
Enter new properties as key=value pairs separated by semicolons.