Open Table format | Catalog type | Catalog authentication type | Storage type | Storage authentication type |
---|---|---|---|---|
Apache Iceberg | AWS Glue Catalog | None | Amazon S3 |
|
Hive Metastore | None | Amazon S3 | Permanent IAM Credentials authentication | |
Hive Metastore | None | Microsoft Azure Delta Lake Storage Gen2 | Service Principal authentication | |
REST Catalog | OAuth 2.0 Credentials | Amazon S3 | Permanent IAM Credentials authentication | |
Delta Lake | AWS Glue Catalog | None | Amazon S3 | Permanent IAM Credentials authentication |
Property | Description |
---|---|
Connection Name | Name of the connection. Each connection name must be unique within the organization. Connection names can contain alphanumeric characters, spaces, and the following special characters: _ . + -, Maximum length is 255 characters. |
Description | Description of the connection. Maximum length is 4000 characters. |
Type | Open Table |
Use Secret Vault | Stores sensitive credentials for this connection in the secrets manager that is configured for your organization. This property appears only if secrets manager is set up for your organization. This property is not supported by Data Ingestion and Replication. When you enable the secret vault in the connection, you can select which credentials that the Secure Agent retrieves from the secrets manager. If you don't enable this option, the credentials are stored in the repository or on a local Secure Agent, depending on how your organization is configured. For information about how to configure and use a secrets manager, see Secrets manager configuration. |
Runtime Environment | The name of the runtime environment where you want to run tasks. Select a Secure Agent, Hosted Agent, or serverless runtime environment. You cannot run a database ingestion task on a Hosted Agent or in a serverless runtime environment. |
Open Table Format | The Open Table format that you want to use to read from or write data to a catalog. Select Apache Iceberg or Delta Lake from the list. |
Property | Description |
---|---|
Athena JDBC URL | Enter the JDBC URL in the following format: jdbc:athena://Region=<AWS_Region>;OutputLocation=<S3_Location> For example, jdbc:athena://Region=us-west1;OutputLocation=s3://working/dir. |
Catalog Authentication Type | The authentication method to connect to the catalog. Select one of the following options:
|
Property | Description |
---|---|
Hive Metastore URI | The Hive thrift server URL to connect to Hive Metastore. |
Hive JDBC URL | The JDBC URL to connect to Hive4 server. |
Hive User Name | The user name of your Hive account to connect to Hive Metastore. |
Hive Password | The password of your Hive account to connect to Hive Metastore. |
Catalog Authentication Type | The authentication method to connect to the catalog. Select one of the following options:
|
Property | Description |
---|---|
REST Catalog Type | The type of REST catalog that you want to connect to. Select Polaris Catalog. |
Catalog Endpoint | The endpoint URL of the REST catalog. |
Catalog Authentication Type | The authentication method to connect to the catalog. Select one of the following options:
|
Access Token URL | The URL provided by the OAuth 2.0 authorization server to obtain an access token. |
Client ID | The client ID of the REST endpoint registered with the OAuth 2.0 authorization server. |
Client Secret | The client secret of the REST endpoint registered with the OAuth 2.0 authorization server. |
Scope | The scope parameters that define the permissions an access token grants to the REST endpoint. |
Credential Vending | Determines if the storage for the REST catalog requires authentication. If credential vending is enabled, it indicates that the REST catalog is configured to automatically generate the temporary credentials to access the associated storage. You do not need to provide the storage credentials separately. If credential vending is disabled, it indicates that you need to provide the storage credentials separately. When credential vending is disabled, temporary staging directory is not deleted from table storage location for the update, upsert, and delete operations. |
Property | Description |
---|---|
Access Key | The key to access the AWS Glue Catalog. |
Secret Key | The secret key to access the AWS Glue Catalog. The secret key is associated with the access key and uniquely identifies the account. |
Property | Description |
---|---|
EC2 Role | The ARN of the IAM role assumed by the EC2 role to generate the temporary session credentials. |
External ID | A unique, user-defined string value that the IAM role requires the EC2 role to provide when calling the sts:AssumeRole API. |
Property | Description |
---|---|
Azure Account Name | The name of the Microsoft Azure Data Lake Storage Gen2 account to stage the files. |
Azure Client ID | The client ID of your application. Enter the application ID or client ID for your application registered in the Azure Active Directory. |
Azure Client Secret | The client secret for your application. |
Azure Tenant ID | The directory ID or tenant ID for your application. |