Property | Description |
|---|---|
Connection Name | Name of the connection. Each connection name must be unique within the organization. Connection names can contain alphanumeric characters, spaces, and the following special characters: _ . + -, Maximum length is 255 characters. |
Description | Description of the connection. Maximum length is 4000 characters. |
Type | Google BigQuery V2 |
Use Secret Vault | Stores sensitive credentials for this connection in the secrets manager that is configured for your organization. This property appears only if secrets manager is set up for your organization. This property is not supported by Data Ingestion and Replication. When you enable the secret vault in the connection, you can select which credentials that the Secure Agent retrieves from the secrets manager. If you don't enable this option, the credentials are stored in the repository or on a local Secure Agent, depending on how your organization is configured. For information about how to configure and use a secrets manager, see Secrets manager configuration. |
Runtime Environment | The name of the runtime environment where you want to run tasks. Select a Secure agent, Hosted Agent, or serverless runtime environment. You cannot run an application ingestion and replication task, database ingestion and replication task, or streaming ingestion and replication task on a Hosted Agent or serverless runtime environment. |
Property | Description |
|---|---|
Service Account Email | The client_email value from the Google service account key JSON file. |
Service Account Key | The private_key value from the Google service account key JSON file. To use the private key from the JSON file, enter the entire block of text found within the double quotation marks in the private_key field, starting with -----BEGIN PRIVATE KEY-----\n and ending at -----END PRIVATE KEY-----\n. |
Project ID | The project_id value from the Google service account key JSON file. If you have created multiple projects with the same service account, enter the ID of the project that contains the dataset that you want to connect to. |
Property | Description |
|---|---|
Enable BigQuery Storage API | Uses Google BigQuery Storage to stage the files when you read or write data. Default is unselected. |
Storage Path | Path in Google Cloud Storage where the agent creates a local stage file to store the data temporarily. The agent uses this storage when it reads data in staging mode or writes data in bulk mode. Use one of the following formats:
When you enable cross-region replication in Google BigQuery, enter a Google Cloud Storage path that supports dual region storage. This property is not applicable if you use Google BigQuery Storage to stage the files. |
Connection Mode | The mode that you want to use to read data from or write data to Google BigQuery. Select one of the following connection modes:
Default is Simple. This property is applicable if you use Google Cloud Storage to stage the files. |
Use Legacy SQL for Custom Query¹ | Select this option to use legacy SQL to define a custom query. If you clear this option, use standard SQL to define a custom query. This property is applicable if you use Google Cloud Storage to stage the files. This property doesn't apply if you configure the Google BigQuery V2 connection in hybrid or complex mode. |
Dataset Name for Custom Query¹ | The name of the Google BigQuery dataset that you want to specify in a custom query. |
Schema Definition File Path¹ | Directory on the Secure Agent machine where the Secure Agent creates a JSON file with the sample schema of the Google BigQuery table. The JSON file name is the same as the Google BigQuery table name. Alternatively, you can specify a storage path in Google Cloud Storage where the Secure Agent creates a JSON file with the sample schema of the Google BigQuery table. You can download the JSON file from the specified storage path in Google Cloud Storage to a local machine. The schema definition file is required if you configure complex connection mode in the following scenarios:
When you use a serverless runtime environment, specify a storage path in Google Cloud Storage. This property is applicable if you use Google Cloud Storage to stage the files. |
Region ID | The region name where the Google BigQuery dataset that you want to access resides. Note: Ensure that you specify a bucket name or the bucket name and folder name in the Storage Path property that resides in the specified region. For more information about the regions supported by Google BigQuery, see Dataset locations. |
Staging Dataset¹ | The Google BigQuery dataset name where you want to create the staging table to stage the data. You can define a Google BigQuery dataset that is different from the source or target dataset. This property is applicable if you use Google Cloud Storage to stage the files. |
Provide Optional Properties¹ | Comma-separated key-value pairs of custom properties in the Google BigQuery V2 connection to configure certain source and target functionalities. For more information about the list of custom properties that you can specify, see Optional Properties configuration Knowledge Base. |
¹ Doesn't apply to mappings in advanced mode. | |
Property | Description |
|---|---|
Project Number | The unique numeric identifier of the project in the Google service account that contains the dataset that you want to connect to. |
Authorization URL | The endpoint URL provided by the identity provider where user authentication requests are redirected to initiate the authorization process. |
Access Token URL | The endpoint URL provided by the identity provider where the client exchanges authorization codes or credentials to obtain access tokens. |
Client Id | A unique identifier assigned to your application by the identity provider. This ID is used to recognize the requester during authentication and authorization processes. |
Client Secret | A confidential key issued alongside the client ID, acting as a password between your application and the identity provider to prove the authenticity of the client during token exchanges. |
Pool Id | The identifier for the Workload Identity Pool configured in Google Cloud Platform which aggregates trusted external identities. |
Provider Id | The ID of the specific external identity provider configured within the Workload Identity Pool. |
Access Token | The access token granted by Google Cloud Platform to securely access Google BigQuery resources. Click Generate Access Token to generate a new access token. |
Property | Description |
|---|---|
Enable BigQuery Storage API | Uses Google BigQuery Storage to stage the files when you read or write data. Default is unselected. |
Storage Path | Path in Google Cloud Storage where the agent creates a local stage file to store the data temporarily. The agent uses this storage when it reads data in staging mode or writes data in bulk mode. Use one of the following formats:
When you enable cross-region replication in Google BigQuery, enter a Google Cloud Storage path that supports dual region storage. This property is not applicable if you use Google BigQuery Storage to stage the files. |
Dataset Name for Custom Query | The name of the Google BigQuery dataset that you want to specify in a custom query. |
Region ID | The region name where the Google BigQuery dataset that you want to access resides. Note: Ensure that you specify a bucket name or the bucket name and folder name in the Storage Path property that resides in the specified region. For more information about the regions supported by Google BigQuery, see Dataset locations. |
Provide Optional Properties | Comma-separated key-value pairs of custom properties in the Google BigQuery V2 connection to configure certain source and target functionalities. For more information about the list of custom properties that you can specify, see Optional Properties configuration Knowledge Base article. |
Property | Description |
|---|---|
Enable Retry¹ | Select this option if you want the Secure Agent to attempt a retry to receive the response from the Google BigQuery endpoint. You can configure the retry strategy to read data from Google BigQuery in direct or staging mode and write data to Google BigQuery in bulk mode. The retry strategy is not applicable in the CDC and streaming modes when you write data to a Google BigQuery target. The connection retry option also applies to a connection configured to use the proxy server to connect to the endpoint. Default is unselected. |
Maximum Retry Attempts | Appears only if you select the Enable Retry property. The maximum number of retry attempts that the Secure Agent performs to receive the response from the Google BigQuery endpoint. If the Secure Agent fails to connect to Google BigQuery within the maximum retry attempts, the connection fails. Default is 6 attempts. |
Initial Retry Delay | Appears only if you select the Enable Retry property. The initial wait time in seconds before the Secure Agent attempts to retry the connection. Default is 1 second. |
Retry Delay Multiplier | Appears only if you select the Enable Retry property. The multiplier that the Secure Agent uses to exponentially increase the wait time between successive retry attempts up to the maximum retry delay time. Default multiplier is 2.0. You can also use fractional values. |
Maximum Retry Delay | Appears only if you select the Enable Retry property. The maximum wait time in seconds that the Secure Agent waits between successive retry attempts. Default is 32 seconds. |
Total Timeout | Appears only if you select the Enable Retry property. The total time duration in seconds that the Secure Agent attempts to retry the connection after which the connection fails. Default is 50 seconds. |
¹ Doesn't apply to mappings in advanced mode. | |