Read Modes
When you use Google BigQuery Connector, you can read data by using direct mode or staging mode. Before you choose a mode, see the Google documentation to understand the cost implications and trade-offs for each mode.
You can read data from a Google BigQuery source by using one of the following modes:
- Direct Mode
- Use direct mode when the volume of data that you want to read is small. In direct mode, the Secure Agent directly reads data from a Google BigQuery source. You can configure the number of rows that you want the Secure Agent to read in one request.
- Staging Mode
- Use staging mode when you want to read large volumes of data in a cost-efficient manner.
In staging mode, the Secure Agent first downloads the data from the Google BigQuery source into a local stage file in Google Cloud Storage. You can configure the local stage file directory in the advanced source properties. The Secure Agent then reads the data from the local stage file.
When you enable staging file compression, the Secure Agent compresses the size of the staging file in Google Cloud Storage. The Secure Agent then downloads the staging file and decompresses the staging file before it reads the file. To improve the performance and download data in parallel, you can configure the number of threads for downloading the staging file.