Catalog Source Configuration > Google BigQuery > Introduction to Google BigQuery catalog sources
  

Introduction to Google BigQuery catalog sources

You can use Metadata Command Center to extract metadata from a source system.
A source system is any system that contains data or metadata. For example, Google BigQuery is a source system from which you can extract metadata through a Google BigQuery catalog source. A catalog source is an object that represents and contains metadata from the source system.
Before you extract metadata from a source system, you first create and register a catalog source that represents the source system. Then you configure capabilities for the catalog source. A capability is a task that Metadata Command Center can perform, such as metadata extraction, data profiling, data classification, or glossary association.
When Metadata Command Center extracts metadata, Data Governance and Catalog displays the extracted metadata and its attributes as technical assets. You can then perform tasks such as analyzing the assets, viewing lineage, and creating links between those assets and their business context.
The following table describes the capabilities of the catalog source:
Capability
Description
Serverless Runtime Environment
A serverless runtime environment is an advanced serverless deployment solution that doesn't require downloading, installing, configuring, or maintaining a Secure Agent or Secure Agent group. You can use a serverless runtime environment in the same way that you use a Secure Agent when you configure a catalog source.
Data Profiling and Quality
  • - Data Profiling. Assesses source metadata and analyzes the collected statistics to discover content and structure, such as value distribution, patterns, and data types.
  • - Data Quality. Measures the reliability of the data and enables data usage.
  • - Data Observability. Identifies anomalies in the characteristics of the data.
Data Classification
Data classification is the process of identifying and organizing data into relevant categories based on the functional meaning of the data. Classifying data can help your organization manage risks, compliance, and data security.
Relationship Discovery
The relationship discovery capability identifies pairs of similar columns and relationships between tables within a catalog source.
Glossary Association
You can associate terms that are in the glossary with technical assets to provide user-friendly business names to technical assets. Glossary Association automatically associates glossary terms with technical assets or recommends glossary terms that you can manually associate with technical assets in Data Governance and Catalog.

Extraction and view process

To extract metadata from a source system, configure the catalog source and run the extraction job in Metadata Command Center. Then view the results in Data Governance and Catalog.
The following image shows the process to extract metadata from a source system:
The image shows the process of metadata extraction from a Google BigQuery source system that begins with prerequisites verification, continues with the creation of the catalog source, and ends with viewing the extraction results.
After you verify prerequisites, perform the following tasks to extract metadata from Google BigQuery:
  1. 1Register a catalog source. Create a catalog source object, select Google BigQuery, and then select and test the connection.
  2. 2Configure the catalog source. Specify the runtime environment and configure parameters for metadata extraction. Optionally, add filters to include or exclude source system assets from metadata extraction. You can also configure other capabilities such as data profiling and quality, data classification, or glossary association.
  3. 3Optionally, associate stakeholders. Associate users with technical assets, giving the users permission to perform actions determined by their roles.
  4. 4Run or schedule the catalog source job.
  5. 5Optionally, if the catalog source job generates referenced asset objects, you can assign a connection to referenced source system assets.
    6. You can view the lineage with object references without performing connection assignment. After connection assignment, you can view the objects.
After you run the catalog source job, you view the results in Data Governance and Catalog.

About the Google BigQuery catalog source

You can use the Google BigQuery catalog source to extract metadata from a Google BigQuery source system.
Google BigQuery is a serverless data warehouse that performs scalable analysis over petabytes of data. Use a Google BigQuery catalog source to collect metadata from the assets in a Google BigQuery source system.

Extracted metadata

You can use the Google BigQuery catalog source to extract metadata from a Google BigQuery source system.
Metadata Command Center extracts the following metadata from a Google BigQuery source system:
Note: Each time Metadata Command Center runs a scan on a Google BigQuery catalog source, Data Governance and Catalog displays the extracted objects along with the objects from the previous scans. This means that if you modify the filters in Metadata Command Center and run a scan, Data Governance and Catalog does not replace the objects from the previous scan. Instead, Data Governance and Catalog adds the newly extracted objects to the existing scanned objects. To see the results of only the latest scan, choose Delete as the Metadata Change Option before you run the scan.

Data profiling for Google BigQuery objects

Configure data profiling to run profiles on the metadata extracted from a Google BigQuery source system. You can run data profiles on the following Google BigQuery objects:
The data profiling task runs profiles on the following data types:

Compatible connectors

Before you configure a Google BigQuery catalog source, you must connect to the Google BigQuery source system.
Use the Google BigQuery V2 connector to connect to the Google BigQuery source system.
For information about configuring a connection, see Connections.