Discovery Agent > Data exploration > Explore source data of assets from the catalog

Explore source data of assets from the catalog

Explore source data and preview sample data sets in your catalog with CLAIRE GPT.

The following table lists source systems from which you can preview sample data:

Source System	Connection Type	Authentication Type
Amazon Redshift	Amazon Redshift v2	Default
Databricks	Databricks	Personal Access Token
Google BigQuery	Google BigQuery V2	Service Account
Microsoft Azure Synapse	Microsoft Azure Synapse SQL	Shared Key
Microsoft Azure SQL Server	SQL Server	SQL Server
Microsoft SQL Server	SQL Server	SQL Server Windows Authentication V2
Oracle	Oracle	Oracle Database
Snowflake	Snowflake Data Cloud	Standard

Consider the following guidelines before you perform data exploration:

•You can perform data exploration of tables that do not have primary and foreign key constraints defined at the schema level of the source system.
•When you perform data exploration, critical data elements appear first in the response, followed by other data elements sorted by precedence. You can view a maximum of 100 rows and 100 columns of sample source data.
•You need to run data profiling on assets in the catalog to get accurate results before you explore source data that contains specific information. For example, if a source system contains information on 'grass pollen' and you use the prompt 'find all patients allergic to grass,' the response might be inaccurate. To ensure that you get accurate results, run data profiling before you explore source data.
•When you perform data exploration, CLAIRE GPT might not find source data if the letter case of the search query differs from that of the source data. This occurs on the EM SouthEast 1 Azure and Canada Central 1 Azure PODs.
•When you perform data exploration on data from a Snowflake source system, sorting by the Order By SQL clause might not work as expected. This happens if you sort on the same SQL clause multiple times and the data has more than 100 rows.
•You can't view data preview of tables, databases, and schemas with names that contain special characters. You can view data preview of tables that include columns with names that contain special characters except the following characters:

Prerequisites

Before you perform data exploration tasks in CLAIRE GPT for Data Governance and Catalog, ensure that you perform the following tasks:

•Enable the Data Governance and Catalog service for the runtime environment that your organization uses. For more information, see Runtime Environments in Administrator.
•Verify that you extracted metadata from the required source systems into the catalog. For more information, see Asset Discovery in Data Governance and Catalog.
•Configure the following connections on the Connections page of Administrator:

- Amazon Redshift
- Databricks

Note:

Before you perform data exploration tasks in CLAIRE GPT, verify that the Databricks server is up and running to avoid delays in responses from CLAIRE GPT.

- Google BigQuery
- Microsoft Azure Synapse
- Microsoft Azure SQL Server
- Microsoft SQL Server
- Oracle
- Snowflake

For information about configuring a connection in Administrator, see Configuring a connection.

•Enable the connections you configured earlier on the Secure Agent group from the Enable/Disable Components in runtime environment page of Administrator.

For information about enabling connections for a Secure Agent group in Administrator, see Service and connector assignment for Secure Agent groups.

•Download the dynamic link libraries (DLLs). Required if you configure an SQL Server connection with Windows Authentication V2 authentication to access and preview sample data from Microsoft SQL Server source systems.

Perform the following steps:

1Download the DLLs from the following location: MS SQL Server JBDC DLL
2Copy the DLLs to the Secure Agent installation directory. For example: <Secure Agent installation directory>/ext/connectors/thirdparty/informatica.jdbc_v2/common

•To get precise data exploration responses, enrich the data catalog and associate assets with accurate business terms and descriptions.
•For accurate and reliable data exploration responses, ensure that the primary and foreign key constraints of tables are properly defined at the schema level of the source system. For Google BigQuery and Microsoft Azure Synapse sources, create and run custom catalog sources in Metadata Command Center to extract and ingest the primary and foreign key constraints and their relationships into Data Governance and Catalog.

For information about creating a custom catalog source in Metadata Command Center, see Custom metadata integration.

•To get data exploration responses for tables that do not have primary key constraints defined at the schema level of the source system, you must enable data profiling and run the catalog source.

For information about how to set up CLAIRE GPT for users in your organization, see Administer CLAIRE GPT.

Configuring IDMC Data Gateway Service properties

Secure Agent services such as IDMC Data Gateway Service are pluggable microservices that the Secure Agent uses for data processing. To configure Secure Agent service properties, open the Runtime Environments page in Administrator and edit the Secure Agent. To change or optimize the behavior of the IDMC Data Gateway Service, you can configure the system and custom properties when you edit a Secure Agent.

To change or optimize the behavior of the IDMC Data Gateway Service, you can configure the system and custom properties when you edit a Secure Agent.

You can configure the following system properties of the IDMC Data Gateway Service in the System Configuration Details section:

Type	Name	Description	Value Range	Default Value
APP_CFG	idmc_dgs_connectionPoolMaxSize	The maximum number of data source connections cached for a Secure Agent.	Minimum is 10. Maximum is 1000.	100
APP_CFG	idmc_dgs_connectionPoolCacheExpiryDuration	The maximum duration for which the connection with a data source is kept open when it is unused. Note: The value of the parameter must be in minutes.	Minimum is 10. Maximum is 120.	30 minutes
APP_CFG	idmc_dgs_queryExecutionTimeout	The maximum duration for which a database query is run on a data source. Note: The value of the parameter must be in seconds.	Minimum is 1 There is no maximum value.	60 seconds

Note:

APP_CFG refers to application configuration properties.

For more information about the system properties you can configure for the IDMC Data Gateway Service, see Secure Agent services.

Sample conversation

Let's take a look at a sample conversation to help you get started on your Data Governance and Catalog data exploration journey in CLAIRE GPT.

You’re a marketing analyst. You want to create a report on customer data for an e-commerce company in order to design effective marketing campaigns for customers. Your organization catalogs the customer data in Data Governance and Catalog. To create the report, you need a list of customer profiles along with their subscription status and demographic details.

To achieve the goal, perform the following steps:

1Log in to CLAIRE GPT.
2Start a conversation to get a list of customers along with their subscription status.

Enter the following prompt:

I want to do reporting on customer data. Can you provide a list of customers along with their subscription status - Active/Inactive?

The following image shows the response displaying the relevant data sets across catalog sources that have customer details and their subscription status: The response displays three identified data sets that are retrieved for customer information and the subscription status. The displayed data sets are SUBSCRIPTIONS, customerdetails, and CUSTOMERSPROFILE.

The response displays three identified data sets that are retrieved for customer information and the subscription status. The displayed data sets are SUBSCRIPTIONS, customerdetails, and CUSTOMERSPROFILE.

The discovery agent identifies and discovers the relevant data sets that contain customer information or the customer subscription status. The agent clearly states the reason for the selection of each data set. It incorporates human-in-the-loop interaction and asks you to select the data set that is best suited for your report.

The following image highlights the question, demonstrating the human-in-the loop interaction to help you select the right data set: The response highlights a question asking the user to specify a preferred dataset for customer subscription status, listing three options: SUBSCRIPTIONS, customerdetails, and CUSTOMERSPROFILE, with additional context for each option and a request to reply with the corresponding number.

The response highlights a question asking the user to specify a preferred dataset for customer subscription status, listing three options: SUBSCRIPTIONS, customerdetails, and CUSTOMERSPROFILE, with additional context for each option and a request to reply with the corresponding number.

3Specify which data set you want to use.

Enter the following prompt:

Let's go with the first one

The following image shows the response to the prompt about using the SUBSCRIPTIONS data set which is the first among the discovered data sets: The response displays the reasoning and the card for the customer subscription status sample table that CLAIRE GPT generates.

The response displays the reasoning and the card for the customer subscription status sample table that CLAIRE GPT generates.

The data exploration skill of the discovery agent pulls data from two relevant data sets, CUSTOMERSPROFILE and SUBSCRIPTIONS. To create a sample table for the customer subscription status, the data integration agent joins the two data sets. Further, it creates full customer names by concatenating the first and last names, and categorizes subscription status as either Active or Inactive.

4Click the card to view or download the sample table.

The following image displays the Customer Subscription Status sample table. The response displays the Customer Subscription Status sample table which shows a tabular list with columns labeled customer_name and subscription_status. The panel on the right shows the explanation of how the Customer Subscription Status sample table was created and some suggested prompts for next actions.

The response displays the Customer Subscription Status sample table which shows a tabular list with columns labeled customer_name and subscription_status. The panel on the right shows the explanation of how the Customer Subscription Status sample table was created and some suggested prompts for next actions.

You can download the sample table as a CSV file or copy the table to another file.