AI Agent workflow using CDGC Metadata

1. 1The agent interprets and answers user questions by combining metadata discovery, semantic understanding, automated SQL generation, and validation.
2. 2The process begins when the user submits a question in natural language. The main planning prompt defines the execution steps and configuration conditions for the task.
3. 3Next, the agent queries CDGC through the API to fetch all business glossaries linked to the Catalog Source selected by the user. It retrieves glossary names, descriptions, and types, then selects several business glossaries that are semantically related to the user's question. For example, for the question “How many customers under 50 live in the USA?”, it might select relevant terms such as “Customer,” “Country,” and “Age.”
4. 4Following this, the agent fetches basic table-level metadata from CDGC related to the selected business terms, including table descriptions and schema locations. Optionally, if enabled by the user, relationship metadata is also extracted to provide additional context.
5. 5The collected metadata is then passed to the IdentifyTablesAgent, which filters and narrows down the tables. Based on metadata like table descriptions and names, it selects only those tables most likely to contain relevant data, removing irrelevant ones to improve efficiency.
6. 6After the tables are identified, the main agent retrieves detailed column-level metadata for these tables, including column names, data types, descriptions, and nullability. This completes the metadata extraction related to the user's question.
7. 7Using the gathered metadata and the original user query, the agent generates an SQL query in the appropriate query language. The query is executed on the corresponding database, for example, Snowflake, and both the query and the results are returned to the main agent.
8. 8At this point, the ValidationAgent reviews whether the generated SQL query and its results answer the user's question. If validation succeeds, the process concludes. If not, the ValidationAgent suggests improvements, prompting the SQLGenerationAgent to iteratively refine and regenerate the query until it meets the required criteria.
9. 9Finally, the main agent displays a concise answer to the user's question, the executed SQL query, and a summary highlighting essential insights and key information extracted from the data.
10. 10This workflow enables users to effortlessly gain business insights using natural language queries, leveraging metadata discovery, semantic analysis, query generation, and validation to bridge the gap between raw data and actionable intelligence.