Link catalog sources to generate lineage

Due to technological limitations or security constraints, you might not always see complete lineage after metadata extraction. You can use Metadata Command Center to link catalog sources and construct data lineage. You can choose source and target catalog sources to link and create lineage. You can also choose source and target schemas to restrict lineage inference to specific subsets of data objects within the data sources.

You can either generate automated lineage with CLAIRE or define rules to generate catalog source links between assets of the source and target catalog sources.

You can link catalog sources and generate lineage automatically with CLAIRE. You can choose to automatically accept CLAIRE-generated lineage recommendations or manually accept them. CLAIRE-generated lineage recommendations are automatically accepted based on a threshold limit. If the confidence score of the CLAIRE-generated catalog source links between a source and target asset is higher than the configured threshold limit, the recommended links are automatically accepted.

When you define rules to generate lineage, you can use name-based matching or create an inclusion rule with expressions. Name-based matching matches objects based on their names, whereas expressions allow you to construct an inclusion rule. You can specify prefixes and suffixes to omit metadata from the source and target data sets and data elements. This removes extraneous naming conventions, as well as prefixes and suffixes that were added or removed to the asset names while moving data from one source system to another. You can use a combination of attributes, operators, functions, and comments to construct an inclusion rule with expressions.

You can create, view, update, run, delete, or purge a configuration. When you delete a configuration, Metadata Command Center first deletes generated catalog source links and then deletes the configuration. Purging a configuration deletes catalog source links generated from the configuration but retains the configuration. You can also clone a configuration to create another configuration with the same details. You can monitor the jobs that run when you create or perform other tasks on configurations on the Monitor page.

The linked assets and catalog source links generated based on rules are auto-accepted by default and appear on the Catalog Source Links page in Data Governance and Catalog. Stakeholders of the source and target catalog sources can reject the auto-accepted and manually accepted catalog source links from the Action menu. If stakeholders initially reject the generated catalog source links and later accept them, they are marked as accepted in Data Governance and Catalog. Stakeholders can also view the generated lineage on the Lineage tab of the asset.

For information about curation of the generated catalog source links, see Linked lineage.