Lineage defines the data flow across systems in your organization. You can create custom lineage to view the data lineage information for the assets in your organization. You can do this either by using the links.csv file or using reference assets that are derived from reference catalog sources.
Note: When you create a custom catalog source without the core.externalID for an object, the custom catalog source job generates an external ID. This ID appears in the Reference ID field on the System Attributes tab of the asset in Data Governance and Catalog. You can use this external ID to create custom lineage.
Using the links.csv file
To create custom lineage using the links.csv file, define associations between any two existing objects in the catalog. Verify that the links.csv file contains the following details for defining associations between objects:
Header Fields
Description
Source
Required. Unique Reference ID of the object in the catalog source that has been synced. You can find the reference ID of the catalog source object in the Properties tab in Data Governance and Catalog.
Target
Required. Unique Reference ID of the object in the catalog source that has been synced. You can find the reference ID of the catalog source object in the Properties tab in Data Governance and Catalog.
Association
Required. Indicates the type of association. Specify values based on the type of source and target object that you want to link. You can define the following types of associations:
- core.ResourceParentChild. To link a parent object to a top-level child object. For example, catalog source to schema.
- core.DataSourceParentChild. To link a source system to a data set. For example, schema to table object.
- core.DataSetToDataElementParentship. To link a data set to a data element. For example, table to column.
- core.DataSetDataFlow. To link source and target objects at the data set level. This association creates data set-level lineage. For example, table to view.
- core.DirectionalDataFlow. To link source and target objects at the data element level. This association creates data element-level lineage. For example, table column to view column.
Sample custom lineage file
The following sample shows a links.csv file to define lineage between relational objects:
You can configure predefined or custom catalog sources to view lineage with reference assets from referenced catalog sources. For information about referenced catalog sources and reference assets, see Referenced catalog sources.
When you configure a custom catalog source, set the core.reference and core.assignable header fields to true in the metadata CSV file for each object of the source system. See Step 6. Create the custom catalog source. The source system objects are treated as reference assets and you can assign connections to the custom catalog source in order to define lineage.
After creating the custom catalog source and running the catalog source job, the ingested objects from the referenced catalog source appear as reference assets and Data Governance and Catalog displays the lineage with the reference assets. You can also see the reference catalog source listed on the Connection Assignment page in Metadata Command Center. To link the reference assets to actual objects and to view the complete lineage, assign a connection for the reference catalog source. See Assign connections to reference catalog sources.