Custom Lineage Resource CSV File
Create a custom lineage resource to view the data lineage from data sources, such as the relational databases and flat file databases. The custom lineage resource uses a csv file to extract the lineage information. The elements in the csv file include Association, From Connection, To Connection, From Object, and To Object.
The following table describes the elements in the csv file:
Element | Description |
|---|
Association | Indicates the type of association. Specify correct values in the association column based on the type of source and target object that you want to link. The following are a few types of dataflow associations: - - core.DataSourceDataFlow. To link source and target object at the data source level. The dataflow is at the schema level for the relational sources.
- - core.DataSetDataFlow. To link source and target object at data set level. The dataflow is at the table level for the relational sources.
- - core.DirectionalDataFlow. To link source and target object at data element level. The dataflow is at the column level for the relational sources.
- - core.DirectionalControlFlow. To link source and target object at data element level. The control flow is at the column level for the relational sources.
- - core.DataSetControlFlow. To link source and target object at data set level. The control flow is at the table level for the relational sources
- - core.DataSourceControlFlow To link source and target object at the data source level. The control is at the schema level for the relational sources.
Note: If you do not enter values in the association column, the custom lineage resource extracts lineage information only between relational and file sources. |
From Connection | Connection parameter for the source object. |
To Connection | Connection parameter for the target object. |
From Object | Partial ID of the source object after you perform the connection assignment. |
To Object | Partial ID of the target object after you perform the connection assignment. |
Custom Lineage Resource CSV File Examples
You can enter values in the csv file for the custom lineage resource using connection parameters and absolute object UID.
- Connection Parameters
- You can provide connection parameter values in the csv file to extract the lineage information. After you run the custom lineage resource, you must perform connection assignment to display the lineage in catalog.
The following sample shows a .csv file to import lineage between two relational objects:
Association,From Connection,To Connection,From Object,To Object
core.DirectionalDataFlow,<FromConnection>,<ToConnection>,<FromTableName>/<FromColumnName>,<ToTableName>/<ToColumnName>
core.DataSetDataFlow,<FromConnection>,<ToConnection>,<FromTableName>,<ToTableName>
The following sample shows a csv file to import lineage between two structured file objects:
Association,From Connection,To Connection,From Object,To Object
core.DirectionalDataFlow,<FromConnection>,<ToConnection>,<SourceObjectFilePath>/<FromColumnName>,<TargetObjectFilePath>/<ToColumnName>
core.DataSetDataFlow,<FromConnection>,<ToConnection>,<SourceObjectFilePath>,<TargetObjectFilePath>
The following sample shows a csv file to import lineage between relational and structured file objects:
Association,From Connection,To Connection,From Object,To Object
core.DirectionalDataFlow,<FromConnection>,<ToConnection>,<SourceObjectFilePath>/<FromColumnName>,<ToTableName>/<ToColumnName>
core.DataSetDataFlow,<FromConnection>,<ToConnection>,<SourceObjectFilePath>,<ToTableName>
- Absolute Object UID
- Provide the absolute object UID in the csv file to extract the lineage information. To identify the absolute object UID of an asset, navigate to the asset in the catalog, and then copy the object UID from the URL. For example, the object UID for a file system asset is ResourceName://FileServer/<Absolute Path>/File.
The following sample shows a csv file to import lineage from relational and structured file objects using the object UID:
Association,From Connection,To Connection,From Object,To Object
core.DirectionalDataFlow,,,Flatfile://FileServer/test2/cust2.csv/column1,Oracle_nc://ORCL/NC/TEST/column1
core.DataSetDataFlow,,,Flatfile://FileServer/test2/cust3.dat,Oracle_nc://ORCL/NC/TEST2
Note: If you want to extract the lineage information using the absolute object UID, you should not specify values in the FROM CONNECTION and TO CONNECTION columns in the csv file.