View Automated Lineage from Enterprise Data Catalog
In Axon, you can choose to automatically create links between attributes based on the recommendations from Enterprise Data Catalog. You need not manually review the relationship recommendations and curate attributes for each data set. You can view lineage recommendations for single and multiple hop relationships between the physical fields in Enterprise Data Catalog.
A physical field F1 from table T1 is directly linked to physical field F2 from table T2. The physical field F1 is linked to attribute A1, and the physical field F2 is linked to attribute A2. Enterprise Data Catalog provides lineage recommendation between attributes A1 and A2. Run the Axon scanner to discover the relationships. If you choose to accept lineage recommendations and create links, you can see that the link between the attributes A1 and A2 are automatically created in Axon after automated onboarding of attributes. A daily job runs to discover additional relationships that can be onboarded from Enterprise Data Catalog. You can see the linked attribute A1 in the Relationships tab of the data set that contains attribute A2. The Type column displays the value "Sourced From" to indicate that the related attribute relationship is created based on Enterprise Data Catalog recommendations. You can view the attribute links in both inbound and outbound relationship grids. To find the newly created attribute links, you can filter based on the Created Date column that is hidden by default.
To accept the lineage recommendations, you need to enable the Automatically Accept Lineage Recommendations option from the Admin Panel. If the fields are linked to attributes and the fields are linked with each other in Enterprise Data Catalog, the links between attributes are automatically created. After the link is created between attributes, Axon notifies the data set stakeholders only with the Stewardship Role type. The notification appears on the Catalog tab of the notification center that you can access from the toolbar. The notification contains information about all the attribute links that are created. If you delete an automatically created relationship, the same link is not created again in the next onboarding process.
The onboarded links between the attributes is unique. When the onboarding script runs, duplicate attribute links are not onboarded. The uniqueness of the Attribute X Attribute links is not validated based on the Sourcing Type. If you delete a link between attributes, the link is not onboarded again when the onboarding script runs. You can manually activate the deleted link when you click Show Recommendations from the recommended attributes.
If you do not select the Automatically Accept Lineage Recommendations option from the Admin Panel, Axon sends an email and interface notification. The notification contains information about the discovery of additional linked attributes and the link that directs you to the Relationship tab of a Data Set object. Click Show Recommendations to view the recommended attributes. There might be a delay in showing the recommendations even after the Axon scanner run is complete. You can accept or reject recommendations for each data set.
Lineage Onboarding Prerequisites
Automated lineage between attributes is created in Axon based on the following prerequisites:
- •Physical fields must be directly connected to each other in Enterprise Data Catalog.
- •Attributes must be associated with the physical fields.
- •Run the Axon scanner after a physical field is linked to an attribute to discover attribute relationships.
- •Run the Axon scanner if you manually curate attributes and then run the onboarder job to discover additional relationships.
- •If a default change request is configured for data sets, the onboarding of attributes initiate a change request workflow. After the change request is complete, run the Axon scanner and then run the onboarder script to onboard the attribute to attribute relationships and lineage recommendations.
Lineage Onboarding Scenarios
The attribute links are automatically created in the following scenarios:
- •Consider that physical fields F1 and F2 are linked to each other in Enterprise Data Catalog, and the physical field F1 is linked to attribute A1 and F2 is linked to A2. Run the Axon scanner after a physical field is linked to an attribute. If you choose to accept the lineage recommendations from the Admin Panel, the attributes A1 and A2 are automatically linked in Axon.
- •Consider that physical fields F1 is linked to F2 and F2 is linked to F3. The physical field F1 is linked to attribute A1, F2 is not linked to any attribute, and F3 is linked to A3. Run the Axon scanner after a physical field is linked to an attribute. If you choose to accept the lineage recommendations from the Admin Panel, the attributes A1 and A3 are automatically linked in Axon. Since F2 is not linked to any attribute, the immediate inbound relationship is checked and onboarded if the physical field is linked to an attribute.
- •Consider that physical fields F1 and F2 are linked to each other in Enterprise Data Catalog, and the physical field F1 is linked to glossary G1 and F2 is linked to G2. From the Admin Panel, you choose to automatically onboard attributes, accept the lineage recommendations, and accept onboarded attributes. Run the Axon scanner after a physical field is linked to a glossary. After the onboarder job runs, the attributes with names G1 and G2 are automatically onboarded and the relationship between attributes G1 and G2 is created in Axon.
- •Consider that physical fields F1 and F2 are linked to each other in Enterprise Data Catalog, and the physical field F1 is linked to glossary G1 and F2 is linked to G2. From the Admin Panel, you choose to automatically onboard attributes and accept the lineage recommendations but you do not choose to accept onboarded attributes. After the onboarder job runs, the attributes with names G1 and G2 are automatically onboarded with the "Discovered" review status. The relationship between attributes G1 and G2 are not automatically created in Axon. To automatically create the relationships, you need to manually curate the discovered attributes G1 and G2. Run the Axon scanner and then run the onboarder job again. After the onboarder job runs, the relationship between attributes G1 and G2 is created in Axon.
The attribute links are not automatically created in the following scenarios:
- •Consider that physical fields F1 and F2 are linked to each other in Enterprise Data Catalog, and the physical field F1 is linked to attribute A1 and F2 is not linked to any attribute. If you choose to accept the lineage recommendations from the Admin Panel, you cannot see any lineage recommendation for attribute A1 in Axon.
- •Consider that physical fields F1 and F2 are linked to each other in Enterprise Data Catalog, and the physical field F1 is linked to glossary G1 and F2 is linked to G2. If you do not choose to automatically onboard attributes and choose to accept the lineage recommendations from the Admin Panel, you cannot see any lineage recommendation in Axon.
Single Hop Lineage Example
The following image shows the sample automated lineage onboarding process for single hop physical field relationships:
Consider that there are Employee, Finance, Admin, and Org tables that contain salary information in Enterprise Data Catalog. The physical fields of these tables are linked based on the salary information. Emp_Sal and Admin_Sal physical fields are linked to Fin_Sal physical field, which is linked to Org_Sal physical field. All physical fields are linked through a single hop in Enterprise Data Catalog. The linking between Enterprise Data Catalog physical fields and Axon attributes is as follows:
- •Emp_Sal physical field to Emp_attr1 attribute
- •Admin_Sal physical field to Admin_attr2 attribute
- •Fin_Sal physical field to Fin_attr3 attribute
- •Org_Sal physical field to Org_attr4 attribute
If you choose to accept the lineage recommendations from Enterprise Data Catalog, the links between attributes are automatically created in Axon based on the linked fields in Enterprise Data Catalog. For attribute Fin_attr3, you can see the linked attributes Emp_attr1 and Admin_attr2 in the Relationships > Inbound Relationships view of the data set. You can see the linked attribute Org_attr4 in the Relationships > Outbound Relationships view.
Multiple Hop Lineage Example
The following image shows the sample automated lineage onboarding process for multiple hop physical field relationships:
Axon supports automated lineage creation for single hop, multiple hops, and direct relationships between the physical fields. In the example, Column 1 is directly linked to Column 2, Column 1 is linked to Attribute1, and Column 2 is linked to Attribute2. You need to run the Axon scanner to discover the relationships. When the onboarding job runs, automated lineage is created between Attribute1 and Attribute2. Column 2 is linked to Column 5 through multiple hops and the lineage between Attribute2 and Attribute3 is created automatically. But the link between Attribute1 and Attribute3 is not created automatically because the onboarding job checks for the first inbound relationship in the multiple hops. After the Attribute2 to Attribute3 link is created, the onboarding job does not check for Attribute1.
Lineage Notifications
Axon sends email and interface notifications to the stakeholders based on configuration of the Automatically Accept Lineage Recommendations and Enable Lineage Recommendation Notifications options on the Admin Panel.
The following table describes the different combination of the values based on which Axon sends lineage notifications:
Automatically Accept Lineage Recommendations | Enable Lineage Recommendation Notifications | Notifications |
|---|
True | True | Data Steward receives email and interface notifications about the automatically created relationships. |
True | False | Data Steward receives email and interface notifications about the automatically created relationships. |
False | True | Data Steward receives email and interface notifications about the lineage recommendations. |
False | False | Axon does not send any notification. |
