Data Quality Rules
The Data Quality facet displays the data quality rules. You can configure different types of data quality rules to check the quality of the data in your systems. Technical users implement the rules in a third-party data quality product.
You can use Informatica Analyst and Informatica Developer to analyze the source data and metadata. Axon can import data quality information from profiles and scorecards that you define in Informatica Data Quality. You can create profiles to analyze the content, quality, and structure of data sources. Scorecards monitor data quality. You must install the Axon Agent before you can configure technical references to profiles and scorecards in Informatica Data Quality.
Data Quality Report holds the scores or ratings of the data quality rules for a given period. Typically, a set of rows in a table are validated against a data quality rule and the report list the percentage of compliant rows as well as the number of exceptions. The report displays the scores in red, amber, or green color according to the rule thresholds that you have specified in the respective rule.
Some Data Quality systems provide results as exceptions or variances from a perfect score. For example, a 5% value might not meet the rule requirements. Axon requires the data as an achieved score. It only considers values that are in the following format:
- •0 or 1
- •Yes or No
- •True or False
When the profile result for a column is in numeric values other than Axon standard, there is no error logged in Axon. Axon updates the volume count result as 0%. When the profile result for a column has string values, then Axon records an error message.
Local Data Quality Rules
A data quality rule defined within the context of a System is referred to as a local data quality rule. Only local data quality rules can have scores or reports associated to them. All Data Quality dashboards seen in Axon relate to local data quality rules and reports.
Before you can create a local data quality rule, you must create a data set with at least one attribute. Data quality rules are measured against the attributes in the data set. Create the attributes before you create the data quality rules. You can create local data quality rules manually or by bulk upload.
Standard Data Quality Rules
Large organizations often have departments working independently of each other, using their own data sources and systems. In turn, these functions are unaware that other areas use the same data and hence, set up their own controls and quality measurement. As a result the same data is often measured in different Data Quality tools, systems, and data sets. Where this occurs, it is desirable that similar data is subject to the same quality measurement standards. Axon facilitates this via standard data quality rules.
Standard data quality rules set the guidelines for how anything linked to a standard data quality rule for a data set or attributes needs to be measured. Standard data quality rules are described at the glossary level. Since similar data must be associated to the same Glossary term, it is the ideal place to share common standards. Any local data quality rules controlling attributes linked to this glossary object can therefore be set up against these standards.
Creating Local Data Quality Rules
Create a local data quality rule for a data set. You must have edit permissions on Data Set objects to create a local data quality rule. You provide initial details when you create a new data quality rule. After you save the rule, Axon redirects you to the edit mode where you can provide more details. In both create and edit modes, Axon marks the mandatory fields with a red asterisk. You can also create a data quality rule without adding an attribute, and Axon lists the data quality rule in Unison search.
1. From the Unison search page, open a Data Set object.
2. From the Actions menu, click + New Data Quality Rule.
The New Data Quality Rule page opens.
3. In the Definition section of the Summary tab, configure the following properties:
Option | Description |
|---|
Name | Required. Specify a name for the data quality rule. |
Description | Required. Describe the logic for the data quality rule. |
Ref | Unique identifier for each object. Axon automatically assigns a unique reference ID if you leave this field blank. |
Type | Required. Select from one of the following types of data quality rules: - - Completeness
- - Accuracy
- - Consistency
- - Validity
- - Timeliness
You can enter up to nine data quality rule types using the dropdown configuration in the Admin Panel. |
System Measured In | Required. Name of the system in which the rule is measured. This is often the name of the Data Quality profiling or measurement system used for the given data set. Note: Axon automatically populates the system name in the System Measured In field when you create a local data quality rule. You cannot modify the system name when you create or modify the local data quality rule. |
Measuring Method | Required. How the data quality rule is evaluated. You can specify one of the following options: - - System Function. The rule is measured by a data quality system, such as Informatica Data Quality.
- - Technical Script. The rule is measured by a script that is manually run by an analyst.
- - Business Extract. The rule is measured on data that is exported for a particular business case.
|
4. In the Optional Fields section of the Summary tab, configure the following properties:
Option | Description |
|---|
Measuring Method Description | Optional. Add relevant information about the measurement process. |
Population | Optional. Describe the population in scope for the rule. |
Technical Description | A technical description of the rule logic. Can include SQL or pseudo SQL. |
Technical Rule Reference | A reference link that associates local data quality rule to profile and scorecard. A clickable link appears when you configure the Analyst tool URL in the System Settings section of the Admin Panel. |
5. In the Attribute Items in Measure section of the Summary tab, configure the following optional fields:
Option | Description |
|---|
Attribute | Select the name of the attribute that the rule is measuring. Click the Add button to select multiple attributes. |
Measured Against | Optional section for rule types only to show against which attribute in the current data set is being measured. Click the Edit icon. In the Add Attribute dialog box, you can select a system, data set, and attribute values based on the segment that are assigned to you. Note: If a data set belongs to a segment and you are not assigned to that particular segment, you cannot view Measured Against field. |
6. Click Add Item if you want to add more attributes to measure the data quality rule against. Informatica recommends that you measure only one data quality rule against one attribute.
7. In the Classifications section of the Summary tab, configure the following properties.
Option | Description |
|---|
Axon Status | Status of the data quality rule in Axon. Select one of the following values: - - Active
- - Inactive
- - Pending Review
- - Obsolete
- - Deleted
If you want to delete a standard data quality rule, you need to set the Axon Status to Deleted. |
Lifecycle | Lifecycle status of the object. Select one of the following values: - - Active
- - Draft
- - Obsolete
|
Axon Viewing | Choose if the data quality rule is public or non-public. |
Criticality | Level of criticality . Select from one of the following options: |
Frequency | How often the rule is measured. Select one of the following values from the drop-down menu: - - Daily
- - Weekly
- - Monthly
- - Quarterly
- - AdHoc
- - Bi-Weekly
|
Automation | How automated the Data Quality measurement process is. Select from one of the following options: - - Fully Automated
- - Partially Automated
- - Manual
- - Unknown
|
Green Target | Enter this as a number from 0 to 100 to reflect the proportion of the measured data has achieved this level of performance Do not enter characters e.g. %. Decimal Points are acceptable, although results are rounded. Some Data Quality measurement systems report deviation from a perfect score rather than stating the actual score - do not enter as deviation, i.e. if 5% deviation is reported, enter ‘95’ |
Amber Target | A number from 0 to 100 that indicates the threshold value for amber color. |
8. Click Save and Close.
Linking Profiles and Scorecards to Local Data Quality Rules
You can link data quality rules to columns in Informatica Data Quality. Informatica Data Quality runs a profile or scorecard which you define against the data in a column or metric. Axon fetches the profile score or scorecard data and displays it as a report or dashboard in the data set, data quality rule, or system.
You must create and save a data quality rule before you link profiles and scorecards to the rules.
1. From Unison search, open a data set object.
2. In the Data Quality tab, click the Rules sub-tab.
3. In the Ref column, click the data quality rule that you want to edit.
Axon displays the data quality rule.
4. Optionally, search for a data quality rule from Unison or open the Data Quality facet in Unison to see existing rules.
5. Click Edit.
6. In the Defintion section of the Summarytab, click the 
icon for theTechnical Rule Reference field. Axon displays the Select Technical Rule Reference dialog box.
7. Choose Scorecard or Profile.
8. Select a project from the list of projects that appear. The list of projects is from the Model Repository Service in the Informatica domain.
9. Select a scorecard or profile. Based on the selection, choose a column or metric.
10. In the Classifications section, select the frequency under Basic Classifications.
11. Select a frequency and configure the scheduling details based on the frequency that you selected. Axon retrieves scores from Informatica Data Quality based on these parameters.
12. Select Run Rule and Retrieve Data for Axon to run the profile or scorecard in Informatica Data Quality at the scheduled frequency. If you do not select Run Rule and Retrieve Data, Axon fetches the latest data available in Informatica Data Quality.
13. Click Save and Close.
When you associate a profile or scorecard to a local data quality rule, you can select the Run Rule and Retrieve Data check box. If you select the check box, make sure that you choose the option for all local rules to which you associate the same profile or scorecard. Conversely, if you do not select the check box for a local data quality rule, make sure that you do not choose the option for all local rules to which you associate the same profile or scorecard.
If you do not choose the Run Rule and Retrieve Data check box for any local rule, make sure that you run the rule manually once so that Axon can retrieve and display the scores at the next scheduled interval. If you do not run the rule manually, Axon might not have any score to display for the rule at the next scheduled interval.
Enhance Local Data Quality Rules
Add information to data quality rules after you create and save the rule. You can update the existing data to provide more information or link it to other items in Axon.
All users of Axon can view the details in the data quality rule objects. Only users with permissions on individual items can make edits. Users with no edit rights may be able to make associations to an item if they hold edit rights against other item types.
You can add more information to a data quality rule in the following tabs.
- Stakeholder Tab
- Displays a list of stakeholders and followers for the data quality rule.
- You can view information about stakeholders in the following sections:
- - Direct Stakeholders. Responsible for providing, defining or maintaining the item
- - Stakeholder Community. If this item is connected to any other items in Axon, the stakeholders of those connected items collectively make up the wider stakeholder community.
- Report Tab
- Displays reports and results about the data quality rules. You can upload data or manually add data here. Displays historical scores for the data quality rule. Axon considers only the latest score for calculation in the Data Quality dashboard. The same report is displayed in the Data Quality Dashboard in the Data Quality tab of a data set.
- You can view information in the following sections:
- - Data Quality Results. Displays a chart of data quality scores for 100 records by default. You can apply a filter based on the number of records or choose a date range to view the Data Quality Results chart. If you choose to filter based on the number of records, enter a number and click Apply. If you do not enter any value, Axon loads the chart with all the records. If you choose to filter based on the date range, choose From Date and To Date values, and click Apply.
- - Data Quality Reports. Gives greater detail on all data quality reports added for the rule. Axon applies a limit of 100 reports when the grid loads.
- Relationships Tab
- Summarises information on Data Quality into the following sub-views:
- - Standardised Data Quality Rules
- - Non-Standardised Data Quality Rules
- Impact Tab
- The Impact tab shows how connected an item is across many other facets.
Create a Standard Data Quality Rule
Create a standard data quality rule in a Glossary object. You must have edit permissions on the Glossary object.
1. From Unison search, open the Glossary object for which you want to create a standard data quality rule.
2. Click Edit.
3. From the Actions menu, click New Standard Data Quality Rule.
The New Standard Data Quality Rule tab appears.
4. In the New Standard Data Quality Rule tab, specify the properties of the standard rule. You can enter the basic properties of the rule, the rule type and status, and the settings for generating the rule scores. To know more about each property, see the Create a Standard Data Quality Rule topic in the Automate Data Quality Rules chapter. 5. Click Save and Close.
Associating Local Data Quality Rules to a Standard Data Quality Rule
If an existing local data quality rule that applies to an attribute is associated with the Glossary object, you can view it in the Local Data Quality Rules grid in the Data Quality tab of the Glossary object.
Define local data quality rules before you associate a local data quality rule to a standard data quality rule.
1. From Unison search, open the Glossary object where you want to associate a standard data quality rule.
2. Click the Data Quality tab.
3. Click the + icon below the local data quality rule to copy it as a standard data quality rule.
Manually Add Data Quality Report Data
Manually add Data Quality report data by editing a local data quality rule. You cannot add scores to a standard data quality rule.
1. From Unison search, open a data set object.
2. In the Data Quality tab, click the Rules sub-tab.
3. In the Refcolumn, click the data quality rule that you want to edit.
Axon displays the data quality rule.
4. Click the Report tab.
5. Click Edit.
6. Configure the following properties:
Option | Description |
|---|
Report Date | The date of the Data Quality report in the format DD/MM/YYYY. This is not the date the reports are uploaded to Axon, it is when the evaluation of the data was performed in your DQ system. Multiple reports for the same item and multiple items can be uploaded at the same time. |
Result | The score expressed as a number between 0 and 100, of items that meet the expectations of the quality rule. Axon displays this number as a percentage. Do not use a % sign when entering data. You can enter scores expressed with decimal places. Most Axon screens round off the score with a decimal value. |
Report URL | Link to the full report generated by the Data Quality system. |
Exception Count | Some Data Quality systems report the number of records that did not meet standards. Specify that information. |
Volume | Number of observations in the report. |
7. Click the + icon to add another row.
8. Click Save and Close.