Monitor MDM SaaS > Monitoring match model jobs > Monitoring match and merge jobs

Monitoring match and merge jobs

You can define a match job and a merge job separately or define a single match and merge job. You can also define a job to update the key metrics on the match analysis and explainability dashboard based on the most recent match jobs, and choose to reset the matched records.

After you run a job, you can monitor and view the job details. Each job performs a series of steps, and the steps vary based on the process type that you select.

After a match and merge job completes, you can view the job metrics, such as the total records processed for matching. You can evaluate the metrics to determine the performance of the match and merge job and the number of record pairs generated.

The following table lists the process types and their steps that are part of match and merge jobs:

Process Type	Process Steps
Match and Merge	Matches records and then merges the matching records based on the match model configuration. The match and merge process type contains the following steps: - Lock the Job - Match - Merge - Index for Search - Group Potential Matches - Release the Job
Match Only	Matches records based on the match model configuration. The match only process type contains the following steps: - Lock the Job - Match - Release the Job
Merge Only	Merges the matching records based on the match model configuration. The merge only process type contains the following steps: - Lock the Job - Identify Match Pair Groups - Merge - Index for Search - Group Potential Matches - Release the Job
Reset the Matched Records	Resets the matched records that are not yet merged. The reset the matched records process type contains the following steps: - Lock the Job - Reset - Release the Job
Update Match Explainability Metrics	Updates the key metrics, including large cluster metrics, on the match analysis and explainability dashboard based on the most recent match jobs. The update match explainability metrics process type contains the following steps: - Lock the Job - Update Explainability Metrics - Release the Job

Lock the job

The lock the job step locks the job instance so that you can run only a single job instance at a time for a business entity.

If the job fails at this step, check whether a similar job is already running.

Match records

The match step compares records based on the match model configuration and then identifies the matching records. You can evaluate the metrics and decide whether to enhance the match model configuration.

The following table describes the overall metrics:

Metric Name	Metric ID	Description
Total Records	MATCH_TOTAL_RECORDS	The total number of records that the step processed.
Skipped Records	MATCH_SKIPPED_RECORDS	The number of records that the step skipped because of data errors, such as data type mismatch. Fix the data errors so that the job can process the records.
Failed Records	MATCH_FAILED_RECORDS	The number of records that failed to match because of internal errors.
N/A	TOTAL_MATCH_JOBS	The total number of internal jobs that run to complete the match step. Note: To view this metric, copy the detailed metrics of this step in the JSON format.

The following table describes the metrics based on declarative rules:

Metric Name	Metric ID	Description
Records Pairs for Automatic Merge	MATCH_DECLARATIVE_AUTO_MERGE_PAIRS	The number of record pairs that will automatically merge.
Nonmatching Record Pairs	MATCH_DECLARATIVE_NEVER_MERGE_PAIRS	The number of record pairs that didn't match.
Record Pairs for Manual Review	MATCH_DECLARATIVE_STEWARD_REVIEW_PAIRS	The number of record pairs that require manual review based on declarative rules.

The following table describes the metrics based on the machine learning model:

Metric Name	Metric ID	Description
Total Record Pairs	MATCH_AI_MODEL_EVALUATION_PAIRS	The total number of record pairs available.
Records Pairs for Automatic Merge	MATCH_AI_AUTO_MERGE_PAIRS	The number of record pairs that will automatically merge.
Nonmatching Record Pairs	MATCH_AI_NEVER_MERGE_PAIRS	The number of record pairs that didn't match.

Identify match pair groups

The identify match pair groups step identifies groups of record pairs that are potential candidates for automated merges.

For example, if record A matches record B, record B matches record C, and record D matches record E, the step identifies four record pairs. The record pairs are A and B, B and C, A and C, and D and E. The step then groups record pairs into two match pair groups. One group contains A and B, B and C, and A and C while the other group contains D and E.

The following table describes the metrics of the identify match pair groups step:

Metric ID	Description
Total Record Pairs	Number of record pairs identified as potential candidates for automated merges.
Total Potential Groups	Number of match pair groups that can be formed.

Merge records

The merge step merges the records that are set for automatic merge based on declarative rules and the machine learning model.

The merge step forms groups of record pairs to efficiently merge the matching record pairs. Each group contains multiple record pairs.

The following table describes the metrics of the merge step:

Metric Name	Metric ID	Description
Total Record Pairs	number_of_match_pairs_read	The number of record pairs that the step processed.
Failed Groups	number_of_groups_that_failed_to_merge	The number of record pair groups that didn't merge due to internal errors, such as database connection errors.
Successful Groups	number_of_groups_merged	The number of record pair groups that the step successfully merged. The metric doesn't include the groups that require manual review and approval.
Pending Groups	number_of_pending_groups	The number of record pair groups that require manual review and approval.
N/A	number_of_groups	The total number of record pair groups that the step created. Note: To view this metric, copy the detailed metrics of this step in the JSON format.

Index records for search

The index for search step indexes all master records for search after the merge step.

The following table describes the metrics of the index for search step:

Metric ID	Description
SEARCH_INDEX_TOTAL_RECORDS	Total number of records that the step processed.
SEARCH_INDEX_FAILED_RECORDS	Number of records that failed to index.
OVERVIEW_SUCCESS_RECORDS	Number of records that are indexed for search requests.
SEARCH_INDEX_UPSERT_SUCCESS	Number of records for which the search indexes are added or updated.
SEARCH_INDEX_UPSERT_FAILURE	Number of records that failed to index for search.
SEARCH_INDEX_DELETE_SUCCESS	Number of records whose indexes were deleted.
SEARCH_INDEX_DELETE_FAILURE	Number of records whose indexes failed to be deleted.

Group potential matches

The group potential matches step optimizes the number of record pair groups that are candidates for manual merges by identifying and grouping related record pairs.

For example, record A matches with record B, and record B matches with record C. The match process might result in three record pair groups, one for A and B, one for B and C, and one for A and C. The group potential match pairs step identifies these record pair groups and merges them into one record pair group with records A, B, and C.

If you use an approval workflow, when you generate merge tasks, you get one approval task instead of three tasks.

If the number of record pairs or groups exceeds the processing limit, a warning message appears, and the match and merge job runs successfully without completing the group potential match pairs step. You must reconfigure the match model so that the number of record pairs or groups doesn't exceed the limit. After you configure the match model, you can run the reset the matched records job and then run another match and merge job.

The following table describes the metrics of the group potential matches step:

Metric ID	Description
TOTAL_PAIRS	Number of record pairs that are available for manual review.
NUMBER_OF_GROUPS	Number of groups that the step processed.
NUMBER_OF_MATCH_PAIRS_READ	Number of record pairs that are available for manual review.
number_of_pairs_exceeding_limit	Number of record pairs that exceeds the processing limit of 1,000,000 record pairs.
number_of_groups_exceeding_limit	Number of groups that exceeds the processing limit of 100,000 groups.

Update explainability metrics

The update explainability metrics step updates the key metrics on the match analysis and explainability dashboard.

The following table describes the metrics of the update explainability metrics step:

Metric ID	Description
MATCH_EXPLAINABILITY_JOB_DETAILS	Updates the key metrics on the explainability dashboard, except the large cluster metrics.
MATCH_EXPLAIN_LARGE_CLUSTER_JOB_DETAILS	Updates the large cluster metrics on the explainability dashboard. Appears only when the job updates large cluster metrics.
DATA_PROFILING_JOB_DETAILS	Updates the data profile details related to large clusters. Appears only when the job updates large cluster metrics.

Reset matched records

The reset step resets the already matched records that are yet to be merged. When you reset the matched records, the record state changes from MATCHED to MATCH_DIRTY. The MATCH_DIRTY record state indicates that the record did not undergo the match process.

The following table describes the metrics of the reset step:

Metric ID	Description
MATCH_TOTAL_RECORDS	The total number of records that step processed.
MATCH_FAILED_RECORDS	The number of records that failed to reset.
TOTAL_FAILED_STATE_TRANSIT_COUNT	The number of records that failed to reset as the record state didn't change.
TOTAL_MATCH_PAIRS_DELETED_COUNT	The number of match pairs that the step deleted after resetting the state of records.
TOTAL_RECORDS_RESET_STATE_TRANSIT_COUNT	The number of records for which the step reset the record state.

Release the job

The release the job step releases the locked job instance.