When you train an Adaptive AI match model, you need to evaluate the effectiveness of the model. After the model achieves the effectiveness that your business requires, you can complete the training.
To evaluate the effectiveness of the Adaptive AI match model, review the training metrics that appear on the Adaptive AI Match Model page.
The following Adaptive AI match model training metrics help answer your questions about model effectiveness:
An error matrix summarizes the performance of an Adaptive AI match model based on the training data set. Use the matrix to determine how successful the predictions of the model are by understanding the correlation between the actual labels and the predictions of the model.
The following 3 × 3 matrix explains what each cell in an Adaptive AI match model error matrix represents:
-
Predicted as Match
Predicted as Manual Review
Predicted as Not a Match
Labeled as Match
True positive (TP):
An outcome where the model correctly predicts a record pair as a match.
False positive (FP):
An outcome where the model incorrectly predicts that a matching record pair requires manual review.
False negative (FN):
An outcome where the model incorrectly predicts a record pair as not a match.
Labeled as Manual Review
False positive (FP):
An outcome where the model predicts a record pair that requires manual review as a match.
True positive (TP):
An outcome where the model correctly predicts that a record pair requires manual review.
False positive (FP):
An outcome where the model predicts a record pair that requires manual review as not a match.
Labeled as Not a Match
False positive (FP):
An outcome where the model incorrectly predicts a record pair that's not a match as a match.
False positive (FP):
An outcome where the model incorrectly predicts a record pair that's not a match requires manual review.
True negative (TN):
An outcome where the model correctly predicts a record pair as not a match.
The error matrix shows only 30% of the total record pairs that are labeled as a match, manual review, or not a match. For example, if you label 30 record pairs as a match, 20 record pairs as manual review, and 10 record pair as not a match, the error matrix shows the following values:
•30% of 30 record pairs, which is 9 pairs, predicted as a match.
•20% of 30 record pairs, which is 6 pairs, predicted as requiring manual review.
•30% of 10 record pairs, which is 3 pairs, predicted as not a match.
The following image shows an error matrix with record pairs predicted as matches, non-matches, and requiring manual review:
Accuracy
Accuracy measures how well an Adaptive AI match model correctly classifies records as a match, not a match, or requiring manual review.
Accuracy is the percentage of correct predictions out of the total number of predictions.
For example, if the accuracy of an Adaptive AI match model is 95%, it correctly classifies 95% of the record pairs out of the total number of record pairs.
Accuracy = (Total TPs + TN) / (Total TPs + TN + Total FPs + FN) = (Total TPs + TN) / Total number of record pairs
Where TP indicates true positives, TN indicates true negatives, FP indicates false positives, and FN indicates false negatives.
Precision
Precision measures how many of the matches that the Adaptive AI match model predicts are actually correct matches.
Precision is the percentage of matches that the Adaptive AI match model correctly predicts out of the total number of matches it predicts.
For example, if 80% of the record pairs that the Adaptive AI match model identifies as matches are true positives, the precision of the model is 80%.
Precision = Total TPs / (Total TPs + Total FPs) = Total TPs / Total number of predicted matches
Where TP indicates true positives and FP indicates false positives.
Recall
Recall measures how well an Adaptive AI match model can find all the correct matches.
Recall is the percentage of matches that the Adaptive AI match model correctly predicts out of the total number of actual matches.
For example, if the recall of an Adaptive AI match model is 50%, it correctly identifies 50% of record pairs that are a match.
Recall = Total TPs / (Total TPs + FN) = Total TPs / Number of actual matches
Where TP indicates true positives and FN indicates false negatives.
Learning curve
After you complete the training of record pairs in a batch, you can view a learning curve chart that displays the match training metrics, such as precision, recall, and accuracy.
The learning curve shows the precision, recall, and accuracy for record pairs labeled in each batch individually. You can use these metrics to compare and gain a better understanding of how the model improves as you label record pairs across batches.
The following image shows the training metrics for record pairs labeled across different batches:
