Evaluation metrics enable you to record additional data that pertain to an AI model asset. You can use evaluation metrics to compare various models, identify potential biases, and monitor model behavior over time.
On the Customize > Asset Customization tab in Metadata Command Center, you can add new evaluation metrics or modify or remove existing evaluation metrics for AI Model assets. For more information about AI Model assets, see the Understanding Business Assets help for Data Governance and Catalog.
To have permissions to create, modify, or delete evaluation metrics, define appropriate roles and select the Manage Custom Attributes feature for that role when you configure privileges in Metadata Command Center in Administrator. For more information about feature privileges that the organization administrator can configure for user roles, see the Introduction and Getting Started help.
Creating evaluation metrics
You can create new evaluation metrics for the AI model asset type.
1On the Customize page in Metadata Command Center, go to the Asset Customization tab.
2In the Asset Types panel, select the AI Model asset type.
3For the AI Model asset type, go to the Evaluation Metrics tab.
4Click the Add icon.
Alternatively, you can use the Create button to add a new evaluation metric.
5Configure the following properties of the new evaluation metric:
Property
Description
Name
Unique name for the evaluation metric that will be displayed in Data Governance and Catalog.
Ensure that the metric name doesn't match a name that already exists.
Description
Description of the evaluation metric.
Value Type
The type of value that users can enter in the metric. You can choose one of the following value types:
- Decimal
- Integer
- Percentage
Default value is Decimal.
6Click Save.
This creates a draft version of the new evaluation metric that you can view in the metrics list on the Evaluation Metrics tab.
7Click the name of the evaluation metric that you saved to view and verify the details of the metric. To modify the properties of the metric, edit the values and click Save.
Note: If you do not want to proceed with your changes, you can use the Discard icon to discard the draft version of the evaluation metric. You can also perform this action from the Action menu on the Asset Customization > Evaluation Metrics tab of the Customize page. On the Asset Customization > Evaluation Metrics tab, you can also discard the draft version of multiple evaluation metrics. To do this, select the evaluation metrics for which you want to discard the draft state and click the Discard icon from the Action menu.
8Click Publish to publish the metric in Data Governance and Catalog.
The system initiates the Attributes Publish job to publish your changes. If another job to create, modify or delete an attribute, evaluation metric or relationship is active, the system informs you of the active job. To publish your changes, you must wait for the active job to be completed. On the Job Monitoring Overview page, you can click View Status to monitor the status of the job.
Note: The system displays an error if the previous job has failed. At this stage, you must resolve the errors and reinitiate the previous job and await its completion before you can start your own job.
After the publishing is successful, the lifecycle state of the evaluation metric changes from Draft to Published.
The new metric appears on the Overview page of all AI Model assets in Data Governance and Catalog.
Modifying evaluation metrics
You can modify a predefined evaluation metric or a custom evaluation metric that you created.
1On the Customize page in Metadata Command Center, go to the Asset Customization tab.
2In the Asset Types panel, select the AI Model asset type.
3For the AI Model asset type, open the Evaluation Metrics tab.
4On the Evaluation Metrics tab, select the metric that you want to modify.
You can use filters to assist you in your search for evaluation metrics. You can also use filters to view only the predefined metrics or to view only the custom metrics.
5Depending on the lifecycle state of the evaluation metric, choose one of the following options:
- If the metric is in the Draft state, you can modify any property of the metric.
- If the metric is in the Published state, open the metric and click Create a draft. You can modify only some of its properties.
6Click Save.
Note: If you do not want to proceed with your changes, you can use the Discard icon to discard the draft version of the evaluation metric. You can also perform this action from the Action menu on the Asset Customization > Evaluation Metrics tab of the Customize page. On the Asset Customization > Evaluation Metrics tab, you can also discard the draft version of multiple evaluation metrics. To do this, select the evaluation metrics for which you want to discard the draft state and click the Discard icon from the Action menu.
7Click Publish to publish your changes.
You can also perform this action from the Action menu on the Asset Customization > Evaluation Metrics tab of the Customize page. On the Asset Customization > Evaluation Metrics tab, you can also publish multiple evaluation metrics. To do this, select the evaluation metrics that you want to publish and click the Publish button from the Action menu.
The system initiates the Attributes Publish job to publish your changes. If another job to create, modify or delete an attribute, evaluation metric or relationship is active, the system informs you of the active job. To publish your changes, you must wait for the active job to be completed. On the Job Monitoring Overview page, you can click View Status to monitor the status of the job.
Note: The system displays an error if the previous job had failed. At this stage, you must resolve the errors and reinitiate the previous job and await its completion before you can start your own job.
The status of the evaluation metric changes to Published and the modified attribute appears in Data Governance and Catalog.
8After the publish job is completed, ensure that you refresh the web page on your browser to view the changes to the evaluation metric.
Deleting evaluation metrics
If an evaluation metric is no longer relevant or if you want to modify the properties of a metric, you can delete the metric and create a new metric. Deleting an evaluation metric removes the metric in Data Governance and Catalog along with all the values that are defined for the metric.
1On the Customize page in Metadata Command Center, go to the Asset Customization tab.
2In the Asset Types panel, select the AI Model asset type.
3For the AI Model asset type, open the Evaluation Metrics tab.
4Select the check box for one or more evaluation metrics.
5Click the Action menu for any of the selected metrics and click Delete.
If you have selected more than one evaluation metric to delete, then the Action menu for the selected group appears only if all the evaluation metrics in the group are in the Published or Published with draft available lifecycle states.
6Click Delete to confirm.
The system initiates the Attributes Publish job to publish your changes. If another job to create, modify or delete an attribute, evaluation metric or relationship is active, the system informs you of the active job. To publish your changes, you must wait for the active job to be completed. On the Job Monitoring Overview page, you can click View Status to monitor the status of the job.
Note: The system displays an error if the previous job had failed. At this stage, you must resolve the errors and reinitiate the previous job and await its completion before you can start your own job.
