The transform step applies basic, advanced, and DaaS rule associations to the staged data. The rule associations cleanse and transform the staged data.
If an ingress job fails at this step or the step is in the Running status for a long time, go to the My Jobs page of Data Integration to troubleshoot. If the taskflow is in the Suspended status, view the subtasks and select the failed mapping task. Download the sessions log and check for errors.
Note: When you download the sessions log, you can view up to 10,000 skipped records.
The transform step applies data quality rule associations that are added to the mapped fields in Cloud Data Integration.
The following table lists the metrics that you can view for the transform step:
Metric
Description
Total Records
The total number of records that the step processed.
Error Records
The number of records that failed the data quality rule associations.
Success Records
The number of records that are validated and transformed and moved to the load step.
Starting Records for Asset
The total number of records that the step processed for a specific asset.
Skipped Records
The number of records that the step skipped because of some internal issues.
Applied Data Quality Rules
The number of data quality rule associations that the step applied on the records.
Failed Data Quality Rules
The number of data quality rules that returned errors.
Successful Data Quality Rules
The number of data quality rules that successfully validated records.
Skipped records
The ingress job might skip some of the records that you ingress during the transform step. The records are skipped for the following reasons:
•Records don't contain values in any of the fields.
•The source primary key field value of records contains special characters.
•Multiple records contain the same source primary key value and the number of records exceeds the threshold limit. By default, the threshold limit is set to 30,000.
•The size of a record exceeds 13 MB.
•Records contain data errors, such as a data type mismatch or when the value of a data type exceeds its maximum range.
•Records contain picklist values that are invalid.
Viewing skipped records
You can download the list of skipped records as a CSV file. The file includes the source and parent primary key values and error descriptions of the skipped records.
Note: MDM SaaS stores the skipped records in the database for seven days. Ensure that you download the skipped records within the specified time period.
1On the My Jobs page, open an ingress job, and click Transform > Errors.
2In the Rejected Records column, click Download.
The list of skipped records downloads as a CSV file.
When you download the skipped records in the transform step, the CSV file doesn't include values in the target flat endpoint column for some records.
The transform step processes each target flat endpoint of the records and then merges them. If any error occurs when the step processes the individual target flat endpoints of the records, the CSV file includes the target flat endpoint for those records. If any error occurs after the step merges all target flat endpoints of the records, the CSV file doesn't include the target flat endpoint.
For example, you ingress a Person record that contains an address field group. When you run the ingress job, the transform step processes the target flat endpoints, such as person.root and person.address. After processing the target flat endpoints, the step merges the target flat endpoints.
If errors occur while processing the individual target flat endpoints, the CSV file includes the target flat endpoints for the Person record. If error occurs while merging the target flat endpoints, the CSV file doesn't include the target flat endpoints.