Configure Data Quality > Orchestrating data enrichment and validations > Creating Application Integration processes for data enrichment and validations
  

Creating Application Integration processes for data enrichment and validations

Before you configure data enrichment and validations, create processes based on the type of operation you want to perform, such as enrichment, validation, or cleansing.
When you create rule associations in MDM SaaS, you can use the following plugins to enrich, validate, and cleanse records:
Each of these plugins sends and accepts payloads in predefined formats.
A process accepts a set of fields as input from MDM SaaS and sends a request to an external enrichment and validation service. When the service returns enrichment or validation data, the process sends a response back to MDM SaaS. The response for a data enrichment request can include a set of output fields containing data from the external enrichment service. The response for a validation result includes the validation results.
When you create a process, you need to define the input fields that MDM SaaS needs to send to the process. You also need to define the output payload that the process needs to send to MDM SaaS.
To define the input and output fields, you need to create process objects that you can use within your process. Process objects group and structure data for use in processes. For example, instead of creating separate items for each type of demographic data, you can have a process object that contains the age, education level, income range, and ethnicity fields. For more information about designing process objects, see Designing Process Objects in the Application Integration help.

Payload formats for Application Integration-based enrichment

When you create a process to use with the Application Integration-based Enrichment plugin, ensure that the payload formats of the process are compatible with the plugin.
In the input payload, define the input fields that MDM SaaS needs to send to the process when a record is created or updated. In the output payload, define the output fields that the process needs to send back to MDM SaaS. If the enrichment request is successful, the data from the output fields are applied to the mapped business entity fields. If you create a process to assign records to a hierarchy, send the business IDs of relevant records in the output payload.
You can include confidence scores in your output payload. If the payload includes confidence scores, CLAIRE recommends the enrichment results as recommendations in business applications based on the data enrichment configuration.

Input payload format

The following table lists the parameters that the input payload of an Application Integration process for data enrichment:
Field
Data Type
Description
docId
Text
A system-generated unique identifier of the record for which data enrichment is triggered.
payload
Object List
List of fields that you want to display as input fields in rule associations. It also includes the sys field.
payload > sys
Reference
A system field that references the groupId field.
Note: Include this field at the root level of the payload field.
payload > (field group) > sys
Reference
A system field that references the groupId field. If you add a field group in the payload, include the sys field in each process object for the field groups.
sys > groupId
Text
A system-generated unique identifier of a field group entry.
The following sample JSON snippet represents an input payload that MDM SaaS sends to a process when a record is created or updated in a business application:
[
{
"docId": "c360.item-xyz456",
"payload": [
{
"address": {
"city": "clarence town",
"addressLine1": "23 Gaggin St",
"sys": {
"groupId": "acd4985"
}
},
"sys": {
"groupId": "acd4986"
}
}
]
}
]
For more information about creating process objects for the required fields, see Creating a process for data enrichment.

Output payload format

The response from a process to MDM SaaS must contain an array of records.
The following table lists the parameters that you need to include in the output payload of the process:
Parameter
Data Type
Description
docId
Text
A system-generated unique identifier of a record for which data enrichment is triggered.
payload
Object List
A list of fields that you receive from a data provider.
payload > sys
Reference
Optional. A system field that references the groupId field.
payload > (field group) > sys
Reference
Optional. A system field that references the groupId field.
If an output field is a field group, include this parameter to return the ID of the field group entry that needs to be updated.
payload > (confidence score field)
Text
Confidence score to display enrichment data as recommendations.
Specify a value between 0 to 100.
sys > groupId
Text
Optional. A system-generated unique identifier of a nested field group entry.
Based on the groupId value, the field group entries are enriched.
If the groupId value isn't available, a new field group entry is created.
The following sample JSON snippet represents an output payload that a process sends to MDM SaaS as a result of data enrichment:
[
{
"docId": "c360.item-xyz456",
"payload": [
{
"cleansedAddress": [
{
"address1": "23rd Gaggin Street",
"city": "clarence town",
"country": "Australia",
"PinCode": 5432,
"sys": {
"groupId": "acd4985"
}
}
],
"matchConfidence": 57
}
]
}
]
For more information about creating process objects for the required fields, see Creating a process for data enrichment.

Creating a process for data enrichment

To define a set of fields that you want to display as input and output fields in Business 360 Console, create a process and define process objects that contain the fields.
As a best practice, define input and output fields based on the data model of the business entity.
    1In Application Integration, create a folder within a project.
    2Create a process object named System with the text field named groupId.
    The following image shows a sample process object named System with the field named groupId:
    3To define the payload for the request that MDM SaaS sends, create a process object named InputPayload.
    The following image shows a sample process object named InputPayload with two fields named Addresses and sys:
    4To define payload for the response to MDM SaaS, create a process object named OutputPayload.
    The following image shows a sample process object named OutputPayload with two fields named confidenceScoreInPercent and Addresses:
    5To define the request structure, create a process object named InputPerRecord.
    The following image shows a process object named InputPerRecord with two fields named payload and docId:
    6To define the response structure, create a process object named OutputPerRecord.
    The following image shows a process object named OutputPerRecord with two fields named docId and payload:
    7Create a process.
    8In the Start Properties panel of the process, perform the following steps:
    1. aClick the Start tab, and select Allow anonymous access.
    2. bClick the Input Fields tab.
    3. cSelect Whole Payload, add an input field named request, and set its type to List of InputPerRecord by selecting the InputPerRecord process object.
    4. The following image shows the input field named request and its type as List of InputPerRecord:
    5. dClick the Output Fields tab.
    6. eSelect Whole Payload, add an output field named response, and set its type to List of OutputPerRecord by selecting the OutputPerRecord process object.
    7. The following image shows the output field named response and its type as List of OutputPerRecord:
    9Based on your enrichment requirements, add necessary process steps to the process.

Payload formats for Application Integration-based validation

When you create a process for the Application Integration-based Validation plugin, ensure that the payload formats of the process are compatible with the plugin.
A process used for validation either validates the input fields or uses an external validation service to validate them.
When fields fail validations, return the names of the input fields to which the business entity fields are mapped. For example, if the First Name business entity field is mapped to the FirstName input field and the value of First Name fails validation, return the FirstName input field in the response. MDM SaaS resolves the input fields to the business entity fields to display validation errors correctly.
When you send a response, you can include a value for downgrading the trust score of the fields that fail validation. The value for downgrading the trust score must be between 0 to 100.

Input payload format

The following table lists the parameters of the input payload for a process for data validation:
Field
Data Type
Description
docId
Text
A system-generated unique identifier of the record for which data enrichment is triggered.
payload
Object List
List of fields that you want to display as input fields in rule associations. It also includes the sys field.
payload > sys
Reference
A system field that references the groupId field.
Note: Include this field at the root level of the payload field.
payload > (field group) > sys
Reference
A system field that references the groupId field. If you add a field group in the payload, include the sys field in each process object for the field groups.
sys > groupId
Text
A system-generated unique identifier of a field group entry.
The following sample JSON snippet represents the input payload that MDM SaaS sends to a process when a record is created or updated in a business application:
[
{
"docId": "business_entity_master_id1",
"payload": [
{
"drugCode": " 12345 - 6789 -01 ", // Extra spaces and incorrect spacing around hyphens
"regulatoryInformation": [
{
"approvalStatus": "apprved", // Typo: should be 'Approved'
"regulatoryAuthority": "fda ", // Lowercase + trailing space instead of uppercase trimmed
"patentExpiry": "2038.08.14", // Wrong date separator, should be dashes
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e" // Incomplete UUID format
}
}
],
"sys": {
"recordId": "payload-01" // Root level sys info for the payload record
}
}
]
}
]

Output payload format

The response from a process to MDM SaaS must contain an array of records.
The following table lists the parameters that you need to include in the output payload of the process:
Parameter
Data Type
Description
docId
Text
A system-generated unique identifier of a record for which data enrichment is triggered.
validation
Object List
An array of validation results.
validation > passed
Boolean
A value of false if validation fails. A value of true if validation passes.
validation > fields
Object List
An array of fields that fail validation.
fields >fieldName
Text
Name of the input field that fails validation. Ensure that the name of the field matches the input field.
fields > dtsp
Integer
Optional. The value for downgrading trust score of the field that fails validation.
fields > fieldName > sys
Reference
Optional. A system field that references the groupId field. If the input field is a field group, include this parameter to send the ID of the field group entry.
sys > groupId
Text
Optional. A system-generated unique identifier of a nested field group entry.
Based on the groupId value, the field group entries are enriched.
If the groupId value isn't available, a new field group entry is created.
The following sample JSON snippet represents an output payload that a process sends to MDM SaaS as a result of data enrichment:
[
{
"docId": "business_entity-master_id1",
"validation": [
{
"passed": false,
"fields": [
{
"fieldName": "drugCode",
"dtsp": 23
},
{
"fieldName": "regulatoryInformation",
"dtsp": 23,
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e"
}
}
]
}
]
}
]
The following sample output payload shows a response for a successful validation:
[
{
"docId": "business_entity-master_id1",
"validation": [
{
"passed": true
}
]
}
]

Creating a process for validations

To perform Application Integration-based validations, create a process and define the input fields and response of the process.
As a best practice, define input and output fields based on the data model of the business entity.
    1In Application Integration, create a folder within a project.
    2Create a process object named System with the text field named groupId.
    The following image shows a sample process object named System with the field named groupId:
    3To define the payload for the request that MDM SaaS sends, create a process object. For example, create a process object named InputPayload.
    The following image shows a sample process object named InputPayload with four fields named FieldToShowValidationMessage, phone, email, and sys:
    4Create a process objects to define fields for validation results. For example, create a process object named fields.
    1. aTo return the name of the field that fail validation, add a text field named fieldName.
    2. bTo return a value to downgrade trust score, add a text field named dtsp.
    3. cTo return the field group ID if the input field is a field group, add a field named sys, and reference the System process object that you created in step 2.
    The following image shows a sample process object named fields with three fields named fieldName, dtsp, and sys:
    5To define payload for the response to MDM SaaS, create a process object. For example, create a process object named ValidationPayload.
    The following image shows a sample process object named ValidationPayload with two fields named passed and fields:
    6To define the request structure, create a process object. For example, create a process object named InputPerRecord.
    The following image shows a process object named InputPerRecord with two fields named payload and docId:
    7To define the response structure, create a process object. For example, create a process object named OutputPerRecord.
    The following image shows a process object named OutputPerRecord with two fields named docId and validation:
    8Create a process.
    9In the Start Properties panel of the process, perform the following steps:
    1. aClick the Start tab, and select Allow anonymous access.
    2. Note: Ensure that the Run On property is set to Cloud Server.
    3. bClick the Input Fields tab.
    4. cSelect Whole Payload, add an input field named request, and set its type to List of InputPerRecord by selecting the InputPerRecord process object.
    5. The following image shows the input field named request and its type as List of InputPerRecord:
    6. dClick the Output Fields tab.
    7. eSelect Whole Payload, add an output field named response, and set its type to List of OutputPerRecord by selecting the OutputPerRecord process object.
    8. The following image shows the output field named response and its type as List of OutputPerRecord:
    10Based on your enrichment requirements, add necessary process steps to the process.

Payload formats for Application Integration-based cleansing

When you create a process to use with the Application Integration-based Cleansing plugin, ensure that the payload formats of the process are compatible with the plugin.
A process created to perform cleansing validates or uses an external data provider to validate the input fields. If the input fields fail validation, the process cleanses the data and sends the cleansed data back to MDM SaaS. When you send a response back to MDM SaaS, you can include a value for downgrading trust score. If you don't include validation results in the response, MDM SaaS considers the validation to be successful.

Input payload format

The following table lists the parameters that the input payload of an Application Integration process for data enrichment:
Field
Data Type
Description
docId
Text
A system-generated unique identifier of the record for which data enrichment is triggered.
payload
Object List
List of fields that you want to display as input fields in rule associations. It also includes the sys field.
payload > sys
Reference
A system field that references the groupId field.
Note: Include this field at the root level of the payload field.
payload > (field group) > sys
Reference
A system field that references the groupId field. If you add a field group in the payload, include the sys field in each process object for the field groups.
sys > groupId
Text
A system-generated unique identifier of a field group entry.
The following sample JSON snippet represents an input payload that MDM SaaS sends to a process for validation:
[
{
"docId": "business_entity_master_id1",
"payload": [
{
"drugCode": " 12345 - 6789 -01 ", // Extra spaces and incorrect spacing around hyphens
"regulatoryInformation": [
{
"approvalStatus": "apprved", // Typo: should be 'Approved'
"regulatoryAuthority": "fda ", // Lowercase + trailing space instead of uppercase trimmed
"patentExpiry": "2038.08.14", // Wrong date separator, should be dashes
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e" // Incomplete UUID format
}
}
],
"sys": {
"recordId": "payload-01" // Root level sys info for the payload record
}
}
]
}
]

Output payload format

The response from a process to MDM SaaS must contain an array of records.
The following table lists the parameters that you need to include in the output payload of the process:
Parameter
Data Type
Description
docId
Text
A system-generated unique identifier of a record for which data enrichment is triggered.
validation
Object List
An array of validation results.
validation > passed
Boolean
A value of false if validation fails. A value of true if validation passes.
validation > fields
Object List
An array of fields that fail validation.
fields >fieldName
Text
Name of the input field that fails validation. Ensure that the name of the field matches the input field.
fields > dtsp
Integer
Optional. The value for downgrading trust score of the field that fails validation.
fields > fieldName > sys
Reference
Optional. A system field that references the groupId field. If the input field is a field group, include this parameter to send the ID of the field group entry.
payload
Object List
A list of fields that you receive from a data provider.
payload > sys
Reference
Optional. A system field that references the groupId field.
payload > (field group) > sys
Reference
Optional. A system field that references the groupId field.
If an output field is a field group, include this parameter to return the ID of the field group entry that needs to be updated.
sys > groupId
Text
Optional. A system-generated unique identifier of a nested field group entry.
Based on the groupId value, the field group entries are enriched.
If the groupId value isn't available, a new field group entry is created.
The following output field adds a new entry of the regulatoryInformation field group and updates the drugCode field with the cleansed value:
[
{
"docId": "business_entity-master_id1",
"validation": [
{
"passed": false,
"fields": [
{
"fieldName": "drugCode",
"dtsp": 23
},
{
"fieldName": "regulatoryInformation",
"dtsp": 23
}
]
}
],
"payload": [
{
"drugCode": "12345-6789-01", // Trimmed spaces, standardized hyphen placement
"regulatoryInformation": [
{
"approvalStatus": "Approved", // Corrected spelling and capitalized
"regulatoryAuthority": "FDA", // Uppercase and no trailing spaces
"patentExpiry": "2038-08-14" // Standardized date separator
}
]
}
]
}
]
The following sample output payload updates the existing regulatoryInformation field group entry and updates the drugCode field with the cleansed data:
[
{
"docId": "business_entity-master_id1",
"validation": [
{
"passed": false,
"fields": [
{
"fieldName": "drugCode",
"dtsp": 23
},
{
"fieldName": "regulatoryInformation",
"dtsp": 23,
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e"
}
}
]
}
],
"payload": [
{
"drugCode": "12345-6789-01", // Trimmed spaces, standardized hyphen placement
"regulatoryInformation": [
{
"approvalStatus": "Approved", // Corrected spelling and capitalized
"regulatoryAuthority": "FDA", // Uppercase and no trailing spaces
"patentExpiry": "2038-08-14", // Standardized date separator
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e" // Complete and valid UUID
}
}
],
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e" // Complete and valid UUID (root level)
}
}
]
}
]
The following sample output payload validates and updates a field in the regulatoryInformation field group:
[
{
"docId": "business_entity-master_id1",
"validation": [
{
"passed": false,
"fields": [
{
"fieldName": "drugCode",
"dtsp": 23
},
{
"fieldName": "regulatoryInformation.approvalStatus",
"dtsp": 23,
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e"
}
}
]
}
],
"payload": [
{
"drugCode": "12345-6789-01", // Trimmed spaces, standardized hyphen placement
"regulatoryInformation": [
{
"approvalStatus": "Approved" // Corrected spelling and capitalized
],
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e" // Complete and valid UUID (root level)
}
}
]
}
]
The following sample output payload shows a response for a successful validation:
[
{
"docId": "business_entity-master_id1",
"validation": [
{
"passed": true
}
],
"payload": [
{
"drugCode": "12345-6789-01",
"regulatoryInformation": [
{
"approvalStatus": "Approved",
"regulatoryAuthority": "FDA",
"patentExpiry": "2038-08-14",
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e"
}
}
]
}
]
}
]

Guidelines for creating Application Integration processes

Consider the following guidelines when you create an Application Integration process:

Data types applicable to enrichment, validation and cleansing

When you create an Application Integration process for data enrichment, you can specify the following data types for the fields in the process:
Don't use the following data types for any fields in a process:

Compatible data types of process fields and business entity fields

When you add input fields in an Application Integration process, select data types that are compatible with the data types of business entity fields. This approach helps you map compatible fields when you configure rule associations.
The following table lists the compatible data types of business entity fields:
Data types of business entity fields
Data types of Application Integration process fields
Double
Number, Text
Integer
Integer, Number, Text
Date
Date, Text
Date and Time
Date Time, Text
Picklist
Picklist, Text
Text
Text, Integer, Number, Date, Date Time, Picklist
Boolean
Text, Checkbox
Decimal
Text
When you add output fields in an Application Integration process, select data types that are compatible with the data types of the business entity fields. This approach helps you map compatible fields when you configure rule associations.
The following table lists the compatible data types of a process:
Data types of Application Integration process fields
Data types of business entity fields
Integer
Integer, Double, Decimal, Text
Number
Double, Text
Text
Integer, Double, Decimal, Date, Date and Time, Boolean, and Text, Picklist
Picklist
Picklist, Text
Date
Date, Text
Date Time
Date and Time, Text
Checkbox
Text, Boolean
Note: The decimal data type isn’t available in Application Integration. Use a text field in the process for decimal values. When you map input and output fields in a rule association, you can map a decimal business entity field to a text field. For more information about the guidelines for field mappings, see Guidelines for field mappings.