Configure Data Quality > Orchestrating data enrichment and validations > Creating Application Integration processes for data enrichment and validations

Creating Application Integration processes for data enrichment and validations

Before you configure data enrichment and validations, create processes based on the type of operation you want to perform, such as enrichment, validation, or cleansing.

When you create rule associations in MDM SaaS, you can use the following plugins to enrich, validate, and cleanse records:

•Application Integration-based Enrichment
•Application Integration-based Validation
•Application Integration-based Cleansing

Each of these plugins sends and accepts payloads in predefined formats.

A process accepts a set of fields as input from MDM SaaS and sends a request to an external enrichment and validation service. When the service returns enrichment or validation data, the process sends a response back to MDM SaaS. The response for a data enrichment request can include a set of output fields containing data from the external enrichment service. The response to a validation request specifies whether the validation passed or failed. If the validation fails, the response must contain the list of fields that failed validation.

When you create a process, you need to define the input fields that MDM SaaS needs to send to the process. You also need to define the output payload that the process needs to send to MDM SaaS.

To define the input and output fields, you need to create process objects that you can use within your process. Process objects group and structure data for use in processes. For example, instead of creating separate items for each type of demographic data, you can have a process object that contains the age, education level, income range, and ethnicity fields. For more information about designing process objects, see Designing Process Objects in the Application Integration help.

Payload formats for Application Integration-based enrichment

When you create a process to use with the Application Integration-based Enrichment plugin, ensure that the payload formats of the process are compatible with the plugin.

In the input payload, define the input fields that MDM SaaS needs to send to the process when a record is created or updated. In the output payload, define the output fields that the process needs to send back to MDM SaaS. If the enrichment request is successful, the data from the output fields are applied to the mapped business entity fields. If you create a process to assign records to a hierarchy, send the business IDs of relevant records in the output payload.

You can include confidence scores in your output payload. If the payload includes confidence scores, CLAIRE recommends the enrichment results as recommendations in business applications based on the data enrichment configuration.

Input payload format

The following table lists the parameters that the input payload of an Application Integration process for data enrichment:

Field	Data Type	Description
docId	Text	A system-generated unique identifier of the record for which data enrichment is triggered.
payload	Object List	List of fields that you want to display as input fields in rule associations. It also includes the sys field.
payload > sys	Reference	A system field that references the groupId field. Note: Include this field at the root level of the payload field.
payload > (field group) > sys	Reference	A system field that references the groupId field. If you add a field group in the payload, include the sys field in each process object for the field groups.
sys > groupId	Text	A system-generated unique identifier of a field group entry.

The following sample JSON snippet represents an input payload that MDM SaaS sends to a process when a record is created or updated in a business application:

[
{
"docId": "c360.item-xyz456",
"payload": [
{
"address": {
"city": "clarence town",
"addressLine1": "23 Gaggin St",
"sys": {
"groupId": "acd4985"
}
},
"sys": {
"groupId": "acd4986"
}
}
]
}
]

For more information about creating process objects for the required fields, see Creating a process for data enrichment.

Output payload format

The response from a process to MDM SaaS must contain an array of records.

The following table lists the parameters that you need to include in the output payload of the process:

Parameter	Data Type	Description
docId	Text	A system-generated unique identifier of a record for which data enrichment is triggered.
payload	Object List	A list of fields that you receive from a data provider.
payload > sys	Reference	Optional. A system field that references the groupId field.
payload > (field group) > sys	Reference	Optional. A system field that references the groupId field. If an output field is a field group, include this parameter to return the ID of the field group entry that needs to be updated.
payload > (confidence score field)	Text	Confidence score to display enrichment data as recommendations. Specify a value between 0 to 100.
sys > groupId	Text	Optional. A system-generated unique identifier of a nested field group entry. Based on the groupId value, the field group entries are enriched. If the groupId value isn't available, a new field group entry is created.

The following sample JSON snippet represents an output payload that a process sends to MDM SaaS as a result of data enrichment:

[
{
"docId": "c360.item-xyz456",
"payload": [
{
"cleansedAddress": [
{
"address1": "23rd Gaggin Street",
"city": "clarence town",
"country": "Australia",
"PinCode": 5432,
"sys": {
"groupId": "acd4985"
}
}
],
"matchConfidence": 57
}
]
}
]

The following sample JSON snippet represents an output payload that includes confidence scores for enrichment recommendations:

[
{
"payload": [
{
"category": "500_Damen_Blusen",
"confidence": 65
},
{
"category": "500_Waesche_Mieder",
"confidence": 58
},
{
"category": "300_AudioSpieler",
"confidence": 25
}
],
"docId": "1sd54bd"
}
]

This sample payload returns the business IDs of three category records along with confidence scores. CLAIRE displays these records as recommendations based on the configuration in the related rule association.

For more information about creating process objects for the required fields, see Creating a process for data enrichment.

Creating a process for data enrichment

To define a set of fields that you want to display as input and output fields in Business 360 Console, create a process and define process objects that contain the fields.

As a best practice, define input and output fields based on the data model of the business entity.

1In Application Integration, create a folder within a project.

2Create a process object named System with the text field named groupId.

The following image shows a sample process object named System with the field named groupId:

3To define the payload for the request that MDM SaaS sends, create a process object named InputPayload.

- Add the data provider fields that you want to display as input fields in Business 360 Console.
- Add a field named sys, and reference the System process object that you created in step 2.

The following image shows a sample process object named InputPayload with two fields named Addresses and sys:

4To define payload for the response to MDM SaaS, create a process object named OutputPayload.

- Add the data provider fields that you want to display as output fields in Business 360 Console.
- To display the data from the data provider as recommendations, add a text field named confidenceScoreInPercent.

The following image shows a sample process object named OutputPayload with two fields named confidenceScoreInPercent and Addresses:

5To define the request structure, create a process object named InputPerRecord.

- Add a field named payload, set its type to ObjectList, and select the InputPayload process object.
- Add a text field named docId.

The following image shows a process object named InputPerRecord with two fields named payload and docId:

6To define the response structure, create a process object named OutputPerRecord.

- Add a text field named docId.
- Add a field named payload, set its type to ObjectList, and select the OutputPayload process object.

The following image shows a process object named OutputPerRecord with two fields named docId and payload:

7Create a process.

8In the Start Properties panel of the process, perform the following steps:

aClick the Start tab, and select Allow anonymous access.
bClick the Input Fields tab.
cSelect Whole Payload, add an input field named request, and set its type to List of InputPerRecord by selecting the InputPerRecord process object.

The following image shows the input field named request and its type as List of InputPerRecord:

dClick the Output Fields tab.
eSelect Whole Payload, add an output field named response, and set its type to List of OutputPerRecord by selecting the OutputPerRecord process object.

The following image shows the output field named response and its type as List of OutputPerRecord:

9Based on your enrichment requirements, add necessary process steps to the process.

- Add a service connection.
- Add steps to receive input fields from MDM SaaS and send output fields to MDM SaaS.

Payload formats for Application Integration-based validation

When you create a process for the Application Integration-based Validation plugin, ensure that the payload formats of the process are compatible with the plugin.

A process used for validation either validates the input fields or uses an external validation service to validate them.

When fields fail validations, return the names of the input fields to which the business entity fields are mapped. For example, if the First Name business entity field is mapped to the FirstName input field and the value of First Name fails validation, return the FirstName input field in the response. MDM SaaS resolves the input fields to the business entity fields to display validation errors correctly.

When you send a response, you can include a value for downgrading the trust score of the fields that fail validation. The value for downgrading the trust score must be between 0 to 100.

Input payload format

The following table lists the parameters of the input payload for a process for data validation:

Field	Data Type	Description
docId	Text	A system-generated unique identifier of the record for which data enrichment is triggered.
payload	Object List	List of fields that you want to display as input fields in rule associations. It also includes the sys field.
payload > sys	Reference	A system field that references the groupId field. Note: Include this field at the root level of the payload field.
payload > (field group) > sys	Reference	A system field that references the groupId field. If you add a field group in the payload, include the sys field in each process object for the field groups.
sys > groupId	Text	A system-generated unique identifier of a field group entry.

The following sample JSON snippet represents the input payload that MDM SaaS sends to a process when a record is created or updated in a business application:

[
{
"docId": "business_entity_master_id1",
"payload": [
{
"drugCode": " 12345 - 6789 -01 ", // Extra spaces and incorrect spacing around hyphens
"regulatoryInformation": [
{
"approvalStatus": "apprved", // Typo: should be 'Approved'
"regulatoryAuthority": "fda ", // Lowercase + trailing space instead of uppercase trimmed
"patentExpiry": "2038.08.14", // Wrong date separator, should be dashes
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e" // Incomplete UUID format
}
}
],
"sys": {
"recordId": "payload-01" // Root level sys info for the payload record
}
}
]
}
]

Output payload format

The response from a process to MDM SaaS must contain an array of records.

The following table lists the parameters that you need to include in the output payload of the process:

Parameter	Data Type	Description
docId	Text	A system-generated unique identifier of a record for which data enrichment is triggered.
validation	Object List	An array of validation results.
validation > passed	Boolean	A value of false if validation fails. A value of true if validation passes.
validation > fields	Object List	An array of fields that fail validation.
fields >fieldName	Text	Name of the input field that fails validation. Ensure that the name of the field matches the input field.
fields > dtsp	Integer	Optional. The value for downgrading trust score of the field that fails validation.
fields > fieldName > sys	Reference	Optional. A system field that references the groupId field. If the input field is a field group, include this parameter to send the ID of the field group entry.
sys > groupId	Text	Optional. A system-generated unique identifier of a nested field group entry. Based on the groupId value, the field group entries are enriched. If the groupId value isn't available, a new field group entry is created.

The following sample JSON snippet represents an output payload that a process sends to MDM SaaS as a result of data enrichment:

The following sample output payload shows a response for a successful validation:

[
{
"docId": "business_entity-master_id1",
"validation": [
{
"passed": true
}
]
}
]

Creating a process for validations

To perform Application Integration-based validations, create a process and define the input fields and response of the process.

As a best practice, define input and output fields based on the data model of the business entity.

1In Application Integration, create a folder within a project.

2Create a process object named System with the text field named groupId.

The following image shows a sample process object named System with the field named groupId:

3To define the payload for the request that MDM SaaS sends, create a process object. For example, create a process object named InputPayload.

- Add the data provider fields that you want to display as input fields in Business 360 Console.
- Add a field named sys, and reference the System process object that you created in step 2.

The following image shows a sample process object named InputPayload with four fields named FieldToShowValidationMessage, phone, email, and sys:

4Create a process objects to define fields for validation results. For example, create a process object named fields.

aTo return the name of the field that fail validation, add a text field named fieldName.
bTo return a value to downgrade trust score, add a text field named dtsp.
cTo return the field group ID if the input field is a field group, add a field named sys, and reference the System process object that you created in step 2.

The following image shows a sample process object named fields with three fields named fieldName, dtsp, and sys:

5To define payload for the response to MDM SaaS, create a process object. For example, create a process object named ValidationPayload.

- Add a text field named passed. Use this field to return true or false based on the validation results.
- Add a field named fields, set its data type to Object List, and then reference the process object that you created in step 4.

The following image shows a sample process object named ValidationPayload with two fields named passed and fields:

6To define the request structure, create a process object. For example, create a process object named InputPerRecord.

- Add a field named payload, set its type to ObjectList, and select the process object that you created in step 3.
- Add a text field named docId.

The following image shows a process object named InputPerRecord with two fields named payload and docId:

7To define the response structure, create a process object. For example, create a process object named OutputPerRecord.

- Add a text field named docId.
- Add a field named validation, set its type to ObjectList, and select the process object that you created in step 5.

The following image shows a process object named OutputPerRecord with two fields named docId and validation:

8Create a process.

9In the Start Properties panel of the process, perform the following steps:

aClick the Start tab, and select Allow anonymous access.

Note: Ensure that the Run On property is set to Cloud Server.

bClick the Input Fields tab.
cSelect Whole Payload, add an input field named request, and set its type to List of InputPerRecord by selecting the InputPerRecord process object.

The following image shows the input field named request and its type as List of InputPerRecord:

dClick the Output Fields tab.
eSelect Whole Payload, add an output field named response, and set its type to List of OutputPerRecord by selecting the OutputPerRecord process object.

The following image shows the output field named response and its type as List of OutputPerRecord:

10Based on your enrichment requirements, add necessary process steps to the process.

- Add a service connection.
- Add steps to receive input fields from MDM SaaS and send validation results to MDM SaaS.

Payload formats for Application Integration-based cleansing

When you create a process to use with the Application Integration-based Cleansing plugin, ensure that the payload formats of the process are compatible with the plugin.

A process created to perform cleansing validates or uses an external data provider to validate the input fields. If the input fields fail validation, the process or external enrichment and validation service cleanses the data and sends the cleansed data back to MDM SaaS. When you send a response back to MDM SaaS, you can include a value for downgrading trust score. If you don't include validation results in the response, MDM SaaS considers the validation to be successful.

Input payload format

The following table lists the parameters that the input payload of an Application Integration process for data enrichment:

Field	Data Type	Description
docId	Text	A system-generated unique identifier of the record for which data enrichment is triggered.
payload	Object List	List of fields that you want to display as input fields in rule associations. It also includes the sys field.
payload > sys	Reference	A system field that references the groupId field. Note: Include this field at the root level of the payload field.
payload > (field group) > sys	Reference	A system field that references the groupId field. If you add a field group in the payload, include the sys field in each process object for the field groups.
sys > groupId	Text	A system-generated unique identifier of a field group entry.

The following sample JSON snippet represents an input payload that MDM SaaS sends to a process for validation:

Output payload format

The response from a process to MDM SaaS must contain an array of records.

The following table lists the parameters that you need to include in the output payload of the process:

Parameter	Data Type	Description
docId	Text	A system-generated unique identifier of a record for which data enrichment is triggered.
validation	Object List	An array of validation results.
validation > passed	Boolean	A value of false if validation fails. A value of true if validation passes.
validation > fields	Object List	An array of fields that fail validation.
fields >fieldName	Text	Name of the input field that fails validation. Ensure that the name of the field matches the input field.
fields > dtsp	Integer	Optional. The value for downgrading trust score of the field that fails validation.
fields > fieldName > sys	Reference	Optional. A system field that references the groupId field. If the input field is a field group, include this parameter to send the ID of the field group entry.
payload	Object List	A list of fields that you receive from a data provider.
payload > sys	Reference	Optional. A system field that references the groupId field.
payload > (field group) > sys	Reference	Optional. A system field that references the groupId field. If an output field is a field group, include this parameter to return the ID of the field group entry that needs to be updated.
sys > groupId	Text	Optional. A system-generated unique identifier of a nested field group entry. Based on the groupId value, the field group entries are enriched. If the groupId value isn't available, a new field group entry is created.

The following output field adds a new entry of the regulatoryInformation field group and updates the drugCode field with the cleansed value:

[
{
"docId": "business_entity-master_id1",
"validation": [
{
"passed": false,
"fields": [
{
"fieldName": "drugCode",
"dtsp": 23
},
{
"fieldName": "regulatoryInformation",
"dtsp": 23
}
]
}
],
"payload": [
{
"drugCode": "12345-6789-01", // Trimmed spaces, standardized hyphen placement
"regulatoryInformation": [
{
"approvalStatus": "Approved", // Corrected spelling and capitalized
"regulatoryAuthority": "FDA", // Uppercase and no trailing spaces
"patentExpiry": "2038-08-14" // Standardized date separator
}
]
}
]
}
]

The following sample output payload updates the existing regulatoryInformation field group entry and updates the drugCode field with the cleansed data:

[
{
"docId": "business_entity-master_id1",
"validation": [
{
"passed": false,
"fields": [
{
"fieldName": "drugCode",
"dtsp": 23
},
{
"fieldName": "regulatoryInformation",
"dtsp": 23,
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e"
}
}
]
}
],
"payload": [
{
"drugCode": "12345-6789-01", // Trimmed spaces, standardized hyphen placement
"regulatoryInformation": [
{
"approvalStatus": "Approved", // Corrected spelling and capitalized
"regulatoryAuthority": "FDA", // Uppercase and no trailing spaces
"patentExpiry": "2038-08-14", // Standardized date separator
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e" // Complete and valid UUID
}
}
],
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e" // Complete and valid UUID (root level)
}
}
]
}
]

The following sample output payload validates and updates a field in the regulatoryInformation field group:

[
{
"docId": "business_entity-master_id1",
"validation": [
{
"passed": false,
"fields": [
{
"fieldName": "drugCode",
"dtsp": 23
},
{
"fieldName": "regulatoryInformation.approvalStatus",
"dtsp": 23,
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e"
}
}
]
}
],
"payload": [
{
"drugCode": "12345-6789-01", // Trimmed spaces, standardized hyphen placement
"regulatoryInformation": [
{
"approvalStatus": "Approved" // Corrected spelling and capitalized
],
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e" // Complete and valid UUID (root level)
}
}
]
}
]

The following sample output payload shows a response for a successful validation:

[
{
"docId": "business_entity-master_id1",
"validation": [
{
"passed": true
}
],
"payload": [
{
"drugCode": "12345-6789-01",
"regulatoryInformation": [
{
"approvalStatus": "Approved",
"regulatoryAuthority": "FDA",
"patentExpiry": "2038-08-14",
"sys": {
"groupId": "f88114db-461c-34c3-87b6-d2cebc76e85e"
}
}
]
}
]
}
]

Guidelines for creating Application Integration processes

Consider the following guidelines when you create an Application Integration process:

•Ensure that the process is published.
•Ensure that the process has anonymous or public access.
•Create process with unique API names. If you copy an existing process to create a new process, ensure that you update the process with a unique API name.
•When you define input and output fields for a process, model them after the data model of the business entity instead of the data provider. This approach helps minimize issues related to field mappings in Business 360 Console. You can use assignment steps in the process to specify the desired request and response formats of the data provider.

For example, consider the following sample request format of a data provider:

{
"firstName": "string",
"lastName": "string",
"contacts": {
"address": {
"addressLineOne": "string",
"addressLineTwo": "string",
"city": "string",
"state": "string",
"country": "string"
},
"email": {
"usageType": "string",
"emailAddress": "string"
},
"phone": {
"usageType": "string",
"phoneNumber": "string"
}
}
}

Also, consider the following data model of the Person business entity:

{
"firstName": "string",
"lastName": "string",
"address": {
"addressLineOne": "string",
"addressLineTwo": "string",
"city": "string",
"state": "string",
"country": "string"
},
"email": {
"usageType": "string",
"emailAddress": "string"
},
"phone": {
"usageType": "string",
"phoneNumber": "string"
}
}

The sample data model of the business entity doesn't include a field group, such as contacts with nested field groups. In this case, model the input payload of the process after the business entity.

Consider the following sample input payload format of a process that replicates the data model of the business entity instead of the request format of the data provider:

[
{
"docId": "string",
"payload": [
{
"firstName": "string",
"lastName": "string",
"address": {
"addressLineOne": "string",
"addressLineTwo": "string",
"city": "string",
"state": "string",
"country": "string",
"sys": {
"groupId": "string"
}
},
"email": {
"usageType": "string",
"emailAddress": "string",
"sys": {
"groupId": "string"
}
},
"phone": {
"usageType": "string",
"phoneNumber": "string",
"sys": {
"groupId": "string"
}
},
"sys": {
"groupId": "string"
}
}
]
}
]

•Ensure that the input payload includes the required fields. For more information about the required fields, see Input payload format.
•Ensure that you include the sys > groupId fields at the root level of the input payload. If the payload includes field groups, include the sys > groupId fields for each field group.
•Ensure that the output payload includes the required fields. For more information about the required fields, see Output payload format.

Note: The sys > groupId fields are optional in the output payload. Include the sys > groupId fields in the process object to enrich the field group entries. If the groupId value isn't available, a new field group entry is created.

•If you want to enrich the existing entries of a field group, ensure that you include the field group as one of the input fields. For example, to enrich the existing Phone field group entries, include the Phone field group as an input field. When you configure a rule association in Business 360 Console, you can map the Phone field group of the business entity to the input field.
•To display the data from the data provider as CLAIRE recommendations in the business applications, ensure that you include confidence scores for each recommendation when you send a response to MDM SaaS.
•In the start properties of the process in Application Integration, ensure that you enable Whole Payload as the input and output formats.
•When you configure a process, ensure that you set the Input Fields and Output Fields properties in the start properties of the process to a list of objects. If you add or delete any fields from any process objects, configure the Input Fields and Output Fields properties again in the start properties.
•To use business IDs in a process that helps assign records to a hierarchy, ensure that you enable the searchable property of the Business ID system field for the business entity.
•Ensure that the process is configured to run on cloud server. You can't use a process that's configured to run on a Secure Agent.
•If the response to a validation request includes the passed field set to false in the validation object, ensure that you include the fields array in the validation object of the response.

Data types applicable to enrichment, validation and cleansing

When you create an Application Integration process for data enrichment, you can specify the following data types for the fields in the process:

•Integer
•Number
•Date
•Date Time
•Picklist
•Checkbox
•Text

Don't use the following data types for any fields in a process:

•Currency
•Percent
•Phone
•Email
•Time
•Text Area
•Rich Text Area
•Image
•Multi-select Picklist
•Attachment
•Attachments

Compatible data types of process fields and business entity fields

When you add input fields in an Application Integration process, select data types that are compatible with the data types of business entity fields. This approach helps you map compatible fields when you configure rule associations.

The following table lists the compatible data types of business entity fields:

Data types of business entity fields	Data types of Application Integration process fields
Double	Number, Text
Integer	Integer, Number, Text
Date	Date, Text
Date and Time	Date Time, Text
Picklist	Picklist, Text
Text	Text, Integer, Number, Date, Date Time, Picklist
Boolean	Text, Checkbox
Decimal	Text

When you add output fields in an Application Integration process, select data types that are compatible with the data types of the business entity fields. This approach helps you map compatible fields when you configure rule associations.

The following table lists the compatible data types of a process:

Data types of Application Integration process fields	Data types of business entity fields
Integer	Integer, Double, Decimal, Text
Number	Double, Text
Text	Integer, Double, Decimal, Date, Date and Time, Boolean, and Text, Picklist
Picklist	Picklist, Text
Date	Date, Text
Date Time	Date and Time, Text
Checkbox	Text, Boolean

Note: The decimal data type isn’t available in Application Integration. Use a text field in the process for decimal values. When you map input and output fields in a rule association, you can map a decimal business entity field to a text field. For more information about the guidelines for field mappings, see Guidelines for field mappings.