Configure Data Quality > Configuring data enrichment > Creating Application Integration processes for data enrichment
  

Creating Application Integration processes for data enrichment

Before you configure data enrichment, you need to create a process that connects to an external data provider. The process accepts a set of fields as input from MDM SaaS. The process then sends a request to the data provider. When the data provider returns data for enrichment, the process sends a response back to MDM SaaS. The response includes a set of output fields that contain data from the data provider.
When you create a process, you need to define the input fields that MDM SaaS sends to the process. You also need to define the output fields that the process sends to MDM SaaS.
To define the input and output fields, you need to create process objects that you can use within your process. Process objects group and structure data for use in processes. For example, instead of creating separate items for each type of demographic data, you can have a process object that contains the name, address, and phone number fields.
When you create process objects for input and output fields, ensure that the input and output formats of the process are compatible with the required MDM SaaS request and response formats.

Request format

Currently, you can process one record in each request.
The following table lists the parameters that an Application Integration process requires from an MDM SaaS enrichment request:
Parameter
Description
docId
A system-generated unique identifier of the record for which data enrichment is triggered.
payload
List of fields that you need to include in the request to an external data provider.
payload > sys > groupId
A system-generated unique identifier of a field group entry that's mapped at the root level.
payload > (field group) > sys > groupId
A system-generated unique identifier of a nested field group entry.
To create process objects in Application Integration, you might use the following sample request from MDM SaaS to a process:
[
{
"docId": "c360.item-xyz456",
"payload": [
{
"address": {
"city": "clarence town",
"addressLine1": "23 Gaggin St",
"sys": {
"groupId": "acd4985"
}
},
"sys": {
"groupId": "acd4986"
}
}
]
}
]

Response format

The response from a process to MDM SaaS must contain an array of records.
Currently, you can process one record in each request.
The following table lists the parameters that you need to include in the response from the process to MDM SaaS
Parameter
Description
docId
A system-generated unique identifier of a record for which data enrichment is triggered.
payload
A list of fields that you receive from a data provider.
payload > sys > groupId
Optional. A system-generated unique identifier of a field group entry that is mapped at the root level.
Based on the groupId value, the field group entries are enriched.
If the groupId value isn't available, a new field group entry is created.
payload > (field group) > sys > groupId
Optional. A system-generated unique identifier of a nested field group entry.
Based on the groupId value, the field group entries are enriched.
If the groupId value isn't available, a new field group entry is created.
To create process objects in Application Integration, you might use the following sample response from a process to MDM SaaS:
[
{
"docId": "c360.item-xyz456",
"payload": [
{
"cleansedAddress": [
{
"address1": "23rd Gaggin Street",
"city": "clarence town",
"country": "Australia",
"PinCode": 5432,
"sys": {
"groupId": "acd4985"
}
}
],
"matchConfidence": 57
}
]
}
]

Creating a process for data enrichment

To define a set of fields that you want to display as input and output fields in Business 360 Console, create a process and define process objects that contain the fields.
As a best practice, define input and output fields based on the data model of the business entity.
    1In Application Integration, create a folder within a project.
    2Create a process object named System with the text field named groupId.
    The following image shows a sample process object named System with the field named groupId:
    3To define the payload for the request that MDM SaaS sends, create a process object named InputPayload.
    The following image shows a sample process object named InputPayload with two fields named Addresses and sys:
    4To define payload for the response to MDM SaaS, create a process object named OutputPayload.
    The following image shows a sample process object named OutputPayload with two fields named confidenceScoreInPercent and Addresses:
    5To define the request structure, create a process object named InputPerRecord.
    The following image shows a process object named InputPerRecord with two fields named payload and docId:
    6To define the response structure, create a process object named OutputPerRecord.
    The following image shows a process object named OutputPerRecord with two fields named docId and payload:
    7Create a process.
    8In the Start Properties panel of the process, perform the following steps:
    1. aClick the Start tab, and select Allow anonymous access.
    2. bClick the Input Fields tab.
    3. cSelect Whole Payload, add an input field named request, and set its type to List of InputPerRecord by selecting the InputPerRecord process object.
    4. The following image shows the input field named request and its type as List of InputPerRecord:
    5. dClick the Output Fields tab.
    6. eSelect Whole Payload, add an output field named response, and set its type to List of OutputPerRecord by selecting the OutputPerRecord process object.
    7. The following image shows the output field named response and its type as List of OutputPerRecord:
    9Based on your enrichment requirements, add necessary process steps to the process.

Guidelines for creating Application Integration processes

Consider the following guidelines when you create an Application Integration process:

Creating an Application Integration process scenario

You work for Acme, and your goal is to enrich the Person records with correct addresses. You use AddressMaster as the data provider for this enrichment.
Consider the following sample request that AddressMaster requires you to send:
{
"addressline1":"GP",
"postalcode":"",
}
Also, consider the following sample response that you receive from AddressMaster:
{
"Items": [
{
"Id": "US|LP|A|100001|_ENG",
"Type": "Address",
"line1": "Sunset Villa;",
"line2": "22 Oakwood Lane",
"city": "Maplewood",
"state": "New Jersey",
"country": "USA",
"PostCode": "07040"
},
{
"Id": "US|LP|A|100002|_ENG",
"Type": "Address",
"line1": "Elm Street Apartment;",
"line2": "1010 Elm Street",
"city": "Springfield",
"state": "Illinois",
"country": "USA",
"PostCode": "62704"
}
]
}
To create a process for enriching records with data from AddressMaster, perform the following steps:
  1. 1Create process objects to define input and output formats.
  2. 2Create a process.

Step 1: Create process objects for input and output fields

You want to define input and output fields for a process to enrich records with the data from AddressMaster. Define process objects in Application Integration for this scenario.
    1In Application Integration, create a folder within your project.
    2Create a process object named System with a text field named groupId.
    The following image shows a process object named System with a field named groupId:
    3To define a set of fields that you want to accept as input fields from MDM SaaS, create a process object named InputAddress with the following fields:
    Field
    Label
    Type
    addressline1
    addressline1
    Text
    postalcode
    postalcode
    Text
    sys
    sys
    Reference
    Note: Reference the System process object that you created in step 2.
    Note: Even though the request to AddressMaster requires a flat structure, you need a field group in this scenario because addressline1 and postalcode are child attributes of the Address field group in the Person business entity.
    The following image shows a process object named InputAddress with the fields named addressline1, postalcode, and sys:
    4To define a set of fields that you want to send back to MDM SaaS, create a process object named OutputAddress with the following fields:
    Field
    Label
    Type
    addressline1
    addressline1
    Text
    addressline2
    addressline2
    Text
    city
    city
    Text
    state
    state
    Text
    postalcode
    postalcode
    Text
    country
    country
    country
    sys
    sys
    Reference
    Note: Reference the System process object that you created in step 2. If you include this parameter, the original field group entry is updated. Otherwise, a new field group entry is added to the record when a user accepts a CLAIRE recommendation.
    The following image shows a process object named OutputAddress with the fields named addressline1, addressline2, city, state, postalcode, country, and sys:
    5To define the payload for the request from MDM SaaS, create a process object named InputPayload with the following fields:
    Field
    Label
    Type
    Addresses
    Addresses
    ObjectList
    Note: Select the InputAddress process object that you created in step 3.
    sys
    sys
    Reference
    Note: Reference the System process object that you created in step 2.
    The following image shows a process object named InputPayload with the fields named Addresses and sys:
    6To define payload for the response to MDM SaaS, create a process object named OutputPayload with the following fields:
    Field
    Label
    Type
    Addresses
    Addresses
    ObjectList
    Note: Select the OutputAddress process object that you created in step 4.
    confidenceScoreInPercent
    confidenceScoreInPercent
    Text
    The following image shows a process object named OutputPayload with the fields named confidenceScoreInPercent and Addresses:
    7To define the request structure, create a process object named InputPerRecord with the following fields:
    Field
    Label
    Type
    payload
    payload
    ObjectList
    Note: Select the InputPayload process object that you created in step 5.
    docId
    docId
    Text
    Note: docId is a required parameter in the response.
    The following image shows a process object named InputPerRecord with the fields named payload and docId:
    8To define the response structure, create a process object named OutputPerRecord with the following fields:
    Field
    Label
    Type
    docId
    docId
    Text
    payload
    payload
    ObjectList
    Note: Select the OutputPayload process object that you created in step 6.
    The following image shows a process object named OutputPerRecord with the fields named docId and payload:

Step 2: Create a process

To enrich records with data from the AddressMaster data provider, you need a process that receives a set of input fields from MDM SaaS and sends a response back to MDM SaaS.
    1In Application Integration, create a process.
    2In the Start Properties panel of the process, perform the following steps:
    1. aClick the Start tab, and select Allow anonymous access.
    2. bClick the Input Fields tab.
    3. cSelect Whole Payload, add an input field named request, and set its type to List of InputPerRecord by selecting the InputPerRecord process object.
    4. The following image shows the input field named request and its type as List of InputPerRecord:
    5. dClick the Output Fields tab.
    6. eSelect Whole Payload, and add an output field named response and set its type to List of OutputPerRecord by selecting the OutputPerRecord process object.
    7. The following image shows the output field named response and its type as List of OutputPerRecord:
    3Create a temp field named temp_recordId, and set its type to Text.
    4Add an assignment step to the process.
    5In the Properties panel of the assignment step, click the Assignment tab, and set the temp_recordId field to the $input.request[1]/docId formula.
    Note: Additionally, you can create a process object that represents the request to the data provider. You can then assign the process input fields to the AddressMaster request fields. This sample process doesn't connect to a real data provider.
    The following image shows that the $input.request[1]/docId formula is assigned to the temp_recordId field:
    6Add an assignment step to the process.
    7In the Properties panel of the assignment step, click the Assignment tab and perform the following assignment tasks:
    1. aAdd a field named response.
    2. bIn the Assigned Using field, select Formula.
    3. cAdd the response that you want to send to MDM SaaS, such as the following sample response:
    4. let $items :=
      <Items>
      <Item>
      <Id>IN|LP|A|19635268|_ENG</Id>
      <Type>Address</Type>
      <Line1>Prakash 1;</Line1>
      <Line2>Prakash Building 1 BG Kher Road</Line2>
      <City></City>
      <State></State>
      <Country>United Kingdon</Country>
      <PostCode></PostCode>
      <Highlight>8-9,27-28,29-31;</Highlight>
      <Description>Mumbai 400006</Description>
      </Item>
      <Item>
      <Id>IN|LP|A|19637698|_ENG</Id>
      <Type>Address</Type>
      <Line1>Building 1;</Line1>
      <Line2>BG Kher Road</Line2>
      <City></City>
      <State></State>
      <Country>United Kingdon</Country>
      <PostCode></PostCode>
      <Highlight>8-9,27-28,29-31;</Highlight>
      <Description>Mumbai 400006</Description>
      </Item>
      </Items>

      return (
      <OutputPerRecord>
      <docId> {$temp.temp_recordId }</docId>
      {
      for $item in $items/Item
      return (
      <payload>
      <confidenceScoreInPercent>70</confidenceScoreInPercent>
      <Addresses>
      <id>{$item/Id/text() }</id>
      <addressline1>{$item/Line1/text()}</addressline1>
      <addressline2>{$item/Line2/text()}</addressline2>
      <city>{$item/City/text()}</city>
      <state>{$item/State/text()}</state>
      <postalcode>{$item/PostCode/text()}</postalcode>
      <country>{$item/Country/text()}</country>
      <sys>
      <groupId> { $input.request[1]/payload[1]/Addresses[1]/sys[1]/groupId }</groupId>
      </sys>
      </Addresses>
      </payload>
      )
      }
      </OutputPerRecord>
      )
    $items represents a sample response from AddressMaster. In the return block, you can loop through the fields in the AddressMaster response and assign the required fields to the fields in the OutputPayload process object.
    The following image shows that the response variable is assigned to a formula: