Before you configure a midstream transformation, you must create a business service from the New > Components tab. You must select a REST V2 connection and an operation when you create a business service.
When you configure the midstream transformation, select the business service and the operation on the Web Service tab.
You can map the input fields from source file to elements in request structure on the Request Mapping tab.
You can view the response structure in the field mapping. When you map the elements from the response structure to the output fields, the Secure Agent creates the output groups, along with the primary and foreign keys for the field names.
Advanced midstream properties
The following table describes the advanced properties that you can configure for a midstream transformation:
Property
Description
Paging Type
Specify one of the following values:
Page. Enables paging and considers the values of Page Parameter, Start Page, End Page, and End of Response Expression properties.
None. Ignores the values of Page Parameter, Start Page, End Page, and End of Response Expression properties.
Page Parameter
The name of the parameter that you want to use for the paging operation. You can use a query parameter or a path parameter.
The parameter must be of the integer type and from the request message.
Start Page
The page number that indicates the first page in the range, on which you want to perform the paging operation.
End Page
The page number that indicates the last page in the range, on which you want to perform the paging operation. The default is 10000.
Paging stops when the End Page is reached or the End of Response Expression is met.
Page Increment Factor
An integer to increment the Page Parameter. Page Increment Factor must be same as the number of records being fetched per request. If the Page Increment Parameter option is not same as the number of records being fetched per request, you might have missing or duplicate records between two calls.
End of Response Expression
Specify an expression or a string to control paging. You can observe one of the following behaviors:
- If you specify a string or an expression, the paging stops when the value matches with the page response. It does not parse the page that has matching end of response.
- If you do not specify a string or an expression, the paging stops on reaching a page that has an empty or a Null response. If an empty or a Null response is never reached, the paging stops at the default end page.
- If you specify both, End Page and End of Response Expression, the paging stops on whichever condition is met first.
- If you do not specify End Page and End of Response Expression, the paging stops on reaching an empty or a Null response. If an empty or a Null response is never reached, the paging stops at the default end page.
Override URL
Overrides the URL specified in the swagger specification. The override URL cannot have query parameters. When a path parameter is included in the Override URL, enclose the path parameter with curly brackets {}. For example:
URL specified in the swagger specification: http://invr28pers102:13080/sample/day/20170505?a:b
If you define 20170505 as a path variable with the path variable name as path1, the Override URL will be as follows: http://invr28pers102:13080/sample/day/{path1}
Cache Size for Web Service Request (KB)
Default is 100 KB. If the request is more than 100 KB, you can increase the cache size.
Cache Size for Web Service Response (KB)
Memory available for the web service response. If the web service response contains many rows or columns, you might want to increase the cache size. Default is 100 KB.
Allow Input Flush
Not Applicable.
Transaction Commit Control
Not Applicable.
Midstream transformation mapping example
You are a human resources administrator and you want to retrieve details of an employee from Apache CouchDB.
To retrieve employee details from CouchDB and to write the employee details to a flat file, perform the following tasks:
1Create a REST V2 connection to read data from CouchDB.
The following image shows the configured CouchDB connection:
2Create a business service to associate the REST V2 connection and an operation.
The following image shows the configured business service:
3Create a REST V2 mapping.
4Add a Web Services transformation. Specify a name and description in the general properties.
5On the Web Service tab, select the business service and the operation that you configured. Also, specify paging parameters.
The following image shows the configured web service:
6On the Request Mapping tab, map the incoming fields from the source to the respective fields in CouchDB.
7On the Response Mapping tab, select the employee details fields that you want to write to the target file on the Response Structure.
8On the Advanced tab, specify the cache size details.
The fields that you selected from the Response Structure of the Web Service transformation appear as incoming fields for the target object.
9 If required, map the incoming fields to the flat file fields.
10Click New > Tasks > Mapping Task, and select the mapping for the task.
11When you save and run the mapping task, the Secure Agent retrieves the employee record for the employee that you map on the Request Mapping tab from CouchDB and writes the data to the corresponding flat files.
