Developer Transformation Guide > Parser Transformation > Token Parsing Properties
  

Token Parsing Properties

Configure properties for token parsing operations on the Strategies view in the Parser transformation.

General Properties

General properties apply to all token parsing operations you define in the strategy. You use the general properties to name the strategy, specify input and output ports, and specify whether the strategy enables probabilistic matching techniques.
The following table describes the general properties.
Property
Description
Name
Provides a name for the strategy.
Inputs
Identifies the input ports that the strategy operations can read.
Outputs
Identifies the output ports that the strategy operations can write to.
Description
Describes the strategy. The property is optional.
Use probabilistic matching techniques
Specifies that the strategy can use a probabilistic model to identify tokens.
Input Join Character
Specifies the character used to join input data ports. The transformation merges all input ports into a combined data string and parses the complete string.
Output Join Character
Specifies the character used to join output data values when multiple parsing operations use the same output.
Reverse Enabled
Configures the strategy to parse data from right to left. This property is disabled for probabilistic matching.
Overflow Reverse Enabled
Configures the strategy to parse the overflow data from right to left. This property is disabled for probabilistic matching.
Detailed Overflow Enabled
Creates a unique overflow field for each parsing operation.
Delimiters
Specifies the delimiter that separates the input data into separate tokens. Default is space.

Probabilistic Model Properties

You can select a probabilistic model in place of a token set when you configure a token parsing strategy. Select the Parse using Token Set operation and select the option to use probabilistic matching techniques.
The following table describes the probabilistic model properties:
Property
Description
Name
Provides a name for the operation.
Filter Text
Uses characters or wildcards you enter to filter the list of token sets, probabilistic models, or regular expressions.
Probabilistic Model
Identifies the probabilistic model you select.

Reference Table Properties

Reference table properties apply when you configure a labeling operation to use a reference table.
The following table describes the reference table properties:
Property
Description
Name
Provides a name for the operation.
Reference Table
Specifies the reference table that the operation uses to parse input values.
Case Sensitive
Determines whether input strings must match the case of reference table entries.
Replace Matches with Valid Values
Replaces parsed data with the data from the Valid column in the reference table.
Outputs
Specifies the output ports for the parsed data.

Token Set Properties

Token set properties apply when you configure a parsing operation to use token sets.
Select the Parse using Token Set operation to parse input with token sets. Clear the option to use probabilistic matching techniques.
The following tables describe the token set properties:
Property
Description
Name
Provides a name for the operation.
Token Sets (Single Output Only)
Specifies the token set that the operation uses to parse data. The operation writes the data to a single port.
Regular Expression (Single or Multiple Outputs)
Specifies the regular expression that the operation uses to parse data. The operation writes the data to multiple ports if it finds multiple strings in the input field.
Outputs
Identifies the output ports that the operations writes to.
You can add, edit, import, or remove a token set or a regular expression. You can also filter the list of the token sets.
The following table describes the properties that you use to perform the tasks:
Property
Description
Filter text
Filters the list of token sets or regular expressions. Use text characters and wildcard characters as a filter.
Add
Use to define a custom token set or regular expression.
Edit
Edits the contents of a custom token set.
Import
Imports a nonreusable copy of a token set or regular expression from a folder in the Model repository. If you update the source object for the token set or regular expression, the Data Integration Service does not update the nonreusable copy.
Remove
Deletes a custom token set or regular expression.