Transformations
Active and passive transformations
Transformation types
Transformations available in mappings
Transformations available in advanced mode
Transformations available in SQL ELT mode
Licensed transformations
Incoming fields
Field name conflicts
Field rules
Data object preview
Variable fields
Transformation caches
Cache types
Cache files
Cache size
Optimizing the cache size
Expression macros
Macro types
Macro input fields
Vertical macros
Horizontal macros
Hybrid macros
File lists
Manually created file lists
File list commands
Using a file list in a Source transformation
Using a file list in a Lookup transformation
Configuration for multibyte hierarchical data
Source transformation
Source object
File sources
Database sources
Database source properties
Related objects
Advanced relationships
Custom queries
Source filtering and sorting
Web service sources
Web service operations for sources
Request messages
Field mapping for web service sources
Partitions
Partitioning rules and guidelines
Partitioning examples
Reading hierarchical data in advanced mode
Reading documents in advanced mode
Configuration for multibyte hierarchical data
Source fields
Editing native data types in complex file sources
Editing transformation data types
Target transformation
Target object
Target file creation on advanced clusters
File targets
File target properties
Flat file targets created at run time
Flat file targets with static file names
Flat file targets with dynamic file names
Creating a flat file target at run time
Database targets
Database target properties
Vector database target properties
Database targets created at run time
Update columns for relational targets
Target update override
Web service targets
Web service operations for targets
Field mapping for web service targets
Partitions
Writing hierarchical data in advanced mode
Configuration for multibyte hierarchical data
Target fields
Target transformation field mappings
Configuring a Target transformation
Access Policy transformation
Using parameters in Access Policy transformations
Data filter policy best practices
Access Policy transformation configuration
Access Policy transformation example
Aggregator transformation
Group by fields
Sorted data
Aggregate fields
Aggregate functions
Nested aggregate functions
Conditional clauses
Advanced properties
Hierarchical data in advanced mode
Aggregator transformation example
B2B transformation
B2B Incoming Fields
B2B settings
Output fields
Field mapping
Advanced settings
Chunking transformation
Chunking methods
Chunking output fields
Cleanse transformation
Cleanse transformation configuration
Cleanse asset considerations
Synchronizing data quality assets
Cleanse transformation field mappings
Cleanse transformation output fields
Advanced properties
Data Masking transformation
Masking techniques
Configuration properties for masking techniques
Repeatable output
Optimize dictionary usage
Seed
Unique substitution
Mask format
Source filter characters
Target filter characters
Range
Blurring
Connections in a Data Masking transformation
Credit card masking
Email masking
Advanced email masking
IP address masking
Key masking
Phone number masking
Random masking
Social Insurance number masking
Social Security number masking
Custom substitution masking
Dependent masking
Dependent masking parameters
Substitution masking
URL address masking
Mask rule parameter
Mask rule parameter example
Create a mapping with parameters
Run the mapping
Creating a Data Masking transformation
Consistent masked output
Rules and guidelines
Example
Data Masking transformation example
Data Services transformation
Dynamic service name
Status tracing messages
Data Services properties
Data Services transformation input fields
Data Services transformation output fields
Data Services transformation field mapping
Deduplicate transformation
Deduplication and consolidation operations
Identity population data
Groups in duplicate analysis
Example: Selecting a group key column
Deduplicate transformation configuration
Deduplicate transformation field mappings
Metadata fields on the Deduplicate transformation
Link scores and driver scores
Deduplicate transformation output fields
Advanced properties
Expression transformation
Expression fields
Output fields
Variable fields
Input macro field
Output macro field
Expression editor
Transformation language components for expressions
Expression syntax
String and numeric literals
Adding comments to expressions
Reserved words
Window functions
Frame
Partition and order keys
Example: Use a window to calculate expiration dates
Example: Use a window to flag GPS pings
Example: Run an aggregate function on a window
Advanced properties
Hierarchical data in advanced mode
Filter transformation
Filter conditions
Advanced properties
Hierarchical data in advanced mode
Hierarchy Builder transformation
Configure output settings
Rules and guidelines for hierarchical schemas
Single output example
Rules and guidelines for optimizing the transformation
Join and map fields for data conversion
Joining incoming data
Mapping relational fields to hierarchy fields
Configure advanced properties
Configuration for multibyte hierarchical data
Hierarchy Builder transformation example
Hierarchy Parser transformation
Using a Hierarchy Parser transformation
Hierarchy Parser rules and guidelines
Choosing a sample or schema file
Hierarchical schemas
Rules and guidelines for hierarchical schemas
Creating a hierarchical schema
Input settings
Selecting a hierarchical schema
Creating a hierarchical schema from sample
Input field selection
Field mapping
Selecting the elements to convert
Output fields
Selecting an output group
Configuration for multibyte hierarchical data
Hierarchy Parser transformation example
Hierarchy Processor transformation
Hierarchy Processor transformation overview
Hierarchical to relational data processing
Relational to hierarchical data processing
Hierarchical to hierarchical data processing
Hierarchical to flattened data processing
Configuration for multibyte hierarchical data
Field restrictions
Processing relational output
Defining relational output with the Hierarchy Processor transformation
Adding incoming fields to relational output groups
Configuring relational output groups and fields
Configuring data sources
Configuring filter conditions
Expression configuration
Running a mapping with JSON data
Hierarchical to relational example
Processing hierarchical output
Defining hierarchical output with the Hierarchy Processor transformation
Adding incoming fields to hierarchical output groups
Configuring hierarchical output group and fields
Configuring data sources
Configuring output data
Expression configuration
Running a mapping with JSON data
Relational to hierarchical example
Hierarchical to hierarchical example
Processing flattened output
Defining flattened output with the Hierarchy Processor transformation
Adding incoming fields to flattened output groups
Renaming flattened output group and fields
Expression format
Running a mapping with JSON data
Hierarchical to flattened example
Input transformation
Input fields
Java transformation
Defining a Java transformation
Classpath configuration
JAR or class files on an advanced cluster
Classpath values
Configuring the JVM Classpath for the Secure Agent
Configuring the CLASSPATH environment variable
Configuring the design time classpath
Configuring the Java Classpath session property
Java transformation fields
Data type conversion
Sort conditions
Group by fields
Configuring Java transformation properties
Active and passive Java transformations
Defining the update strategy
Using high precision
Processing subseconds
Developing the Java code
Creating Java code snippets
Importing Java packages
Defining helper code
Defining input row behavior
Defining end of data behavior
Defining transaction notification behavior
Using Java code to parse a flat file
Compiling the code
Viewing the full class code
Troubleshooting a Java transformation
Finding the source of compilation errors
Identifying the error type
Java transformation example
Create the source file
Configure the mapping
Configure the Java code snippets
Compile the code and run the mapping
Java transformation API reference
failSession
generateRow
getInRowType
incrementErrorCount
invokeJExpression
isNull
logError
logInfo
setNull
setOutRowType
Joiner transformation
Join condition
Join type
Advanced properties
Hierarchical data in advanced mode
Creating a Joiner transformation
Joiner transformation example
Labeler transformation
Labeler transformation configuration
Labeler transformation field mappings
Labeler transformation output fields
Lookup transformation
Lookup object
Lookup object properties
Custom queries
Lookup condition
Lookup return fields
Advanced properties
Lookup SQL overrides
Lookup SQL overrides for cached lookups
Lookup SQL override for uncached lookups
Guidelines for overriding the lookup query
Lookup source filter
Dynamic lookup cache
Static and dynamic lookup comparison
Dynamic cache updates
Inserts and updates for insert rows
Dynamic cache and lookup source synchronization
Dynamic cache and target synchronization
Field mapping
Ignore fields in comparison
Dynamic lookup query overrides
Persistent lookup cache
Rebuilding the lookup cache
Unconnected lookups
Configuring an unconnected Lookup transformation
Calling an unconnected lookup from another transformation
Connected Lookup example
Dynamic Lookup example
Unconnected Lookup example
Machine Learning transformation
Deploying the model as a REST endpoint
Accessing the machine learning model
Mapping fields to the request schema
Mapping hierarchical fields
Request mapping options
Viewing response fields
Configuring bulk requests
Bulk request options
Configuring an API proxy
Bypassing the proxy server
Configuring a Spark proxy
Configuring a custom proxy
Troubleshooting
Error handling
Machine Learning transformation example
Mapplet transformation
Mapplet transformation configuration
Selecting a mapplet
Mapplet transformation field mappings
Mapplet parameters
Mapplet transformation output fields
Mapplet transformation names
Synchronizing a mapplet
Normalizer transformation
Normalized fields
Occurs configuration
Unmatched groups of multiple-occurring fields
Generated keys
Normalizer field mapping
Normalizer field mapping options
Advanced properties
Target configuration for Normalizer transformations
Normalizer field rule for parameterized sources
Mapping example with a Normalizer and Aggregator
Output transformation
Output fields
Generating output fields based on incoming fields
Field mapping
Parse transformation
Parse transformation configuration
Parse transformation field mappings
Parse transformation output fields
Advanced properties
Python transformation
Install and configure Python
Step 1. Enable access to third-party libraries
Step 2. Install prerequisite packages
Step 3. Download the Python distribution
Step 4. Prepare a directory to install Python
Step 5. Install Python
Step 6. Set environment variables
Step 7. Verify the Python installation
Step 8. Install str2bool library
Step 9. Optionally, install third-party libraries
Step 10. Copy Python to the Secure Agent machine
Step 11. Optionally, verify third-party library installations
Python transformation fields
Data type conversion
Data types in input and output fields
Partition keys
Active and passive Python transformations
Resource files
Developing the Python code
Creating Python code snippets
Referencing a resource file
Example: Add an ID column to nonpartitioned data
Example: Use partitions to find the highest salary
Example: Operationalize a pre-trained model
Rank transformation
Ranking string values
Rank caches
Defining a Rank transformation
Rank transformation fields
Defining rank properties
Defining rank groups
Advanced properties
Hierarchical data in advanced mode
Rank transformation example
Router transformation
Working with groups
Guidelines for connecting output groups
Group filter conditions
Configuring a group filter condition
Advanced properties
Hierarchical data in advanced mode
Router transformation examples
Rule Specification transformation
Rule Specification transformation configuration
Rule Specification transformation field mappings
Rule Specification transformation output fields
Advanced properties
Sequence transformation
Sequence transformation uses
Sequence output fields
Sequence properties
Disabling incoming fields
Hierarchical data in advanced mode
Sequence transformation rules and guidelines
Sequence transformation example
Sorter transformation
Sort conditions
Sorter caches
Advanced properties
Hierarchical data in advanced mode
Sorter transformation example
SQL transformation
Stored procedure or function processing
Connected or unconnected SQL transformation for stored procedure processing
Unconnected SQL transformations
Calling an unconnected SQL transformation from an expression
Invoking a stored procedure before or after a mapping run
Stored procedure run order
Unconnected SQL transformation example
Query processing
Static SQL queries
Dynamic SQL queries
Passive mode configuration
SQL statements that you can use in queries
Rules and guidelines for query processing
SQL transformation configuration
Configuring the SQL type
SQL transformation field mapping
SQL transformation output fields
Advanced properties
Structure Parser transformation
Processing input from a Hadoop Files source
Processing input from a flat file source
Configuring the flat file source
Configuring the Structure Parser transformation to access flat files
Structure Parser field mapping
Output fields
Advanced properties
Structure Parser transformation configuration
Configuring a Structure Parser transformation
Selecting an intelligent structure model
Selecting an output group
Rules and guidelines for the Structure Parser transformation
Structure Parser transformation example
Transaction Control transformation
Transaction control condition
Using Transaction Control transformations in mappings
Sample transaction control mappings with multiple targets
Guidelines for using Transaction Control transformations in mappings
Advanced properties
Union transformation
Comparison to Joiner transformation
Planning to use a Union transformation
Input groups
Output fields
Field mappings
Advanced properties
Union Transformation example
Vector Embedding transformation
Vector embedding techniques
Vector embedding output fields
Velocity transformation
Velocity transformation input format
Source configuration for file sources
Velocity template
Testing the template
Velocity transformation output
Target configuration for file targets
Velocity transformation parsers
Examples
XML conversion example
JSON conversion example
Verifier transformation
Address Reference Data
Verifier transformation configuration
Verifier transformation field mappings
Understanding input and output mappings
Verifier transformation output fields
Advanced properties
Web Services transformation
Create a Web Services consumer connection
Define a business service
Configure the Web Services transformation
Configuring the transformation
Transaction commit control
Viewing incoming and request fields
Pass-through fields
Web Services transformation example
Configuration for multibyte hierarchical data
Transformations
Configuration for multibyte hierarchical data
CMD_ADV_PROP
DI_TRANS_MOD_HYLNK
CMD_PROP_SECTION_TX
CMD_FIELD_NAME_CONFLICT_RESOLUTION_DIALOG
CMD_PROP_SECTION_TX_FIELDS
CMD_DATATYPE_RULE_DIALOG
CMD_ALL_RULE_DIALOG
CMD_PROP_SECTION_MPLT_INCOMING_FIELDS
CMD_NAME_RULE_DIALOG
CMD_DATA_PREVIEW_DIALOG
CMD_PROP_SECTION_SRC
CMD_FORMATING_OPTS_DIALOG
CMD_CUSTOM_QUERY_DIALOG
CMD_FMAP_SECTION_SRC
CMD_PROP_SECTION_SRC_PARTITIONING
CMD_PROP_SECTION_SRC_FIELDS
CMD_ADAPTER_FIELD_DIALOG
CMD_PROP_SECTION_TGT
CMD_FORMATTING_OPTS_DIALOG_TGT
CMD_PROP_SECTION_TGT_PARTITIONING
CMD_PROP_SECTION_TGT_FIELDS
DESIGNER_CREATE_TARGET
CMD_PROP_SECTION_AGGREGATOR_GROUPBY
CMD_PROP_SECTION_AGGREGATOR
CMD_FIELD_DIALOG
CMD_ADV_PROP_AGG
CMD_CLEANSE_SELECT_DIALOG
CMD_CLEANSE_INCOMING_FIELDS_SECTION
CMD_CLEANSE_ASSET_SELECTION_SECTION
CMD_CLEANSE_FIELD_MAPPING_SECTION
CMD_CLEANSE_OUTPUT_FIELDS_SECTION
CMD_CLEANSE_ADVANCED_PROPERTIES_SECTION
HIDD_DATAMASK_PORTS
HIDD_DATA_MASKING
CMD_DMASK_MASKRULES
CMD_DEDUP_SELECT_DIALOG
CMD_DEBUG_INCOMING_FIELDS_SECTION
CMD_DEDUP_ASSET_SELECTION_SECTION
CMD_DEDUP_FIELD_MAPPING_SECTION
CMD_DEDUP_OUTPUT_FIELDS_SECTION
CMD_DEDUP_ADVANCED_PROPERTIES_SECTION
CMD_PROP_SECTION_EXPRESSION
CMD_PROP_SECTION_EXPRESSION_WINDOW
CMD_ADV_PROP_EXP
CMD_PROP_SECTION_FLTR_CONDITION
CMD_FILTER_CONDITION_DIALOG
CMD_ADV_PROP_FIL
HL_HIERBUILDERTX
R2H_INPUT_PROPERTIES
R2H_FIELD_MAPPING
HSCHEMAS_PAGE
EDIT_HSCHEMA_PAGE
VIEW_HSCHEMA_PAGE
X2R_INPUT_PROPERTIES
X2R_INPUT_FIELD_MAPPING
X2R_FIELD_MAPPING
X2R_OUTPUT_FIELDS
HSCHEMA_OUTPUT_GROUPS
SELECT_OUTPUT_GROUPS
CMD_PROP_SECTION_HPROCESSOR_RELATIONAL
CMD_HPROCESSOR_INPUT_FIELD_RELATIONAL
CMD_HPROCESSOR_OUTPUT_FIELD_RELATIONAL
CMD_HPROCESSOR_DATA_SOURCE_RELATIONAL
CMD_HPROCESSOR_DATA_CONFIG_RELATIONAL
CMD_PROP_SECTION_HPROCESSOR_HIERARCHICAL
CMD_HPROCESSOR_INPUT_FIELD_HIERARCHICAL
CMD_HPROCESSOR_OUTPUT_FIELD_HIERARCHICAL
CMD_HPROCESSOR_DATA_SOURCE_HIERARCHICAL
CMD_HPROCESSOR_DATA_CONFIG_HIERARCHICAL
CMD_HPROCESSOR_DATA_CONFIG_HIERARCHICAL
CMD_HPROCESSOR_EXPRESSION
CMD_PROP_SECTION_HPROCESSOR_FLATTENED
CMD_HPROCESSOR_INPUT_FIELD_FLATTENED
CMD_HPROCESSOR_SCHEMA_CHANGED
CMD_HPROCESSOR_OUTPUT_FIELD_FLATTENED
INPUT_PROP
INPUT_SECTION_INPUT_FIELDS
CMD_PROP_SECTION_JAVA_FIELDS
CMD_JAVA_FIELD_DIALOG
CMD_ADV_PROP_JAV
CMD_PROP_SECTION_JAVA
CMD_JAV_CODE_SECTION
CMD_JAVA_FULLCODE_DIALOG
CMD_PROP_SECTION_JNR_CONDITION
CMD_ADV_PROP_JNR
CMD_LABELER_ASSET_SELECTION_SECTION
CMD_LABELER_INCOMING_FIELDS_SECTION
CMD_LABELER_SELECT_DIALOG
CMD_LABELER_ADVANCED_PROPERTIES_SECTION
CMD_LABELER_FIELD_MAPPING_SECTION
CMD_LABELER_OUTPUT_FIELDS_SECTION
CMD_PROP_SECTION_LKP
CMD_LKUP_CUSTOM_QUERY_DIALOG
CMD_PROP_SECTION_LKP_CONDITION
CMD_PROP_SECTION_LKP_FIELDS
CMD_ADV_PROP_LKP
HL_LOOKUP_SOURCE_FILTER
CMD_PROP_SECTION_LKP_INCOMING_FIELDS
CMD_ML_MODE_SECTION
CMD_ML_REQUEST_MAPPING_SECTION
CMD_ML_RESPONSE_FIELDS_SECTION
CMD_ML_BATCHING_SECTION
CMD_PROP_SECTION_MPLT
CMD_SELECT_MAPPLET_DIALOG
CMD_PROP_SECTION_MPLT_FIELD_MAPPING
CMD_PROP_SECTION_MLPT_PARAM
CMD_PROP_SECTION_MPLT_FIELDS
CMD_PROP_SECTION_NORMALIZED_FIELDS
CMD_NORMALIZED_FIELD_DIALOG
CMD_NORMALIZED_FIELD_FROM_INCOMING_DIALOG
CMD_PROP_SECTION_NORMALIZED_FIELD_MAPPING
CMD_ADV_PROP_NRM
OUTPUT_PROP
OUTPUT_SECTION_OUTPUT_FIELDS
OUTPUT_SECTION_OUTPUT_FIELD_MAPPING
CMD_PARSE_SELECT_DIALOG
CMD_PARSE_INCOMING_FIELDS_SECTION
CMD_PARSE_ASSET_SELECTION_SECTION
CMD_PARSE_FIELD_MAPPING_SECTION
CMD_PARSE_OUTPUT_FIELDS_SECTION
CMD_PARSE_ADVANCED_PROPERTIES_SECTION
CMD_PYTHON_ADV_PROP
CMD_PYTHON_RESOURCE_FILES
CMD_PYTHON_CODE_SECTION
CMD_PYTHON_PARTITION_KEYS
CMD_PROP_SECTION_RANK
CMD_GROUP_BY_RANK
CMD_ADV_PROP_RANK
CMD_PROP_SECTION_ROUTER_GROUPS
CMD_ROUTER_EDIT_GROUP_NAME_DIALOG
CMD_ROUTER_FILTER_CONDITION_DIALOG
CMD_ADV_PROP_RTR
CMD_PROP_SECTION_RULESPEC
CMD_SELECT_RULESPEC_DIALOG
CMD_PROP_SECTION_RULESPEC_INCOMING_FIELDS
CMD_PROP_SECTION_RULESPEC_FIELD_MAPPING
CMD_PROP_SECTION_RULESPEC_FIELDS
CMD_ADV_PROP_RULESPEC
CMD_PROP_SECTION_SG_SEQ
CMD_PROP_SECTION_SG_OUT_FIELDS
CMD_PROP_SECTION_SG_SEQ
CMD_ADV_PROP_SEQ
CMD_PROP_SECTION_SORTER
CMD_ADV_PROP_SRT
CMD_PROP_SECTION_SQL
STORED_PROC_EXEC_ORDER
DESIGNER_STORED_PROC_SEARCH
CMD_PROP_SECTION_SQL_FIELD_MAPPING
CMD_PROP_SECTION_SQL_FIELDS
CMD_ADV_PROP_SQL
STRPAR_GEN_PROP
STRPAR_FIELD_MAP
STRPAR_OUT_FIELDS
STRPAR_SELECT
CMD_PROP_SECTION_TRANSACTION_CONTROL
CMD_ADV_PROP_TRANSACTION
CMD_PROP_SECTION_UNION_FIELDS
CMD_UNION_FIELD_SELECTION_DIALOG
CMD_UNION_FIELD_DIALOG
CMD_PROP_SECTION_UNION_FIELD_MAPPING
CMD_ADV_PROP_UNI
CMD_PROP_SECTION_VELOCITY_INPUT
CMD_PROP_SECTION_VELOCITY
CMD_PROP_SECTION_VELOCITY_TEST
CMD_PROP_SECTION_VELOCITY_OUTPUT
CMD_VERIFIER_ASSET_SELECTION_SECTION
CMD_VERIFIER_SELECT_DIALOG
CMD_VERIFIER_INCOMING_FIELDS_SECTION
CMD_VERIFIER_FIELD_MAPPING_SECTION
CMD_VERIFIER_OUTPUT_FIELDS_SECTION
CMD_VERIFIER_ADVANCED_PROPERTIES_SECTION
VIEW_BUSINESS_SERVICES_PAGE
VIEW_BUSINESS_SERVICE_PAGE
EDIT_BUSINESS_SERVICE_PAGE
CMD_PROP_SECTION_HIERARCHY
CMD_PROP_SECTION_HIERARCHY_OUTPUT_FIELDS
CMD_PROP_SECTION_HIERARCHY_REQUEST_MAPPING
CMD_PROP_SECTION_HIERARCHY_RESPONSE_MAPPING
CMD_ADV_PROP_WSV