Tasks > Mapping tasks > Mapping task configuration
  

Mapping task configuration

Complete the following steps to create a mapping task:
  1. 1Create the mapping task.
  2. 2Configure the source.
  3. 3Configure the target.
  4. 4Optionally, configure CDC runtime properties.
  5. 5Define input parameters.
  6. 6Optionally, define persistent values.
  7. 7Optionally, configure runtime options.
  8. 8Optionally, configure the runtime strategy.
As you configure the task, you can click Save to save your work at any time.

Creating a mapping task

Define general properties of the task on the General page.
    1To create a mapping task, click New > Tasks, and then complete one of the following steps:
    2To edit a mapping task, on the Explore page, navigate to the mapping task. In the row that contains the task, click Actions and select Edit.
    3Configure the following fields:
    Field
    Description
    Name
    Name of the task.
    Task names can contain alphanumeric characters, spaces, and the following special characters: _ . + -
    Maximum length is 100 characters. Task names are not case sensitive.
    Location
    Project or folder in which the task resides.
    If the Explore page is currently active and a project or folder is selected, the default location for the asset is the selected project or folder. Otherwise, the default location is the location of the most recently saved asset.
    Description
    Description of the task.
    Maximum length is 4000 characters.
    Runtime Environment
    Runtime environment that contains the Secure Agent to run the task.
    Note: In a sub-organization, you can't run a mapping task on a Secure Agent group shared by the parent organization if the task is based on a mapping in SQL ELT mode or advanced mode.
    Mapping
    Mapping associated with the task.
    To select the mapping, click Select and navigate to the mapping that you want to use.
    To open the mapping, click View.
    4Configure the job priority and target duration if the mapping task is based on a mapping in advanced mode.
    The following table describes the properties:
    Property
    Description
    Job Priority
    Priority to schedule the job. High-priority jobs are scheduled sooner than medium-priority or low-priority jobs.
    Target Duration (seconds)
    Time in seconds to complete the job.
    You can view whether the target duration was met or exceeded when you monitor the job. The actual time to complete the job depends on the advanced cluster infrastructure and the runtime strategy.
    5Click Next.

Configuring sources

Configure source and lookup source parameters on the Sources page. The Sources page displays differently based on the basis for the task. If the mapping does not include source or lookup parameters, the Sources page does not appear.
You can add a single source object or multiple source objects based on the connection type and mapping configuration.
If the mapping specifies a connection parameter, and after you create the mapping task you edit the mapping to change the source, you might need to edit the task. Check the specific connection in the mapping task to determine if you need to reset it, and then validate the task again.
    1For each source or lookup source parameter, configure the following details as required:
    Parameter detail
    Description
    Connection
    Select a connection.
    To create a connection, click New. To edit a connection, click View, and in the View Connection dialog box, click Edit.
    Tip: Hover over the source connection tool tip to view more information about the connection.
    Source type
    Source type. The options available depend on the connection type.
    Object
    Select an object or enter a query.
    If a list of objects doesn't appear, click Select.
    The Select Source Object dialog box displays up to 200 objects. If the object you want to use doesn't appear, enter a search string to reduce the number of objects.
    Tip: Hover over the source object tool tip to view more information about the object.
    Display Technical Field Names Instead of Labels
    Displays technical names instead of business names.
    Not available for all connection types.
    Add Currently Processed File Name
    Adds the source file name to each row. Data Integration adds the CurrentlyProcesedFileName field to the source at run time.
    Available for parameterized source objects with flat file connections.
    2Configure formatting options if required.
    The properties available vary based on the connection type. For information about a particular connector's properties, see the help for the appropriate connector.
    3Optionally, expand the Data Preview area to preview the data.
    To display fields in alphabetical order, click Display fields in alphabetical order. By default, fields appear in the order returned by the source system.
    To download the preview results, click Download.
    4For parameterized source objects, configure data filters if necessary.
    5For parameterized source objects, configure sort options if necessary.
    6Configure field metadata if required.
    You can configure field metadata for sources with certain connection types. To see if a connector supports field metadata configuration, see the help for the appropriate connector. You can't update field metadata in mapping tasks that are based on mappings in SQL ELT mode.
    To edit field metadata, expand the Field Data Types area and select the row that contains the field that you want to edit. Configure the following attributes:
    Data type attribute
    Description
    Retain existing fields at runtime
    When enabled, the task uses the field metadata that is configured in the task.
    If field metadata changes after a task is saved, Data Integration uses the updated field metadata. Typically, this is the desired behavior. However, if the task uses a flat file connection and you want to retain the metadata used at design time, enable this option.
    Type
    The data type of the field.
    Precision
    Total number of digits in a number. For example, the number 123.45 has a precision of 5.
    The precision must be greater than or equal to 1.
    Scale
    Number of digits to the right of the decimal point of a number. For example, the number 123.45 has a scale of 2.
    Scale must be greater than or equal to 0.
    The scale of a number must be less than its precision.
    The maximum scale for a numeric data type is 65535.
    Not editable for all data types.
    7Optionally, configure advanced options for the selected connection.
    The options available depend on the connection.
    When you edit advanced options, you can restore default values for individual properties or for all advanced properties. Default values are defined in the mapping that the task is based on.
    For information about a particular connector's properties, see the help for the appropriate connector.
    8Optionally, configure partitions.
    The partitioning methods that you can select depend on the connection.
    If you defined partitions in the mapping, the partition details are displayed in the Partitions area. You can edit these details to override the partitions configured in the mapping at run time.
    9Click Next.

Configuring targets

The Targets page displays differently depending on the basis for the task.
The Targets page displays when the mapping includes parameters for target connections or target objects. The properties that you need to specify are based on the type of parameter.
    1For each target parameter, configure the following details:
    Target parameter detail
    Description
    Connection
    Select a connection.
    To create a connection, click New. To edit a connection, click View, and in the View Connection dialog box, click Edit.
    Tip: Hover over the target connection tool tip to view more information about the connection.
    Object
    Select a target object.
    If a list of objects does not appear, click Select.
    The Select Target Object dialog box displays up to 200 objects. If the object you want to use does not display, enter a search string to reduce the number of objects that display.
    Tip: Hover over the target object tool tip to view more information about the object.
    Operation
    Target operation, either insert, update, upsert, delete, or data driven.
    Not editable for all connection types.
    Truncate target
    Truncates the target object before inserting new rows.
    Applies to insert and data driven operations.
    Not available for all connection types.
    Enable target bulk load
    Uses the database bulk API to perform an insert operation.
    Use the bulk API to write large amounts of data to the database with a minimal number of API calls. Loading in bulk mode can improve performance, but it limits the ability to recover because no database logging occurs.
    Applies to insert operations.
    Not available for all connection types.
    Display technical names instead of labels
    Displays technical names instead of business names.
    Not available for all connection types.
    2Configure formatting options if required.
    The properties available vary based on the connection type. For information about a particular connector's properties, see the help for the appropriate connector.
    3Optionally, expand the Data Preview area to preview the data.
    To display fields in alphabetical order, click Display mapplet fields in alphabetical order. By default, fields appear in the order returned by the target system.
    To download the preview results, click Download.
    4Configure field metadata if required.
    You can configure field metadata for targets with certain connection types. To see if a connector supports field metadata configuration, see the help for the appropriate connector. You can't update field metadata in mapping tasks that are based on mappings in SQL ELT mode.
    To edit field metadata, expand the Field Data Types area and select the row that contains the field that you want to edit. Configure the following attributes:
    Data type attribute
    Description
    Retain existing fields at runtime
    When enabled, the task uses the field metadata that is configured in the task.
    If field metadata changes after a task is saved, Data Integration uses the updated field metadata. Typically, this is the desired behavior. However, if the task uses a flat file connection and you want to retain the metadata used at design time, enable this option.
    Type
    The data type of the field.
    Precision
    Total number of digits in a number. For example, the number 123.45 has a precision of 5.
    The precision must be greater than or equal to 1.
    Scale
    Number of digits to the right of the decimal point of a number. For example, the number 123.45 has a scale of 2.
    Scale must be greater than or equal to 0.
    The scale of a number must be less than its precision.
    The maximum scale for a numeric data type is 65535.
    Not editable for all data types.
    5Optionally, configure advanced options for the selected connection.
    The options available depend on the connection.
    When you edit advanced options, you can restore default values for individual properties or for all advanced properties. Default values are defined in the mapping that the task is based on.
    For information about a particular connector's properties, see the help for the appropriate connector.
    6Click Next.

Configuring CDC runtime properties

The CDC Runtime page displays if the mapping processes change data from a CDC source.
    1For each property, edit or reset the current value.
    2Click Next.

Configuring input parameters

The Input Parameters page displays differently depending on the basis for the task. If the mapping does not contain input parameters, the page doesn't display.
The Input Parameters page displays input parameters such as filter and lookup conditions, field mappings, and expressions. It also displays stored procedure connections. It doesn't display source, target, or lookup connections and objects.
    1Configure the parameters that display.
    Depending on the mapping data flow, you might need to configure some parameters before you configure other parameters. For more information, see Mappings.
    Warning: If the mapping contains a filter parameter, the task might display as valid even if you haven't configured the filter value. You must configure all parameters before the task is valid and you can run it.
    2If necessary, configure stored procedure details.
    This displays when a stored procedure requires a connection. Select a connection or optionally create a new one.
    3If necessary, configure lookup conditions.
    This displays when the lookup condition is parameterized.
    4If necessary, configure mapplet details.
    This displays when a mapplet requires a connection and the mapplet is configured to display on this page.
    For some connection types, you can select Display technical field names instead of labels to display technical names instead of business names.
    To display fields in alphabetical order, click Display mapplet fields in alphabetical order.
    5If necessary, configure field mappings.
    This displays when the field mapping is parameterized. Use the Object list to display fields from different objects.
    Map fields in one of the following ways:
    6Click Next.

Configuring persistent values

The Persistent Values page displays if the mapping contains persisted values such as in-out parameters or sequences.
    1For each sequence value, edit or reset the current value.
    Note: If the Sequence transformation is configured to reset the value for each run, editing the current value in the mapping task has no effect.
    2Configure in-out parameters as necessary.
    In-out parameters don't apply to mapping tasks that are based on mappings in SQL ELT mode.
    3Click Next.

Configuring runtime options

On the Runtime Options page, configure optional runtime options for the task. Expand each area to see the options you can configure.
    1Specify whether to run the task on a schedule or without a schedule.
    Choose one of the following options:
    2Configure email notification options for the task.
    3Optionally, if the mapping contains a Source transformation that incrementally loads files, you can configure the time that the task uses to identify files to load. By default, the task loads files modified after the last job. To load files from a different time, choose one of the following override options:
    Note: If you want to run one job that reprocesses source files without affecting future jobs, run the mapping task with the advanced options configured to reprocess incrementally-loaded source files. For more information about reprocessing jobs, see Reprocessing incrementally-loaded source files .
    4Optionally, in the Advanced Options area, configure the advanced options that are displayed for your connection:
    Field
    Description
    Pre-Processing Commands
    Commands to run before the task.
    Post-Processing Commands
    Commands to run after the task completes.
    Maximum Number of Log Files
    Number of session log files and import log files to retain. By default, Data Integration stores each type of log file for 10 runs before it overwrites the log files for new runs.
    Note: If a dollar sign ($) is present in a custom session log file name, for example, MyLog_$CurrentTime, the file name is dynamic. If you customize the session log file name using a dynamic name, this property doesn't apply. To purge old log files, delete the files manually.
    Schema Change Handling
    Determines how Data Integration picks up changes to the object schema. Select one of the following options:
    • - Asynchronous. Data Integration refreshes the schema when you update the mapping or mapping task, and after an upgrade.
    • - Dynamic. Data Integration refreshes the schema every time the task runs.
    Default is Asynchronous.
    Schema Mismatch Handling
    Determines how Data Integration responds when records in a file don't conform with the schema.
    This setting is available when schema validation is enabled in the Source transformation. Select one of the following options:
    • - Skip mismatched files and continue. Data Integration skips the entire file if at least one record in the file doesn't conform with the schema. Processing continues for all the remaining files.
    • - Stop on first mismatched file. Data Integration stops processing when it encounters a file with at least one record that doesn't conform with the schema.
    Default is "Skip mismatched files and continue." For more information, see Schema mismatch handling.
    Schema mismatch handling doesn't apply to mapping tasks that are based on mappings in SQL ELT mode.
    Dynamic Schema Handling
    Determines how Data Integration applies schema changes from upstream transformations to the target object. Available when the schema change handling is dynamic and the field mapping is automatic.
    For each target, select how Data Integration updates the target schema. The options available are based on the target connection.
    For more information, see Schema change handling or the help for the appropriate connector.
    Not all options appear if the mapping task is based on a mapping in advanced mode.
    5Optionally, to create a parameter file based on the parameters and default values specified in the mapping on which the task is based, click Download Parameter File Template within the parameter file location.
    For more information about parameter file templates, see Mappings.
    6Optionally, if the mapping task contains parameters, you can use parameter values from a parameter file. Choose one of the following options:
    7If the parameter file is stored on a local machine and the task runs a mapping in advanced mode, you can click Download Parameter File to preview the parameter file that the task will use.
    In a serverless runtime environment, the parameter file must be on a data disk.
    8Choose whether to run the task in standard or verbose execution mode, if this option is visible.
    If you select verbose mode, the mapping generates additional data in the logs that you can use for troubleshooting. Select verbose execution mode only for troubleshooting purposes. Verbose execution mode impacts performance because of the amount of data it generates.
    This option does not appear if the mapping task is based on a mapping in advanced mode.
    9Optionally, configure SQL ELT optimization.
    If you use the runtime options to configure SQL ELT optimization for a mapping task that is based on a mapping in advanced mode, you can't enable CLAIRE-powered runtime strategies on the Runtime Strategy page.
    The following table describes the SQL ELT optimization properties:
    Property
    Description
    SQL ELT Optimization Type
    Type of SQL ELT optimization. Use one of the following options:
    • - None. The task processes all transformation logic for the task.
    • - To Source. The task pushes as much of the transformation logic to the source database as possible. Not available in advanced mode.
    • - To Target. The task pushes as much of the transformation logic to the target database as possible. Not available in advanced mode.
    • - Full. The task pushes all of the transformation logic to the source and target databases. The source and target schema must be the same.
    • - $$PushdownConfig. The task uses the SQL ELT optimization type specified in the user-defined parameter file for the task.
      • When you use $$PushdownConfig, ensure that the user-defined parameter is configured in the parameter file.
    When you use SQL ELT optimization, do not use the Error Log Type advanced session property.
    Note: SQL ELT optimization functionality varies depending on the support available for the connector. For more information, see the help for the appropriate connector. SQL ELT optimization doesn't apply to mapping tasks that are based on mappings in SQL ELT mode because mappings in SQL ELT mode are automatically configured to push transformation logic to the cloud data warehouse.
    Optimization Context Type
    Provides context about the mapping configuration for SQL ELT optimization. If you select an option other than None, Data Integration constructs a single query for SQL ELT optimization by combining multiple targets in the mapping based on the target configurations. If you select None, the query is not optimized.
    If Data Integration cannot apply the selected context, Data Integration uses the default SQL ELT optimization behavior.
    Select one of the following options:
    • - None
    • - SCD Type 2 merge
    • - Multi-insert
    Not available in advanced mode.
    For more information, see the help for the appropriate connector.
    SQL ELT Optimization Fallback Option
    If full SQL ELT optimization is not available for the connection, choose how Data Integration handles SQL ELT optimization.
    Choose one of the following options:
    • - Partial SQL ELT. Default. Data Integration pushes as much transformation logic as possible to the source and target database. The task processes any transformation logic that it can't push to a database.
    • - Non SQL ELT. The task runs with no SQL ELT optimization.
    • - Fail Task. Data Integration fails the task.
    Not available in advanced mode.
    Create Temporary View
    Allows the task to create temporary view objects in the database when it pushes the task to the database.
    Use when the task includes an SQL override in the Source Qualifier transformation or Lookup transformation.
    Disabled when the SQL ELT optimization type is None.
    Create Temporary Sequence
    Allows the task to create temporary sequence objects in the database.
    Disabled when the SQL ELT optimization type is None.
    10Optionally, if the task runs in a serverless runtime environment, configure serverless usage properties.
    11Optionally, for mappings in advanced mode, run CLAIRE Tuning to get a set of recommended properties and values for Spark session properties.
    12If necessary, add other session properties now under Advanced Session Properties:
    1. aClick New Session Property.
    2. bSelect a session property from the drop-down list.
    3. cConfigure the session property value.
    13Choose to enable cross-schema SQL ELT optimization.
    Cross-schema SQL ELT optimization doesn't apply to mapping tasks that are based on mappings in SQL ELT mode.
    14If you want to run multiple instances of the task at the same time, enable simultaneous runs of the mapping task.
    Some mapping features might produce unexpected results in simultaneous task runs.

Configuring the runtime strategy

On the Runtime Strategy page, you can get a recommendation for a runtime strategy that optimizes cost or performance.
To get a recommendation for a runtime strategy, the mapping task must be based on a mapping in advanced mode and CLAIRE recommendations must be enabled in your organization. If you select a runtime strategy other than the default, you can't configure SQL ELT optimization on the Runtime Options page.
Note: Runtime strategy recommendations and the target duration confidence level might not appear for some mapping tasks. CLAIRE continues collecting metadata in the background and shows these details when it has enough confidence.
    1Select one of the following runtime strategies:
    2Review the target duration confidence level.
    The target duration confidence level shows the likelihood that the job will run within the target duration that you configured on the General page.
    3Click Save.