You can configure the source fields that you want to use in the data flow. Configure source fields on the Fields tab of the Properties panel.
Configuration options vary based on the connection type. For most connection types, you can add and remove source fields, configure how the fields are displayed, edit field metadata, and restore original fields from the source object. For some connection types, you can restore original fields from the source object only. For more information about configuring source fields, see the help for the appropriate connector.
You can configure source fields in the following ways:
Add the source file name to each row.
If you are using a file list as the source, and you want to identify the source for each row, add the source file name to the field list. You can pass this information to the target table.
To add the source file name to each row, enable the Add Currently Processed Filename Field option. The Add Currently Processed Filename Field option is visible for file sources.
When you enable or disable this option, Data Integration adds or removes the CurrentlyProcessedFileName field but it doesn't synchronize the fields with the source object. To synchronize with the source object, click the Refresh icon. You can synchronize all fields, synchronize new fields only, or skip the synchronization.
Retain existing fields at runtime.
If field metadata changes after a mapping is saved, Data Integration uses the updated field metadata when you run the mapping. Typically, this is the desired behavior. However, if the mapping uses a native flat file connection and you want to retain the metadata used at design time, enable the Retain existing fields at runtime option. When you enable this option, Data Integration mapping tasks will use the field metadata that was used when you created the mapping.
Add and remove fields.
You can add fields to a mapping source. Add a field to retrieve a field from the source object that is not displayed in the list. To add a field, click Add Field, and then enter the field name, type, precision, and scale.
You can also remove fields that you do not want to use in the mapping. To remove fields, select the fields that you want to remove, and then click Delete.
Change the sort order.
You can display source fields in native order, ascending order, or descending order. To change the sort order, click Sort, and select the appropriate sort order.
Use technical field names or labels.
You can display field names by label or technical field name.
To change the display option for field names, select Options > Use Technical Field Names or Options > Use Labels.
Edit field metadata.
You can edit the metadata for a field. You might edit metadata to change information that is incorrectly inferred. When you edit metadata, you can change the name, native type, native precision, and native scale, if applicable for the data type. For some source types, you can also change the transformation data type in the Type column.
To edit the name or metadata for one or more fields, click Options > Edit Metadata. When you edit metadata, you can also display native names by label or technical field name. To change the display option for native names, select Options > Show Technical Field Names or Options > Show Labels.
When you change the metadata for a field, avoid making changes that can cause errors when you run the task. For example, you can usually increase the native precision or native scale of a field without causing errors. But if you reduce the precision of a field, you might cause the truncation of data.
You can't edit field metadata in mappings in SQL ELT mode.
Restore original fields from the source object.
To restore the original fields from the source object, enable the Synchronize option. When you synchronize fields, Data Integration restores deleted source fields and adds fields that are new to the source. Data Integration removes any added fields that do not have corresponding fields in the source object.
Data Integration updates the metadata for existing source fields based on whether you synchronize all fields or synchronize new fields only. When you synchronize all fields, Data Integration replaces any field metadata that you edited with the field metadata from the source object. When you synchronize new fields only, Data Loader retains the metadata for any existing source field. Data Integration does not revert changes that you made to the Name field.
Editing native data types in complex file sources
Data Integration processes native data types in complex file sources differently on an advanced cluster and on the Data Integration Server.
On an advanced cluster, hierarchical data types such as array, map, and struct are assigned those native types. For example, a map field in an Amazon S3 source might have the native data type "map (string_integer)." You cannot edit the metadata for array, map, or struct fields.
On the Data Integration Server, Data Integration flattens complex hierarchical data types into native string datatypes with precision up to 4000 characters. Some native data types come from the connector, and others come from the parser that Data Integration uses when it reads the source data. Parser data types are prefixed with the format type. For example, in an Amazon S3 source with the Avro format, a map field that comes from the parser has the native data type avro_string. You can change the native data type for the connector and parser fields.
To change the native data type, edit the metadata for the source, and select the appropriate data type in the Native Type column.
When you change the native data type, you cannot change a non-parser data type to a parser data type. For example, in an Amazon S3 source, Data Integration sets the native data type for the FileName field to string. You can change the native data type to nstring but not to avro_string. Similarly, you cannot change a parser data type to a non-parser data type.
For more information about editing native data types in complex file sources, see the help for the appropriate connector.
Editing transformation data types
When Data Integration reads source data, it converts the native data types to comparable transformation data types before it transforms the data. When Data Integration writes to a target, it converts the transformation data types to the comparable native data types. When you edit source metadata, you can sometimes change the transformation data type for a field.
You can change the transformation data type for connectors in which the native data type has multiple corresponding transformation data types. For example, in a Kafka source, you can map the native data type binary to the transformation data type binary or string.
To change the transformation data type, edit the metadata for the source, and select the appropriate transformation data type in the Type column.
When you edit the transformation data type for a field, Data Integration updates the data type for the field in the downstream transformations. It also updates the data type for the field in the target if the target is created at runtime. If the mapping contains an existing target, you might need to edit the field metadata in the target to ensure that the data types are compatible.
For more information about editing transformation data types for different source types, see the help for the appropriate connector.