Using intelligent structure models in mappings in advanced mode
To use an intelligent structure model in a mapping in advanced mode, configure the Source transformation to discover the data structure at the source or add a Structure Parser transformation downstream in the mapping.
The model generates hierarchical output. Models that you use in advanced mode can contain up to 8,000 fields.
To use an intelligent structure model when you configure the Source transformation, select an existing model or create a new model when you configure the Source transformation. You can use an intelligent structure model with an Amazon S3 V2 or a Microsoft Azure Data Lake Gen2 connection. The following table describes the actions that you perform when you configure the source properties to use an intelligent structure model:
Property
Action
Connection
Select an Amazon S3 V2 or a Microsoft Azure Data Lake Gen2 connection.
Source Type
Select Single Object.
Object
Select a file or folder as the source object:
- To select a file, select File as the source type in the Advanced area, and then select a file of a type that is supported by Intelligent Structure Discovery.
- To select a folder, select Directory as the source type in the Advanced area, and then select a folder that contains one or more Avro, Parquet, or ORC files.
Format
Select Discover Structure.
Intelligent Structure Model
Select one of the following options to associate a model with the transformation:
- Select. Select an existing model.
- New. Create a new model. Select Design New to create the model. Select Auto-generate from sample file for Intelligent Structure Discovery to generate a model based on sample input that you select.