Amazon S3 V2 Connector > Mappings and mapping tasks with Amazon S3 V2 > Amazon S3 V2 objects in mappings

Amazon S3 V2 objects in mappings

You can configure a source object to read data from Amazon S3 by specifying the source properties and advanced source properties. You can perform lookup tasks and specify file formatting options in the Amazon S3 V2 mappings. You can parameterize the connection and objects. You can also use a parameter file in the task properties to overwrite the parameters at runtime.

Amazon S3 V2 sources in mappings

In a mapping, you can configure a Source transformation to represent an Amazon S3 V2 object as the source to read data from Amazon S3.

The following table describes the Amazon S3 V2 source properties that you can configure in a source transformation:

Property	Description
Connection Name	Name of the Amazon S3 V2 source connection. Select a source connection or click New Parameter to define a new parameter for the source connection. If you want to overwrite the parameter at runtime, select the Allow parameter to be overridden at run time option when you create a parameter. When the task runs, the agent uses the parameters from the file that you specify in the task advanced session properties.
Source Type	Source type. Select one of the following types: - Single Object - Parameter. Select Parameter to define the source type when you configure the mapping task.
Object	Name of the source object. When you select an object, you can also select a .manifest file object when you want to read from multiple files.
Parameter	Select an existing parameter for the source object or click New Parameter to define a new parameter for the source object. The Parameter property appears only if you select Parameter as the source type. If you want to overwrite the parameter at runtime, select the Overwrite Parameter option.
Format	Specifies the file format that the Amazon S3 V2 Connector uses to read data from Amazon S3. You can select the following file format types: - None - Flat - Avro - ORC - Parquet - Delta Default is None. If you select None as the format type, the Secure Agent reads data from Amazon S3 files in binary format. You cannot use parameterized sources when you select the discover structure format. Open the Formatting Options dialog box to define the format of the file. For more information, see File formatting options.

The following table describes the advanced source properties:

Property	Description
Source Type	Type of the source from which you want to read data. You can select the following source types: - File - Directory Default is File. Directory source type doesn't apply to Delta files. For more information, see Source types in Amazon S3 V2 sources.
Folder Path	Overwrites the bucket name or folder path of the Amazon S3 source file. If applicable, include the folder name that contains the source file in the <bucket_name>/<folder_name> format. If you do not provide the bucket name and specify the folder path starting with a slash (/) in the /<folder_name> format, the folder path appends with the folder path that you specified in the connection properties. For example, if you specify the /<dir2> folder path in this property and <my_bucket1>/<dir1> folder path in the connection property, the folder path appends with the folder path that you specified in the connection properties in <my_bucket1>/<dir1>/<dir2> format. If you specify the <my_bucket1>/<dir1> folder path in the connection property and <my_bucket2>/<dir2> folder path in this property, the Secure Agent reads the file in the <my_bucket2>/<dir2> folder path that you specify in this property.
File Name	Overwrites the Amazon S3 source file name.
Encryption Type	Method you want to use to decrypt data. Default is None. Note: You cannot select client-side encryption, server-side encryption, and server-side encryption with KMS encryption types.
Staging Directory	Path of the local staging directory. Default staging directory is the /temp directory on the machine that hosts the Secure Agent. The staging directory source property doesn't apply to Avro, ORC, Parquet, and Delta files.
Hadoop Performance Tuning Options	This property is not applicable for Amazon S3 V2 Connector.
Compression Format	Decompresses data when you read data from Amazon S3. You can choose to decompress the data in the following formats: - None - Lzo Default is None. Note: Amazon S3 V2 Connector does not support the Lzo compression format even though the option appears in this property.
Download Part Size	Downloads the part size of an Amazon S3 object in bytes. Default is 5 MB. Use this property when you read a file of flat format type. This property applies only to flat files.
Multiple Download Threshold	Minimum threshold size to download an Amazon S3 object in multiple parts. To download the object in multiple parts in parallel, ensure that the file size of an Amazon S3 object is greater than the value you specify in this property. Default is 10 MB. This property applies only to flat files.
Temporary Credential Duration	The time duration during which an IAM user can use the dynamically generated temporarily credentials to access the AWS resource. Enter the time duration in seconds. Default is 900 seconds. If you require more than 900 seconds, you can set the time duration maximum up to 12 hours in the AWS console and then enter the same time duration in this property.
Tracing Level	This property is not applicable for Amazon S3 V2 Connector.

Amazon S3 V2 lookups

You can use Amazon S3 V2 objects in a connected and an unconnected cached Lookup transformation.

For more information about the Lookup transformation, see Transformations.

File formatting options

When you select the format of an Amazon S3 file, you can configure the formatting options.

The following table describes the formatting options for flat files:

Property	Description
Read from data file	Imports the schema from the file in Amazon S3.
Import from schema file	Imports schema from a schema definition file in your local machine. If you select Import from schema file, you can select Schema File to upload a schema file.
Flat File Type	The type of flat file. Default is Delimited.
Delimiter	Character used to separate columns of data. You can configure parameters such as comma, tab, colon, semicolon, or others. To set a tab as a delimiter, you must type the tab character in any text editor. Then, copy and paste the tab character in the Delimiter field. Do not specify a multibyte character as a delimiter in the source object.
Escape Char	Character immediately preceding a column delimiter character embedded in an unquoted string, or immediately preceding the quote character in a quoted string. You can specify a character or \<decimal value>. When you specify \<decimal value>, the agent considers the ASCII character for the decimal value as the escape character. For example, if you specify \64, the agent considers the ASCII character @. To ignore the escape character, specify \0.
Qualifier	Quote character that defines the boundaries of data. You can set the qualifier as single quote or double quote.
Code Page	Select the code page that the agent must use to read data. Amazon S3 V2 Connector supports the following code pages: - MS Windows Latin 1. Select for ISO 8859-1 Western European data. - UTF-8. Select for Unicode and non-Unicode data. - Shift-JIS. Select for double-byte character data. - ISO 8859-15 Latin 9 (Western European). - ISO 8859-2 Eastern European. - ISO 8859-3 Southeast European. - ISO 8859-5 Cyrillic. - ISO 8859-9 Latin 5 (Turkish). - IBM EBCDIC International Latin-1.
Header Line Number	Specify the line number that you want to use as the header when you read data from Amazon S3. You can also read a file that does not have a header. To read data from a file with no header, specify the value of the Header Line Number field as 0. To read data from a file with a header, set the value of the Header Line Number field to a value that is greater than or equal to one. Default is 1.
First Data Row	Specify the line number from where you want the Secure Agent to read data. You must enter a value that is greater or equal to one. To read data from the header, the value of the Header Line Number and the First Data Row fields should be the same. Default is 1.
Is Escape Character Data Retained	Not applicable to Amazon S3 V2 Connector.
Max Rows To Preview	Not applicable to Amazon S3 V2 Connector.
Row Delimiter	Not applicable to Amazon S3 V2 Connector.

Delta files

You can read from and write to Delta files in Amazon Redshift.

A Delta file consists of the following components:

•Parquet files where the data is stored.
•JSON files where the metadata and data change logs are stored.

Each transaction that modifies the data results in a new JSON file. The JSON files are stored in _delta_log directory.

Consider the following rules and guidelines when you read from and write to Delta files:

•You cannot read and write Delta files in a mapping in SQL ELT mode or a mapping task enabled with SQL ELT optimization.
•You cannot use source partitioning or target partitioning when you read from or write to Delta files.
•When you read from a Delta file and edit the metadata, do not change the data type. Else, the mapping fails. You can only change the precision of the data types.
• If you select the Delta format type and select Import from schema file as the value of the Schema Source formatting option, you can only upload a schema file in the JSON format.

The following sample shows a schema file for a Delta file:

{"type": "struct","fields": [{"name": "c_custkey","type": "integer","nullable": true,"metadata": {}},{"name": "c_name","type": "string","nullable": true,"metadata": {}},{"name": "c_address","type": "string","nullable": true,"metadata": {}},{"name": "c_comment","type": "string","nullable": true,"metadata": {}}]}

•When there is a change in the metadata of a Delta file, you cannot write data to the same Delta file in its current folder path or bucket. You must specify a different path or bucket name.