Amazon S3 Connector > Mappings and Mapping Tasks with Amazon S3 > Amazon S3 Objects in Mappings
  

Amazon S3 Objects in Mappings

When you create a mapping, you can configure a Source or Target transformation to represent an Amazon S3 object.

Amazon S3 Sources in Mapping

To read data from Amazon S3, configure an Amazon S3 object as the Source transformation in a mapping.
Specify the name and description of the Amazon S3 source. Configure the source and advanced properties for the source object. When you create a task with Amazon S3 as lookup, the lookup condition does not get applied.
The following table describes the source properties that you can configure in a Source transformation:
Property
Description
Connection Name
Name of the source connection.
Source Type
Source type. Select one of the following types:
  • - Single Object.
  • - Parameter. Select Parameter to define the source type when you configure the mapping task.
Object
Source object for the mapping.
Formatting Options
Amazon S3 format options. Opens the Formatting Options dialog box to define the format of the file. Default is delimited.
Configure the following format options:
  • - Delimiter: Delimiter character. You can configure other parameters such as comma, tab, colon, semicolon, or others.
  • - Text Qualifier: Character to qualify text. You can configure other parameters such as single quote or double quote.
The following table describes the advanced source properties that you can configure in a Source transformation:
Property
Description
Enable Downloading S3 File in Multiple Parts
Downloads large Amazon S3 objects in multiple parts.
The Secure Agent downloads the object in multiple parts from Amazon S3.
When the file size of an Amazon S3 object is greater than 5 MB, you can choose to download the object in multiple parts in parallel.
Header Line Number
Amazon S3 Connector does not honor this property.
Read Data From Line
Specify the line number from where you want the Secure Agent to read data.
Default is 1.
Note: To read data from the header, the value of the Header Line Number and the Read Data From Line fields should be the same.
Staging File Location
Amazon S3 staging directory.
When you run a task in Secure Agent runtime environment, specify a directory path that is available on the corresponding Secure Agent machine in the runtime environment.
When you specify the directory path, the Secure Agent create folders depending on the number of partitions that you specify in the following format: InfaS3Staging<00/11><timestamp>_<partition number> where, 00 represents read operation and 11 represents write operation.
For example, InfaS3Staging000703115851268912800_0
Note: The temporary files are created within the new directory.
When you specify a directory name, if a folder with same name already exists, the Secure Agent deletes the contents of the folder. You must have the write permission for the specified location.
If you do not specify a directory path, the Secure Agent uses a temporary directory as the staging file location.
When you run a task in Hosted Agent runtime environment, leave the staging directory location blank. The Hosted Agent creates a directory at a temporary location.
Part Size
Specifies the part size of an object. Default is 5 MB.
Tracing Level
Sets the amount of detail that appears in the log file.
You can choose terse, normal, verbose initialization or verbose data. Default is normal.
Note: Amazon S3 source does not support mappings with source partition.

Amazon S3 Targets in Mapping

To insert data to Amazon S3, configure an Amazon S3 object as the target in a mapping.
Specify the name and description of the Amazon S3 target. Configure the target and advanced properties for the target object.
The following table describes the target properties that you can configure in a Target transformation:
Property
Description
Connection
Name of the target connection.
Target Type
Target type. Select one of the following types:
  • - Single Object.
  • - Parameter. Select Parameter to define the target type when you configure the task.
Object
Name of the target object. You can select an existing object or create an object at runtime.
Create Target
Creates a target.
Enter a name and path for the target object and select the source fields that you want to use.
By default, all source fields are used. The target name can contain alphanumeric characters.
You can use only a period (.), an underscore (_), an at the rate sign (@), a dollar sign ($), and a percentage sign (%) special characters in the file name. When you specify the name and path for the target object, the object is created in the specified path under the bucket name and folder name specified in the connection properties.
For example, if you specify Finance/Reports in the Folder name in the connection properties and East/Sample/Report1 in the Create Target property, the target with name Report1 will be created under Finance/Reports/East/Sample.
Note: Do not use a single slash (/) in the beginning of the path. Do not use double slash (//) or double dots (..) in the path.
Formatting Options
Amazon S3 format options. Opens the Formatting Options dialog box to define the format of the file. Default is delimited.
Configure the following format options:
  • - Delimiter: Delimiter character. You can configure other parameters such as comma, tab, colon, semicolon, or others.
  • - Text Qualifier: Character to qualify text. You can configure other parameters such as single quote or double quote.
  • - Escape: Escape character.
Operation
Select the target operation.
The Amazon S3 target is a delimited file and you can run the task with only the insert operation. When you run the task, the Secure Agent replaces all data in the file.
You can use parameters defined in a parameter file in the target name for a mapping task.
Note: Do not use the $LastRunTime and $LastRunDate system parameters in the file name when you create a target. Else, the mapping task generates an exception.
For a mapping, use input and output parameters to parameterize the target file name. Specifying the folder path in create target is applicable only to a mapping.
The following table describes the advanced target properties:
Property
Description
Encryption Type
Method you want to use to encrypt data. Select one of the following values:
  • - None. The data is not encrypted.
  • - Client Side Encryption.
    • - The Secure Agent encrypts data while uploading the delimited files to Amazon buckets.
    • - You must select client-side encryption in the advanced properties if you specify a master symmetric key or customer master key ID in the Amazon S3 connection properties.
  • - Server Side Encryption.
    • - You must select server-side encryption in the advanced properties if you specify a customer master key ID in the Amazon S3 connection properties.
    • - If you select the server-side encryption in the advanced properties and do not specify the customer master key ID in the connection properties, Amazon S3-managed encryption keys are used to encrypt data.
Folder Path
The complete path to the Amazon S3 objects and must include the bucket name and any folder name.
Ensure that you do not use a forward slash at the end of the folder path.
For example, <bucket name>/<my folder name>
The folder path specified at run time overrides the path specified while creating a connection.
Compression Type
Compress the data in GZIP format when you write the data to Amazon S3.
The target file in Amazon S3 will have .gz extension. The Secure Agent compresses the data and then sends the data to Amazon S3 bucket.
Default is None.
Part Size
Specifies the part size of an object.
Default is 5 MB.
TransferManager Thread Pool Size
Specifies the number of the threads to write data in parallel.
Amazon S3 Connector uses the AWS TransferManager API to upload a large object in multiple parts to Amazon S3.
When the file size is more than 5 MB, you can configure multipart upload to upload object in multiple parts in parallel. If you set the value of the TransferManager Thread Pool Size to greater than 50, the value reverts to 50.
Default is 10.
Merge Partition Files
Specifies whether the Secure Agent must merge all the partition files into a single file or maintain separate files based on the number of partitions specified to write data to the Amazon S3 targets.
Default is not selected.
Distribution Column
Specify the name of the column that is used to create multiple target files during run time.
Staging File Location
Amazon S3 staging directory.
When you run a task in Secure Agent runtime environment, specify a directory path that is available on the corresponding Secure Agent machine in the runtime environment.
When you specify the directory path, the Secure Agent create folders depending on the number of partitions that you specify in the following format: InfaS3Staging<00/11><timestamp>_<partition number> where, 00 represents read operation and 11 represents write operation.
For example, InfaS3Staging000703115851268912800_0
Note: The temporary files are created within the new directory.
When you specify a directory name, if a folder with same name already exists, the Secure Agent deletes the contents of the folder. You must have the write permission for the specified location.
If you do not specify a directory path, the Secure Agent uses a temporary directory as the staging file location.
When you run a task in Hosted Agent runtime environment, leave the staging directory location blank. The Hosted Agent creates a directory at a temporary location.
Object Tags
You can add single or multiple tags to the objects stored on the Amazon S3 bucket.
You can either enter the key value pairs or specify the file path that contains the key value pairs.
For more information, see Object Tag.
Success File Directory
Not applicable.
Error File Directory
Not applicable.
Forward Rejected Rows
Determines whether the transformation passes rejected rows to the next transformation or drops rejected rows.
By default, the mapping application forwards rejected rows to the next transformation.