In a mapping, you can configure a Target transformation to represent an Amazon S3 V2 object as the target to write data to Amazon S3.
Specify the name and description of the Amazon S3 V2 target. Configure the Amazon S3 V2 target and advanced properties for the target object.
The following table describes the Amazon S3 V2 target properties that you can configure in a Target transformation:
Property
Description
Connection
Name of the Amazon S3 V2 target connection. Select a target connection or click New Parameter to define a new parameter for the target connection.
If you want to overwrite the parameter at runtime, select the Allow parameter to be overridden at run time option when you create a parameter. When the task runs, the agent uses the parameters from the file that you specify in the task advanced session properties.
Target Type
Target type. Select one of the following types:
- Single Object
- Parameter: Select Parameter to define the target type when you configure the mapping task.
Object
Name of the target object.
You can select an existing object or create an object at runtime. When you create an object at runtime, enter a name and the path for the target object.
Parameter
Select an existing parameter for the source object or click New Parameter to define a new parameter for the target object. The Parameter property appears only if you select Parameter as the target type.
If you want to overwrite the parameter at runtime, select the Allow parameter to be overridden at run time option when you create a parameter. When the task runs, the agent uses the parameters from the file that you specify in the task advanced session properties.
Create Target
Creates a target.
Enter a name and path for the target object. You can use parameters defined in a parameter file in the target name.
Specifies the file format that the Amazon S3 V2 Connector uses to write data to Amazon S3.
You can select the following file format types:
- None 1. Writes data in binary format.
When you write binary data to an Amazon S3 V2 target created at run time, ensure that the incoming field that contains the binary data is named data. If the field name differs, use field rules or an Expression transformation to change the field name to data.
- Flat. Writes data to a flat file.
- Avro. Writes data in Avro format.
- ORC. Writes data in ORC format.
- Parquet. Writes data in Parquet format.
- JSON2. Writes data in JSON format.
- Delta. Writes data to Delta files.
- Iceberg1. Writes data in Apache Iceberg format.
Default is None.
For more information about format options, see ファイル形式オプション.
Operation
Type of the target operation.
You can perform only insert operation on an Amazon S3 V2 target.
1Doesn't apply to mappings in advanced mode.
2Applies only to mappings in advanced mode.
The following table describes the Amazon S3 V2 advanced target properties that you can configure in a Target transformation:
Property
Description
Overwrite File(s) If Exists
Overwrites an existing target file.
Default is true. For more information about overwriting the existing files, see 既存のファイルの上書き.
Folder Path
Bucket name or folder path where you want to write the Amazon S3 target file. The path that you enter here overrides the path specified for the target configured to create at runtime.
If applicable, include the folder name that contains the target file in the <bucket_name>/<folder_name> format.
If you do not provide the bucket name and specify the folder path starting with a slash (/) in the /<folder_name> format, the folder path appends with the folder path that you specified in the connection properties.
For example, if you specify the /<dir2> folder path in this property and <my_bucket1>/<dir1> folder path in the connection property, the folder path appends with the folder path that you specified in the connection properties in <my_bucket1>/<dir1>/<dir2> format.
If you specify the <my_bucket1>/<dir1> folder path in the connection property and <my_bucket2>/<dir2> folder path in this property, the Secure Agent writes the file in the <my_bucket2>/<dir2> folder path that you specify in this property.
File Name
Creates a new file name or overwrites an existing target file name.
Ensure that you have write permissions to the directory and there is sufficient space to enable staging of the entire file. Default staging directory is the /temp directory on the Secure Agent machine.
When you specify the directory path, the Secure Agent create folders depending on the number of partitions that you specify in the following format: InfaS3Staging<00/11><timestamp>_<partition number> where, 00 represents read operation and 11 represents write operation.
For example, InfaS3Staging000703115851268912800_0
The temporary files are created within the new directory.
The staging directory target property does not apply to Avro, ORC, Parquet, and Delta files.
File Merge
This property is not applicable for Amazon S3 V2 Connector.
Hadoop Performance Tuning Options
This property is not applicable for Amazon S3 V2 Connector.
Compression Format
Compresses data when you write data to Amazon S3.
You can compress the data in the following formats:
- None
- Bzip22
- Deflate
- Gzip
- Lzo
- Snappy
- Zlib
Default is None.
Note:
Amazon S3 V2 Connector does not support the Lzo compression format even though the option appears in this property.
The key value pairs to add single or multiple tags to the objects stored on the Amazon S3 bucket.
You can either enter the key value pairs or specify the file path that contains the key value pairs.
Use this property when you run a mapping to write a file of flat format type. For more information about the object tags, see オブジェクトタグ.
This property applies only to flat files.
TransferManager Thread Pool Size1
The number of threads to write data in parallel.
Default is 10. Use this property when you run a mapping to write a file of flat format type.
Amazon S3 V2 Connector uses the AWS TransferManager API to upload a large object in multiple parts to Amazon S3.
When the file size is more than 5 MB, you can configure multipart upload to upload object in multiple parts in parallel. If you set the value of TransferManager Thread Pool Size to greater than 50, the value reverts to 50.
This property applies only to flat files.
Part Size1
Uploads the part size of an Amazon S3 object in bytes.
Default is 5 MB. Use this property when you run a mapping to write a file of flat format type.
This property applies only to flat files.
Merge Partition Files1
Determines whether the Secure Agent must merge the number of partition files as a single file or maintain separate files based on the number of partitions specified to write data to the Amazon S3 V2 targets.
This property applies only to flat files.
Temporary Credential Duration
The time duration during which an IAM user can use the dynamically generated temporarily credentials to access the AWS resource. Enter the time duration in seconds.
Default is 900 seconds.
If you require more than 900 seconds, you can set the time duration maximum up to 12 hours in the AWS console and then enter the same time duration in this property.
Checksum Algorithm1
Specifies the checksum algorithm used when you write to Amazon S3.
You can select one of the following options:
- Default
- CRC64NVME
- CRC32
- CRC32C
- SHA1
- SHA256
The Default option applies the CRC32 checksum algorithm.
For complex files, if you want to write to multiple target objects in a mapping and use a different checksum algorithm for each object, store each object in a separate Amazon S3 bucket.
If all target objects are stored in the same Amazon S3 bucket, create a separate mapping for each target object to use a different checksum algorithm.
Forward Rejected Rows
This property is not applicable for Amazon S3 V2 Connector.
1Doesn't apply to mappings in advanced mode.
2Applies only to mappings in advanced mode.
When you create a mapping and the column name in the Amazon S3 source or target object contains special characters, the Secure Agent replaces the special characters with an underscore (_) and the mapping fails.
