You can use an Amazon S3 V2 object as a target in a mapping or amapping task.
Specify the name and description of the Amazon S3 V2 target. Configure the Amazon S3 V2 target and advanced properties for the target object.
Data encryption in Amazon S3 V2 targets
To protect data, you can encrypt the Amazon S3 files when you write the files to the target. Do not use the master symmetric key and customer master key together.
Select the type of the encryption in the Encryption Type field under the Amazon S3 V2 advanced target properties.
You can select the following types of encryption:
None
The data is not encrypted.
Server-side encryption
Select Server Side Encryption as the encryption type if you want Amazon S3 to encrypt the data using Amazon S3-managed encryption keys when you write to the target.
If you do not specify the customer master key ID in the connection properties, you must select Server Side Encryption as the encryption type.
Server-side encryption with KMS
Select Server Side Encryption with KMS as the encryption type if you want Amazon S3 to encrypt the data using AWS KMS-managed customer master key encryption keys when you write to the target.
The AWS KMS-managed customer master key specified in the connection property must belong to the same region where Amazon S3 is hosted.
For example, if Amazon S3 is hosted in the US West (Oregon) region, you must use the AWS KMS-managed customer master key enabled in the same region.
Client-side encryption
Select Client Side Encryption as the encryption type if you want the Secure Agent to encrypt the data when you write to the target. Client-side encryption uses a master symmetric key, which is a 256-bit AES encryption key in Base64 format or a customer master key.
Informatica encryption
Select Informatica Encryption as the encryption type if you want to encrypt the data using Informatica crypto libraries when you write to a target. Informatica encryption is applicable only when you run mappings on the Secure Agent machine.
To encrypt a file using Informatica Encryption method, perform the following tasks:
1Ensure that the organization administrator has permission to Informatica crypto libraries when you create an Amazon S3 V2 connection.
2Select Informatica Encryption as the encryption type in the advanced target properties.
Note: For Informatica Encryption in the advanced cluster, you must install the Secure Agent on the Amazon EC2 machine.
The following table lists the encryption type supported for various file types:
Encryption type
File type
Client-side encryption
Binary1, Flat
Server-side encryption
Avro, Binary1, Delta1, Flat, JSON2, ORC, Parquet
Server-side encryption with KMS
Avro, Binary1, Delta1, Flat, JSON2, ORC, Parquet
Informatica encryption
Binary1, Flat
1Doesn't apply to mappings in advanced mode.
2Applies only to mappings in advanced mode.
Rules and guidelines for data encryption in Amazon S3 V2 targets
Consider the following rules and guidelines when you configure data encryption in Amazon S3 V2 targets:
•When you use Informatica encryption in a mapping, the _SUCCESS file is not generated in the target directory for mappings in advanced mode.
•When you use Informatica encryption in a mapping, you cannot encrypt more than 1000 files.
For information about the Amazon S3 client encryption policy, see the Amazon S3 documentation.
Overwriting existing files
You can choose to overwrite the existing target files.
Select the Overwrite File(s) If Exists option in the Amazon S3 V2 target advanced properties to overwrite the existing files. By default, the value of the Overwrite File(s) If Exists check box is true.
If you select the Overwrite File(s) If Exists option, the Secure Agent deletes the existing files with same file name as the file that you overwrite, and creates a new file with the same file name in the target directory.
The Secure Agent also deletes all files and folders that have the same prefix as the file names created at design time. This is applicable for Avro, ORC, and Parquet files.
If you do not select the Overwrite File(s) If Exists option, the Secure Agent does not delete the existing files in the target directory. The Secure Agent adds time stamp at the end of each target file name in the following format: YYYYMMDD_HHMMSS_millisecond. For example, the Secure Agent renames the target file name in the following format: output.txt-20171220_091900_69844051
Target partitioning
You can configure partitioning to optimize the mapping performance at run time when you write data to a file of flat format type. You can configure target partitioning only on mappings.
The partition type controls how the agent distributes data among partitions at partition points. You can define the partition type as passthrough partitioning. With partitioning, the Secure Agent distributes rows of target data based on the number of threads that you define as partition.
You can configure the Merge Partition Files options in the advanced target properties. You can specify whether the Secure Agent must merge the number of partition files as a single file or maintain separate files based on the number of partitions specified to write data to the Amazon S3 V2 targets.
If you do not select the Merge Partition Files option, separate files are created based on the number of partitions specified. The file name is appended with a number starting from 0 in the following format: <file name>_<number>
For example, the number of threads for the Region.csv file is three. If you do not select the Merge Partition Files option, the Secure Agent writes three separate files in the Amazon S3 V2 target in the following format:
<Region_0> <Region_1> <Region_2>
If you configure the Merge Partition Files option, the Secure Agent merges all the partitioned files as a single file and writes the file to Amazon S3 V2 target.
Incremental write to partition directory
You can write to a partition directory incrementally for a mapping in advanced mode and append data to the partition directory.
When you do not select the Overwrite File(s) If Exists option in the Amazon S3 V2 target advanced properties, you can write to a partition directory incrementally and append data to the partition directory.
When you create a target at runtime and the file name ends with /, the Secure Agent appends the incoming source partitions to the existing parent directory and does not add a time stamp to the target file name. For an existing target, the Secure Agent appends the incoming source partitions to the existing parent directory and does not add a time stamp to the target file name.
You can override the folder path for a target that you create at runtime. If you specify a file name ending with / and override the folder path, the Secure Agent considers the path as a folder. The file name is not appended with the folder path.
You cannot override the file name for both an existing target or for a target that you create at runtime.
Distribution column
You can write multiple flat files to Amazon S3 target from a single source in a mapping. Configure the Distribution Column option in the advanced target properties.
You can specify one column name in the Distribution Column field to create multiple target files during run time. When you specify the column name, the Secure Agent creates multiple target files in the column based on the column values that you specify in Distribution Column.
Each target file name is appended with the Distribution Column value in the following format:
Each target file contains all the columns of the table including the column that you specify in the Distribution Column field.
For example, the name of the target file is Region.csv that contains the values North America and South America. The following target files are created based on the values in the Distribution Column field:
Region_North America.csv Region_South America.csv
You cannot specify two column names in the Distribution Column field. If you specify a column name that is not present in target field column, the task fails.
When you specify a column that contains value with special characters in the Distribution Column field, the Secure Agent fails to create target file if the corresponding Operating System do not support the special characters.
For example, the Secure Agent fails to create target file if the column contains date value in the following format: YYYY/MM/DD
Writing to multiple target objects
When you import target objects, the Secure Agent appends a FileName field to the imported target object. When you map the FileName field in the target object to an incoming field, the Secure Agent creates the folder structure and the target files based on the FileName field. For example:
The FileName field is applicable to the following file formats:
•Avro
•ORC
•Parquet
Consider the following guidelines when using the target FileName field in mappings:
•Do not map the source object FileName field to the target object FileName field. If you map the FileName field in the target object to an incoming field, the Secure Agent does not create directory structure as expected.
•When you use the FileName field in a target object, the Secure Agent creates folders with the following different names for null values:
- For mappings: _EMPTY_
- For mappings in advanced mode: _HIVE_DEFAULT_PARTITION_
•When you map a date type incoming field to the FileName field in the target object, the Secure Agent creates a nested folder structure based on the incoming date value for target objects.
•When you map an incoming field to the FileName field in the target object, the mapping runs successfully for the first time. At subsequent runs, the mapping fails with the following error:
Operation failed: Index: 0, Size: 0.
To successfully rerun the mapping, use a dummy target file at design time and override the dummy target file in advanced target properties.
Object tag
You can add a tag to the object stored on the Amazon S3 bucket. Each tag contains a key value pair. You can use an object tag for flat files.
Tagging an object helps to categorize the storage. You can add the object tags in the Object Tags field under the advanced target properties. Enter the object tag in the Key=Value format. You can also enter multiple object tags in the following format:
key1=Value1;key2=Value2
You can either enter the key value pairs or the specify the file path that contains the key value pairs. For example, you can specify the file path in the C:\object\tags.txt format. You can specify any file path on which the Secure Agent is installed.
When you upload new objects in the Amazon S3 bucket, you can add tags to the new objects or add tags to the existing objects. If the Secure Agent overrides a file that contains a tag in the Amazon S3 bucket, the tag is not retained. You must add a new tag for the overridden file. If you upload multiple files to the Amazon S3 bucket, each file that you upload must have the same set of tags associated with the multiple objects.
To add tags in the Amazon S3 V2 target object, you must add the s3:PutObjectTagging permission in the Amazon S3 policy. Following is the sample policy: