Amazon S3 V2 Connector > Troubleshooting > Troubleshooting for Amazon S3 V2 Connector
  

Troubleshooting for Amazon S3 V2 Connector

Java heap size configuration

This section describes the errors that you might encounter if the JVM options in the Secure Agent is not configured accordingly to read a large number of files.
"ERROR java.lang.OutOfMemoryError: GC overhead limit exceeded" occurs when you run a Mapping task to write large number of records.
To resolve this issue, perform the following tasks to configure the JVM options in the Secure Agent to increase the memory for the Java heap size:
  1. 1Log in to IDMC.
  2. 2Select Administrator > Runtime Environments.
  3. 3On the Runtime Environments page, select the Secure Agent.
  4. 4On the Secure Agent page, click the Service Configuration tab.
  5. 5On the Service Configuration tab, select Data Integration Server as the service and DTM as the type.
  6. 6Edit the JVM Option property and set the value to -Xms2048m.
    7. Note:
    Specify the maximum and minimum heap size based on the data you want to process.
  7. 7Click Save.
"[ERROR] java.lang.OutOfMemoryError: Java heap space" occurs when you run a mapping to write a file of size 1.4 GB or higher and select Informatica Encryption as the encryption type.
To resolve this issue, perform the following tasks to configure the JVM options in the Secure Agent to increase the memory for the Java heap size:
  1. 1Log in to IDMC.
  2. 2Select Administrator > Runtime Environments.
  3. 3On the Runtime Environments page, select the Secure Agent.
  4. 4On the Secure Agent page, click the Service Configuration tab.
  5. 5On the Service Configuration tab, select Data Integration Server as the service and DTM as the type.
  6. 6Edit the JVM Option property and set the value to -Xmx8046m.
  7. 7Click Save.

FFParserRetainNullString custom property

When you read from a .csv file that contains a string named null, the task does not write any data to the target. To resolve this issue, perform the following tasks and configure the custom property FFParserRetainNullString:
  1. 1Log in to IDMC.
  2. 2Select Administrator > Runtime Environments.
  3. 3On the Runtime Environments page, select the Secure Agent.
  4. 4On the Secure Agent page, click the Service Configuration tab.
  5. 5On the Service Configuration tab, click Custom Configuration.
  6. 6Click Add new row and configure the following properties:
    1. aIn the Service field, select Data Integration Server.
    2. bIn the Type field, select Tomcat JRE.
    3. cIn the Name field, enter FFParserRetainNullString.
    4. dIn the Value field, enter true.
  7. 7Click Save.

RedirectToSessionLog custom property

When you select a source or a target Amazon S3 V2 file and configure the Parquet file format option, a large number of log files might be generated. To resolve this issue and disable the logs, perform the following tasks and configure the custom property RedirectToSessionLog:
  1. 1Log in to IDMC.
  2. 2Select Administrator > Runtime Environments.
  3. 3On the Runtime Environments page, select the Secure Agent.
  4. 4On the Secure Agent page, click the Service Configuration tab.
  5. 5On the Service Configuration tab, click Custom Configuration.
  6. 6Click Add new row and configure the following properties:
    1. aIn the Service field, select Data Integration Server.
    2. bIn the Type field, select DTM.
    3. cIn the Name field, enter RedirectToSessionLog.
    4. dIn the Value field, enter No.
  7. 7Click Save.

Empty struct data in JSON file

If a JSON file has a field with an empty struct data, the Secure Agent ignores the field and reads the remaining fields during metadata read.
For example, if the JSON file has the following data in the first row: {​"id":123,"address":{​}​}, the address field is ignored and does not appear in the Fields tab. If the JSON file has values for the address field in the consecutive row, you can use the Data elements to sample property to fetch this field.

Amazon S3 bucket does not exist or the user does not have permission to access the bucket

Do not modify the time on the machine that hosts the Secure Agent. The time on the Secure Agent must be correct as per the time zone. Otherwise, the mapping fails with an exception.

Connection timeout error

Network performance and reliability issues when connecting to Amazon S3 can cause slow response times, connection failures, or request errors.
These issues might occur due to suboptimal configuration of client-side connection management parameters such as maximum concurrent connections, connection timeout, socket timeout, and acquisition timeout.
To resolve these issues, configure the following properties for the Secure Agent:
Perform the following tasks to configure the JVM options in the Secure Agent:
  1. 1Log in to IDMC.
  2. 2Select Administrator > Runtime Environments.
  3. 3On the Runtime Environments page, select the Secure Agent.
  4. 4On the Secure Agent page, click the Service Configuration tab.
  5. 5On the Service Configuration tab, select Data Integration Server as the service and DTM as the type.
  6. 6Edit and configure the following JVM options:
    7. JVM option
    Value
    JVMOption1
    -DAWSS3MaxConnectionPool=<number of connections>
    JVMOption2
    -DAWSS3ConnectionTimeOut=<duration in milliseconds>
    JVMOption3
    -DAWSS3SocketTimeOut=<duration in milliseconds>
    JVMOption4
    -DAWSS3AcquisitionTimeOut=<duration in milliseconds>
  7. 7Click Save.

Mapping in advanced mode configured to read or write Date and Int96 data types to Avro or Parquet files fails

A mapping in advanced mode configured to read from or write to an Avro or Parquet file fails in the following cases:
To resolve this issue, specify the following spark session properties in the mapping task or in the custom properties file for the Secure Agent:
spark.sql.legacy.timeParserPolicy=LEGACY
spark.sql.parquet.int96RebaseModeInWrite=LEGACY
spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY
spark.sql.parquet.int96RebaseModeInRead=LEGACY
spark.sql.parquet.datetimeRebaseModeInRead=LEGACY
spark.sql.avro.datetimeRebaseModeInWrite=LEGACY
spark.sql.avro.datetimeRebaseModeInRead=LEGACY

Mapping fails with 407 Proxy Authentication Required error when using authenticated proxy with complex files in Amazon S3

When you use an authenticated proxy in a mapping that reads from or writes to complex files, the mapping might fail with the following error if the fs.s3a.proxy.domain property is set in the Hadoop configurations:
software.amazon.awssdk.services.s3.model.S3Exception: (Service: S3, Status Code: 407, Request ID: null) (Service: S3, Status Code: 407, Request ID: null)
To resolve this issue, you must set the value of the disableS3aProxyDomain property to true in the Secure Agent to prevent the fs.s3a.proxy.domain property from being passed to the Hadoop configurations at design time and run time.
Perform the following steps to configure the property in the Secure Agent:
  1. 1Log in to IDMC.
  2. 2Select Administrator > Runtime Environments.
  3. 3On the Runtime Environments page, select the Secure Agent.
  4. 4On the Secure Agent page, click the Service Configuration tab.
  5. 5On the Service Configuration tab, click Custom Configuration.
  6. 6Click Add new row and configure the following properties:
    1. aIn the Service field, select Data Integration Server.
    2. bIn the Type field, select DTM.
    3. cIn the Name field, enter disableS3aProxyDomain.
    4. dIn the Value field, enter true.
  7. 7Click Save.

Mapping that writes to Avro, Parquet, ORC, Delta, or Iceberg files and uses the CRC64NVME or CRC32C checksum algorithm fails

When you write to Avro, Parquet, ORC, Delta, or Iceberg files and use the CRC64NVME or CRC32C checksum algorithm, the mapping fails if you do not have the execute permissions for the temporary directory where the native JNI library is extracted.
If you have the permissions and the mapping runs successfully, the extracted files are not automatically deleted from the temporary directory.
To resolve these issues, pre-extract the native library and add it to the local directory path:
  1. 1Locate the JAR file that ships with the Amazon S3 V2 Connector package.
    2. The file is available in the following location:
    <Secure Agent installation directory>/downloads/package-AmazonS3V2.<version>/package/s3/thirdparty/infa.amazons3/aws-crt-0.33.9.jar
  2. 2Confirm the native library layout for the Secure Agent machine.
    3. The layout uses the following format:
    <os>/<arch>/<cruntime>/<libraryName>
    The native library in the JAR file for Linux x86_64 with glibc is linux/x86_64/glibc/libaws-crt-jni.so.
  3. 3To confirm the architecture and C runtime, run the following command:
    4. echo "os=linux arch=$(uname -m) libc=$(ldd --version 2>&1 | head -1)"
  4. 4To create a local directory on the Secure Agent machine and extract the native library, run the following command:
    5. mkdir -p /opt/infa/native-libs
    cd /opt/infa/native-libs
    unzip -j <INFAAGENT>/downloads/.../aws-crt-0.33.9.jar "linux/x86_64/glibc/libaws-crt-jni.so"
  5. 5To set read and execute permissions on the native library, run the following command:
    6. chmod 555 /opt/infa/native-libs/libaws-crt-jni.so
  6. 6Append the path to the directory created in step 4 to the LD_LIBRARY_PATH environment variable.
    7. To preserve the existing path, append the directory instead of replacing the value:
    export LD_LIBRARY_PATH=/opt/infa/native-libs:$LD_LIBRARY_PATH
  7. 7Restart the Secure Agent.
Override the native library extraction directory
By default, the native library is extracted in the system temporary directory.
To extract the library to a different directory, perform the following steps:
  1. 1Log in to IDMC.
  2. 2Select Administrator > Runtime Environments.
  3. 3On the Runtime Environments page, select the Secure Agent.
  4. 4On the Secure Agent page, click the Service Configuration tab.
  5. 5On the Service Configuration tab, select Data Integration Server as the service and DTM as the type.
  6. 6Edit the JVM Option property and set the value to -Daws.crt.lib.dir=<directory>.
  7. 7Click Save.
If you do not add the extracted library to the path, monitor the specified directory and clean it at regular intervals.

Mapping that reads from and writes to complex files fails if the source and target connections use different credentials with different permissions to access the same bucket

When you run a mapping to read from and write to complex files, the mapping fails if the source and target connections use different credentials to access the same bucket, but the credentials have a different set of permissions.
For example, the source connection has read-only access and the target connection has write-only access to the same folder.
The following error appears:
[ERROR] java.nio.file.AccessDeniedException: User is not authorized to perform: s3:ListBucket on resource: "arn:aws:s3:::<bucket>" because no identity-based policy allows the s3:ListBucket action
To resolve this issue, consider one of the following solutions: