Connections > Amazon S3 V2 connection > Amazon S3 V2 source properties
  

Amazon S3 V2 source properties

You can use an Amazon S3 V2 object as a source in a data loader task.
When you create a data loader task to read data from Amazon S3 V2, specify the Amazon S3 V2 source properties and formatting options.
The following table describes the Amazon S3 V2 source properties:
Property
Description
Connection
Name of the source connection.
Select a source connection or create a new connection.
Source Path
The path to the directory that contains the source data.
The path is populated from the folder path connection property but you can edit the path.

Rules and guidelines for primary key and watermark fields

When you read data from Amazon S3, you can manually define the primary key fields and watermark fields. The default values are Primary key fields not required and Watermark fields not required.

Formatting options

When you select the format of an Amazon S3 file, you can configure the formatting options.
The following table describes the formatting options for Avro, Parquet, JSON, and ORC files:
Property
Description
Formatting Type
Specifies the file format that the Amazon S3 V2 Connector uses to read data from Amazon S3 V2.
You can select the following file format types:
  • - None
  • - Avro
  • - Parquet
  • - ORC
  • - JSON. Not applicable.
Select the Format Type as None to read binary files. Default is None.
Schema Source
The schema of the source file.
You can select one of the following options to specify a schema:
  • - Read from data file. Imports the schema from the file in Amazon S3.
  • - Import from Schema File. Imports schema from a schema definition file in your local machine.
Schema File
The schema definition file in the agent machine from where you want to upload the schema.
Data elements to sample
Does not apply to a data loader task.
Memory available to process data
Does not apply to a data loader task.
The following table describes the formatting options for delimited files:
Property
Description
Formatting Type
Specifies the file format that the Amazon S3 V2 Connector uses to read data from Amazon S3 V2.
Select the delimited file format type.
Schema Source
The schema of the source file.
You can select one of the following options to specify a schema:
  • - Read from data file. Imports the schema from the file in Amazon S3.
  • - Import from Schema File. Imports schema from a schema definition file in your local machine.
Schema File
The schema definition file in the agent machine from where you want to upload the schema.
Delimiter
The character used to separate columns of data. You can configure parameters such as comma, tab, colon, semicolon, or others.
You cannot set a tab as a delimiter directly in the Delimiter field. To set a tab as a delimiter, type the tab character in any text editor. Then, copy and paste the tab character in the Delimiter field.
EscapeChar
Character immediately preceding a column delimiter character embedded in an unquoted string, or immediately preceding the quote character in a quoted string.
You can specify a character or \<decimal value>. When you specify \<decimal value>, the agent considers the ASCII character for the decimal value as the escape character.
For example, if you specify \64, the agent considers the ASCII character @.
To ignore the escape character, specify \0.
Qualifier
Quote character that defines the boundaries of data.
You can enter a single quote or double quote.
Qualifier Mode
The qualifier behavior for the target object.
You can select one of the following options:
  • - Minimal. Default mode. Applies qualifier to data that has a delimiter value present in the data.
  • - All. Applies qualifier to all non-empty columns.
  • - Non_Numeric
  • - All_Non_Null
Code Page
The code page that the agent must use to read data.
Amazon S3 V2 Connector supports the following code pages:
  • - MS Windows Latin 1. Select for ISO 8859-1 Western European data.
  • - UTF-8. Select for Unicode and non-Unicode data.
  • - Shift-JIS. Select for double-byte character data.
  • - ISO 8859-15 Latin 9 (Western European).
  • - ISO 8859-2 Eastern European.
  • - ISO 8859-3 Southeast European.
  • - ISO 8859-5 Cyrillic.
  • - ISO 8859-9 Latin 5 (Turkish).
  • - IBM EBCDIC International Latin-1.
Header Line Number
The line number that you want to use as the header when you read data from Amazon S3.
You can also read a file that does not have a header. To read data from a file with no header, enter the value as 0. To read data from a file with a header, enter the value greater than or equal to one.
Default is 1.
First Data Row
The line number from where you want to read data.
Enter a value that is greater or equal to one.
To read data from the header, the value of the Header Line Number and the First Data Row fields should be the same.
Default is 1.
Target Header
Does not apply to a data loader task.
Distribution Column
Does not apply to a data loader task.
Fixed Width File Format
Does not apply to a data loader task.
Retain Escape Character in Data
Does not apply to a data loader task.
Maximum Rows to Preview
Does not apply to a data loader task.
Row Delimiter
Does not apply to a data loader task.