Microsoft Azure Data Lake Store Data Object
A Microsoft Azure Data Lake Store data object is a physical data object that represents data in a Microsoft Azure Data Lake Store table. After you create an Azure Data Lake Store connection, create a Microsoft Azure Data Lake store data object write operation to write to a Microsoft Azure Data Lake Store table.
You can use Microsoft Azure Data Lake Store to store data irrespective of size, structure, and format. Use Microsoft Azure Data Lake Store to process large volumes of data to achieve faster business outcomes.
When you configure the data operation properties, specify the format in which the data object writes data. You can specify XML, JSON, or Avro as format. When you specify XML format, you must provide an XSD file. When you specify Avro format, provide a sample Avro schema in a .avsc file. When you specify JSON, you must provide a sample file.
You cannot run a mapping with a Microsoft Data Lake Store data object on a MapR distribution.
You can pass any payload format directly from source to target in Streaming mappings. You can project columns in binary format pass a payload from source to target in its original form or to pass a payload format that is not supported.
Streaming mappings can read, process, and write hierarchical data. You can use array, struct, and map complex data types to process the hierarchical data. You assign complex data types to ports in a mapping to flow hierarchical data. Ports that flow hierarchical data are called complex ports.
For more information about processing hierarchical data, see the Informatica Big Data Management User Guide.
Microsoft Azure Data Lake Store Data Object Properties
The Microsoft Azure Data Lake Store Overview view displays general information about the Microsoft Azure Data Lake Store data object and the object properties that apply to the Microsoft Azure Data Lake Store table you import.
General Properties
You can configure the following properties for a Microsoft Azure Data Lake Store data object:
- •Name. Name of the Microsoft Azure Data Lake Store data object.
- •Description. Description of the Microsoft Azure Data Lake Store data object.
- •Connection. Name of the Microsoft Azure Data Lake Store connection.
Microsoft Azure Data Lake Store Data Object Write Operation Properties
The Data Integration Service writes data to a Microsoft Azure Data Lake Store object based on the data object write operation. The Developer tool displays the data object write operation properties for the Microsoft Azure Data Lake Store data object in the Data Object Operation section.
You can view the data object write operation from the Input and Target properties.
Input properties
Represent data that the Data Integration Service reads from a Microsoft Azure Data Lake Store directory server. Select the input properties to edit the port properties and specify the advanced properties of the data object write operation.
Target properties
Represent data that the Data Integration Service writes to Microsoft Azure Data Lake Store. Select the target properties to view data, such as the name and description of the Microsoft Azure Data Lake Store object.
Input Properties of the Data Object Write Operation
Input properties represent data that the Data Integration Service writes to a Microsoft Azure Data Lake Store directory server. Select the input properties to edit the port properties of the data object write operation. You can also specify advanced data object write operation properties to write data to Microsoft Azure Data Lake Store objects.
The input properties of the data object write operation include general properties that apply to the data object write operation. Input properties also include port, source, and advanced properties that apply to the data object write operation.
You can view and change the input properties of the data object write operation from the General, Ports, Targets, run-time, and Advanced tabs.
Ports Properties - Input Write
The input ports properties list the data types, precision, and scale of the data object write operation.
The following table describes the input ports properties that you must configure in the data object write operation:
Property | Description |
|---|
Name | Name of the port. |
Type | Data type of the port. |
Precision | Maximum number of significant digits for numeric data types, or maximum number of characters for string data types. For numeric data types, precision includes scale. |
Scale | Maximum number of digits after the decimal point for numeric values. |
Description | Description of the port. |
Run-time Properties
The run-time properties display the name of the connection used for write transformation.
The following table describes the run-time properties that you configure for a Microsoft Azure Data Lake Store write operation:
Property | Description |
|---|
Connection | Name of the Microsoft Azure Data Lake Store connection. |
Advanced Properties
You can use the advanced properties to specify data object write operation properties to write data to a Microsoft Azure Data Lake Store server.
The following table describes the advanced properties that you configure for a Microsoft Azure Data Lake Store write operation:
Property | Description |
|---|
Tracing Level | By default, the tracing level for every transformation is Normal. Change the tracing level to a Verbose setting when you need to troubleshoot a transformation that is not behaving as expected. Set the tracing level to Terse when you want the minimum amount of detail to appear in the log. |
Maintain row order | NA |
If file exists | The streaming mapping ignores this value. A new target file is created. |
Column Projection Properties
The Developer tool displays the column projection properties in the Properties view of the write operation.
To specify column projection properties, double click on the write operation and select the data object. The following table describes the columns projection properties that you configure for Azure Data Lake Store targets:
Property | Description |
|---|
Column Name | The field in the target that the data object writes to. This property is read-only. |
Type | The native data type of the target. This property is read-only. |
Enable Column Projection | Indicates that you use a schema to publish the data to the target. By default, the data is streamed in binary format. To change the streaming format, select this option and specify the schema format. |
Schema Format | The format in which you stream data to the target. You can select one of the following formats: |
Schema | Specify the XSD schema for the XML format, asample file for JSON, or .avsc file for Avro format. |
Column Mapping | The mapping of data object to the target. Click View to see the mapping. |
Project as Hierarchical Type | Project columns as complex data type for hierarchical data. For more information, see the Informatica Big Data Management User Guide. |