MapRStreams Data Objects
A MapRStreams data object is a physical data object that represents data in a MapR Stream. After you create a MapRStreams connection, create a MapRStreams data object to write data to MapR Streams.
Before you create and use MapR Stream data objects in Streaming mappings, complete the required prerequisites.
For more information about the prerequisite tasks, see the Informatica Big Data Management Cluster Integration Guide.
When you write data to a MapR stream, specify the name of the stream that you publish to.
After you create a MapRStreams data object, create a write data object operation. You can then add the data object write operation as a target in Streaming mappings.
When you configure the data operation properties, specify the format in which the MapR Streams data object writes data. You can specify XML, JSON, or Avro as format. When you specify XML format, you must provide an XSD file. When you specify Avro format, provide a sample Avro schema in a .avsc file. When you specify JSON, you must provide a sample file.
You can pass any payload format directly from source to target in Streaming mappings. You can project columns in binary format pass a payload from source to target in its original form or to pass a payload format that is not supported.
Streaming mappings can read, process, and write hierarchical data. You can use array, struct, and map complex data types to process the hierarchical data. You assign complex data types to ports in a mapping to flow hierarchical data. Ports that flow hierarchical data are called complex ports.
For more information about processing hierarchical data, see the Informatica Big Data Management User Guide.
MapRStreams Object Overview Properties
Overview properties include general properties that apply to the MapRStreams data object. The Developer tool displays overview properties in the Overview view.
General
Property | Description |
|---|
Name | Name of the MapRStreams data object. |
Description | Description of the MapRStreams data object. |
Native Name | Native name of the MapR Stream. |
Path Information | The path of the MapR Stream. |
Column Properties
The following table describes the column properties that you configure for MapRStreams data objects:
Property | Description |
|---|
Name | The name of the MapRStreams data object. |
Native Name | The native name of the MapRStreams data object. |
Type | The native data type of the MapRStreams data object. |
Precision | The maximum number of significant digits for numeric data types, or the maximum number of characters for string data types. |
Scale | The scale of the data type. |
Description | The description of the MapRStreams data object. |
Access Type | The access type of the port or column. |
MapRStreams Data Object Write Operation Properties
The Data Integration Service uses write operation properties when it writes data to MapR Streams.
General Properties
The Developer tool displays general properties for MapR Stream targets in the Write view.
The following table describes the general properties that you view for MapR Stream targets:
Property | Description |
|---|
Name | The name of the MapR Stream target. This property is read-only. |
Description | The description of the MapR Stream target. |
Port Properties
Ports properties for a physical data object include port names and port attributes such as data type and precision.
The following table describes the ports properties that you configure for MapR Stream targets:
Property | Description |
|---|
Name | The name of the target. |
Type | The native data type of the target. |
Precision | The maximum number of significant digits for numeric data types, or the maximum number of characters for string data types. |
Scale | The scale of the data type. |
Description | The description of the target. |
Target Properties
The targets properties list the targets of the MapRStreams data object.
The following table describes the sources property that you can configure for MapR Stream targets:
Property | Description |
|---|
Target | The target which the MapRStreams data object writes to. You can add or remove targets. |
Run-time Properties
The run-time properties displays the name of the connection.
The run-time property for MapR Stream target includes the name of the MapRStream connection.
Column Projection Properties
The following table describes the columns projection properties that you configure for MapR Stream targets:
Property | Description |
|---|
Column Name | The name field that contains data. This property is read-only. |
Type | The native data type of the resource. This property is read-only. |
Enable Column Projection | Indicates that you use a schema to write data to the target. By default, the data is streamed in binary format. To change the format in which the data is streamed, select this option and specify the schema format. |
Schema Format | The format in which the source streams data. You can select one of the following formats: |
Schema | Specify the XSD schema for the XML format, the sample JSON for the JSON format, or the sample .avsc file for the Avro format. |
Column Mapping | The mapping of source data to the data object. Click View to see the mapping. |
Project Column as Complex Data Type | Project columns as complex data type for hierarchical data. For more information, see the Informatica Big Data Management User Guide. |