Azure EventHub Data Objects
An Azure EventHub data object is a physical data object that represents data in Microsoft Azure Event Hubs data streaming platform and event ingestion service. After you create an Azure EventHub connection, create an Azure EventHub data object to write to Event hub events.
Azure Event Hubs is a highly scalable data streaming platform and event ingestion service, that receives and processes events. Event Hubs can process and store events or data produced by distributed software and devices.
When you configure the Azure EventHub data object, specify the name of the event that you write to. After you create the data object, create a data object write operation to write data to an Event Hub. You can then add the data object write operation as a target in streaming mappings.
Configure partition keys to separate events into different partitions. Events with the same partition key are sent to the same partition on the event hub. Events that are sent in a batch do not need to have the same partition key.
When you configure the data operation properties, specify the format in which the Azure EventHub data object writes data. You can specify XML, JSON, Avro, or Flat as format. When you specify XML format, you must provide a XSD file. When you specify Avro format, provide a sample Avro schema in a .avsc file. When you specify JSON or Flat format, you must provide a sample file.
You can pass any payload format directly from source to target in Streaming mappings. You can project columns in binary format pass a payload from source to target in its original form or to pass a payload format that is not supported.
Streaming mappings can read, process, and write hierarchical data. You can use array, struct, and map complex data types to process the hierarchical data. You assign complex data types to ports in a mapping to flow hierarchical data. Ports that flow hierarchical data are called complex ports.
For more information about processing hierarchical data, see the Informatica Big Data Management User Guide.
Azure EventHub Data Object Overview Properties
Overview properties include general properties that apply to the Azure EventHub data object. The Developer tool displays overview properties of the data object in the Overview view.
You can configure the following overview properties for Azure EventHub data objects:
- General
- You can configure the following general properties for the Azure EventHub data object:
- - Name. Name of the Azure EventHub data object.
- - Description. Description of the Azure EventHub data object.
- - Native Name. Name of the Azure EventHub data object.
- - Path Information. The path of the data object in Azure EventHub. For example, /EventHubs/avroevents
- Column
- You can configure the name, native name, data type, precision, scale, and description of the columns in the Azure EventHub resource.
- Advanced
The following are the advanced properties for the Azure EventHub data object:
- - Location. The location of the Event Hub.
- - Partition. The number of partitions that the Event Hub has when you import the data object.
- - Date of Creation. The date of creation of the Event Hub.
Azure EventHub Data Object Write Operations
The Data Integration Service uses write operation properties when it writes data to an Azure Event Hub.
General Properties
The Developer tool displays general properties for Azure Event Hub sources in the Write view.
The following table describes the general properties for the Azure EventHub data object write operation:
Property | Description |
|---|
Name | The name of the Azure EventHub data object. This property is read-only. You can edit the name in the Overview view. When you use the Azure Event Hub as a target in a mapping, you can edit the name in the mapping. |
Description | The description of the Azure EventHub data object operation. |
Ports Properties
Ports properties for a physical data object include port names and port attributes such as data type and precision.
The following table describes the ports properties that you configure for Azure EventHub targets:
Property | Description |
|---|
Name | The name of the target. |
Type | The native data type of the target. |
Precision | The maximum number of significant digits for numeric data types, or the maximum number of characters for string data types. |
Detail | The detail of the data type. |
Scale | The scale of the data type. |
Description | The description of the resource. |
Target Properties
The targets properties list the targets of the Azure EventHub data object.
The following table describes the sources property that you can configure for Azure Event Hub targets:
Property | Description |
|---|
Target | The target which the Azure EventHub data object writes to. You can add or remove targets. |
Run-time Properties
The run-time properties include properties that the Data Integration Service uses when writing data to the target at run time, such as reject file names and directories.
The run-time property for Azure Event Hub targets includes the name of the Azure EventHub connection.
Advanced Properties
The following table describes the advanced properties for Azure Event Hub targets:
Property | Description |
|---|
Operation Type | Specifies the type of data object operation. This is a read-only property. |
Batch Size | The batch size of the events. Specify an integer value for the batch size. |
Shared Access Policy Name | The name of the Event Hub Namespace Shared Access Policy. To write to Event Hubs, you must have Send permissions. If you specify a value for this property, it overwrites the value configured in the Azure EventHub connection. |
Shared Access Policy Primary Key | The primary key of the Event Hub Namespace Shared Access Policy. If you specify a value for this property, it overwrites the value configured in the Azure EventHub connection. |
Publisher Properties | The Event Hub publisher configuration properties. Specify properties as key-value pairs. For example, key1=value1,key2=value2 |
Column Projections Properties
The Developer tool displays the column projection properties in the Properties view of the write operation.
To specify column projection properties, double click on the write operation and select the data object. The following table describes the columns projection properties that you configure for Azure EventHub targets:
Property | Description |
|---|
Column Name | The field in the target that the data object writes to. This property is read-only. |
Type | The native data type of the target. This property is read-only. |
Enable Column Projection | Indicates that you use a schema to publish the data to the target. By default, the data is streamed in binary format. To change the streaming format, select this option and specify the schema format. |
Schema Format | The format in which you stream data to the target. You can select one of the following formats: |
Schema | Specify the XSD schema for the XML format or the sample file for JSON or Avro format. For the Flat file format, configure the schema to associate a flat file. |
Column Mapping | The mapping of data object to the target. Click View to see the mapping. |
Project as Hierarchical Type | Project columns as complex data type for hierarchical data. For more information, see the Informatica Big Data Management User Guide. |