User Guide > Sources in a Streaming Mapping > Azure EventHub Data Objects

Azure EventHub Data Objects

An Azure EventHub data object is a physical data object that represents an event hub object in Microsoft Azure Event Hubs data streaming platform and event ingestion service. After you create an Azure Eventhub connection, create an Azure Eventhub data object to read from event data from Azure Event Hubs.

Azure Event Hubs is a highly scalable data streaming platform and event ingestion service, that receives and processes events. Event Hubs can process and store events or data produced by distributed software and devices.

When you configure the Azure EventHub data object, specify the name of the event hub that you read from. After you create the data object, create a read operation to read event data from an Event Hub. You can then add the data object read operation as a source in streaming mappings.

When you configure the data operation properties, specify the format in which the Azure EventHub data object reads data. You can specify XML, JSON, Avro, or flat as format. When you specify XML format, you must provide a XSD file. When you specify Avro format, provide a sample Avro schema in a .avsc file. When you specify JSON or Flat format, you must provide a sample file.

You can pass any payload format directly from source to target in Streaming mappings. You can project columns in binary format pass a payload from source to target in its original form or to pass a payload format that is not supported.

Streaming mappings can read, process, and write hierarchical data. You can use array, struct, and map complex data types to process the hierarchical data. You assign complex data types to ports in a mapping to flow hierarchical data. Ports that flow hierarchical data are called complex ports.

For more information about processing hierarchical data, see the Informatica Big Data Management User Guide.

Related Topics:

Creating a Data Object

Azure EventHub Data Object Overview Properties

Overview properties include general properties that apply to the Azure EventHub data object. The Developer tool displays overview properties of the data object in the Overview view.

You can configure the following overview properties for Azure EventHub data objects:

General: You can configure the following general properties for the Azure EventHub data object:
Column: You can configure the name, native name, data type, precision, scale, and description of the columns in the Azure EventHub resource.
Advanced

Azure Eventhub Data Object Read Operation Properties

The Data Integration Service uses read operation properties when it reads data from an Azure Event Hub.

General Properties

The Developer tool displays general properties for Azure Event Hub sources in the Read view.

The following table describes the general properties for the Azure EventHub data object read operation:

Property	Description
Name	The name of the Azure EventHub data object This property is read-only. You can edit the name in the Overview view. When you use the Azure EventHub as a source in a mapping, you can edit the name in the mapping.
Description	The description of the Azure EventHub data object operation.

Ports Properties

Ports properties for a physical data object include port names and port attributes such as data type and precision.

The following table describes the ports properties that you configure for Azure EventHub sources:

Property	Description
Name	The name of the source.
Type	The native data type of the source.
Precision	The maximum number of significant digits for numeric data types, or the maximum number of characters for string data types.
Detail	The detail of the data type.
Scale	The scale of the data type.
Description	The description of the resource.

Sources Properties

The sources properties list the resources of the Azure Eventhub data object.

The following table describes the sources property that you can configure for Azure Event Hub events:

Property	Description
Sources	The sources which the Azure Eventhub data object reads from. You can add or remove sources.

Run-time Properties

The run-time properties include properties that the Data Integration Service uses when reading data from the source at run time.

The run-time property for Azure Event Hub event includes the name of the Azure Eventhub connection.

Advanced Properties

The following table describes the advanced properties for Azure Event Hub sources:

Property	Description
Operation Type	Specifies the type of data object operation. This is a read-only property.
Consumer Group	The name of the Event Hub Consumer Group that you read events from.
Max Rate	The maximum number of events that are consumed in a single batch for each partition.
Shared Access Policy Name	The name of the Event Hub Shared Access Policy. To read from Event Hubs, you must have Listen permission. If you specify a value for this property, it overwrites the value configured in the Azure EventHub connection.
Shared Access Policy Primary Key	The primary key of the Event Hub Shared Access Policy. If you specify a value for this property, it overwrites the value configured in the Azure EventHub connection.
Guaranteed Processing	Guaranteed processing ensures that the mapping processes messages published by the sources and delivers them to the targets at least once. In the event of a failure, there could be potential duplicates but the messages are processed successfully. If the external source or the target is not available, the mapping execution stops to avoid any data loss. Select this option for guaranteed delivery of data streamed from the Azure Event Hub.
Start Position Offset	The time from which the Azure EventHub data object starts reading events from an Event Hub. You can select one of the following options: - CUSTOM. Read messages from a specific time. - EARLIEST Read the earliest messages available on the event hub.
Custom Start Position Timestamp	A UTC timezone datetime value in ISO8601 format from which the Azure EventHub data object starts reading events from an Event Hub. Specify time in the following format: YYYY-MM-DDThhmmss.sssZ
Partition Count	The number of partitions that the Event Hub has. The source reads only from the number of partitions you specify. If the number of partitions on the Event Hub changes, you must change this value to ensure that event from all partitions are consumed.
Consumer Properties	The Event Hub consumer configuration properties. Specify properties as key-value pairs. For example, key1=value1,key2=value2

Column Projections Properties

The following table describes the columns projection properties that you configure for Azure Event Hub sources:

Property	Description
Column Name	The name field that contains data. This property is read-only.
Type	The native data type of the resource. This property is read-only.
Enable Column Projection	Indicates that you use a schema to read the data that the source streams. By default, the data is streamed in binary format. To change the format in which the data is processed, select this option and specify the schema format.
Schema Format	The format in which the source processes data. You can select one of the following formats: - XML - JSON - Flat - Avro
Schema	Specify the XSD schema for the XML format, the sample JSON for the JSON format. Specify a .avsc file for the Avro format.
Column Mapping	The mapping of source data to the data object. Click View to see the mapping.
Project Column as Complex Data Type	Project columns as complex data type for sources with hierarchical data. Select this option if the source has hierarchical data. For more information on hierarchical data, see the Informatica Big Data Management User Guide.