You can use the Kafka catalog source to extract metadata from a Kafka source system.
Kafka is a distributed event streaming platform for data pipelines, streaming analytics, data integration, and mission-critical applications.
Configure a Kafka catalog source to extract metadata from Apache Kafka, Confluent Platform, and Confluent Cloud source systems.
Extracted metadata
You can extract specific metadata from a Kafka source system with a Kafka catalog source.
Metadata Command Center extracts the following objects from a Kafka source system:
•Cluster
•Topic
•Field
•Message field
Metadata Command Center extracts field metadata based on whether a schema registry is configured for the connection. A schema registry centralizes schema storage, management, and version control for data messages in a streaming platform like Kafka.
Metadata Command Center extracts field metadata in the following ways:
•With schema registry: Applies to Confluent Cloud. Extracts field metadata from the defined topic schema in Apache Avro, JSON Schema, and Protobuf formats.
•Without schema registry: Extracts field metadata after parsing messages.
You can extract fields from up to 30 messages per catalog source job in the following data formats:
- Confluent Cloud: JSON, XML, and CSV
- Confluent Platform: JSON, XML, and CSV
- Apache Kafka: JSON, XML, and CSV
Compatible connectors
Before you configure a Kafka catalog source, you must connect to the Kafka source system.
Use the Kafka connector to connect to the Kafka source system.
For information about configuring a connection, see Connections.