Data Engineering Streaming
Read this section to learn what's new for Data Engineering Streaming in version 10.5.1.
Transformations in Databricks
Effective in version 10.5.1, you can add the following transformations to streaming mappings that run in Databricks environment on Azure and AWS platforms:
- •Address Validator
- •Classifier
- •Parser
- •Standardizer
For more information, see the Data Engineering Streaming 10.5.1 User Guide.
Intelligent Structure Models on Databricks
Effective in version 10.5.1, you can incorporate an intelligent structure model in a Confluent Kafka, Kafka,Azure Event Hubs, and Amazon Kinesis Stream data object for streaming mappings that run on the Databricks Spark engine.
The Databricks Spark engine can process any input type that the model can parse.
For more information about intelligent structure models, see the Data Engineering Streaming 10.5.1 User Guide.