Databricks combines data warehouses and data lakes into an AI-driven Databricks Lakehouse platform. Databricks Delta Lake is an open source data format and a transactional data management system on the Databricks platform. Databricks Notebooks is a web-based interface to a document that contains code that you can run, visualizations, and narrative text.
You can use the Databricks catalog source to extract metadata from both Databricks Delta Lake and Databricks Notebooks source systems. You can run connection-aware scans on Databricks sources.
You can use SQL warehouse or all-purpose clusters to extract metadata.
You can extract metadata from Databricks Unity Catalog. Additionally, you can retrieve lineage captured by Databricks Unity Catalog.
Note: Databricks Unity Catalog retains lineage data for 90 days.
Note: To improve wildcard lineage at the directory or file level for Databricks assets, perform connection assignment, and run the Databricks catalog source again. These wildcards can refer to files in Amazon S3 and Microsoft Azure Data Lake Storage Gen2.
Objects extracted
You can extract the following objects from a Databricks Notebooks source system:
•Calculation
•Command
•Folder
•Job Parameter
•Live Table
•Live View
•Notebook Definition
•Notebook Instance
•Notebook Parameter
•Notebook Task
•Pipeline Definition (SQL based)
•Result
•Run Job Task
•Streaming Table
•Streaming View
•Task Parameter
•Workflow Job Definition
•Workflow Job Instance
You can extract the following objects from a Databricks Delta Lake source system:
•File
•File System
•Database
•Schema
•Table
•External Table
•External Column
•Column
- Primary Key
- Foreign Key
You can extract the following complex data type columns along with their nested fields from Databricks Delta Lake source systems:
•Map
•Struct
•Array
Data profiling for Databricks objects
Configure data profiling to run profiles on the metadata extracted from Databricks Delta Lake source systems. You can run profiles on Databricks Delta tables created in all-purpose clusters or SQL warehouse. You can also run profiles on Databricks Unity Catalog objects.
You can run profiles on the following Databricks Delta Lake objects:
•Delta Table
•External Table in Delta format.
•View
You can view the profiling statistics in Data Governance and Catalog. The data profiling task runs profiles on the following data types for Databricks Delta Lake objects:
•Bigint
•Boolean
•Date
•Decimal
•Double
•Float
•Int
•Smallint
•String
•Tinyint
•Timestamp
Sampling type
Determine the sample rows on which you want to run the data profiling task. You can choose one of the following sampling types for a Databricks catalog source:
- All Rows
- Custom Query. Sampling method to specify a percentage of rows on which you want to run the data profiling task. For example, TABLESAMPLE (BUCKET 10 OUT OF 50), TABLESAMPLE (5 PERCENT), LIMIT 2, or TABLESAMPLE (5000 ROWS).
Data lineage
The following lineage data is available for Databricks assets:
•Data set level
•Data element level
You can extract lineage information from the following source systems:
•Databricks Delta Lake
•Amazon S3
•Oracle
•Microsoft Azure Data Lake Storage Gen2
You can extract lineage information from the following technologies, if available in Unity Catalog:
•Python
•SQL
•Scala
•R
For more information about data lineage, see the Working With Assets help.