Databricks metadata collector
The Databricks metadata collector provides read-only access to metadata from an external Databricks account.
The collector harvests metadata from data assets in Databricks Hive Metadata, Unity Catalog (including Delta Lake), Workflows, and Notebooks.
Metadata cataloged
The Databricks collector catalogs the following information.
|
Object |
Information cataloged |
|---|---|
|
Server |
Host |
|
Columns |
Name, Description, JDBC type, Column Type, Is Nullable, Default Value, Column size, Column index Extended metadata: Tags Note: Deprecated columns and any lineage related to these deprecated columns are not cataloged. |
|
Table |
Name, Description, Schema, Primary key, Foreign key Extended metadata: Tags, Owner, Type, Creation date, Last Modified, Location, Provider, Version, Size, File Count, Partition Columns, Properties |
|
Model |
Name, Owner, Description, Created By, Created At, Last Modified By, Last Modified At, Securable Kind, Securable Type |
|
Views |
Name, Description, SQL definition, Tags |
|
Schema |
Name Extended metadata: Tags |
|
Database |
Type, Name, Server, Port, Environment, JDBC URL Extended metadata: Tags |
|
Notebook |
Notebook ID, Path, Language Type (SQL, Python, Scala, R) |
|
Function |
Name, Description, Function Type |
|
Job |
Title, Description, Creator, Created At, Job run as, Format, Max Concurrent Runs, Notification On Start, Timeouts (sec), Notification On Success, Schedule, Git Source, Notification on Failure, Tags, List of tasks, List of clusters |
|
Cluster |
Name, Description, Node Type ID, Driver Node Type ID, Spark Version, Number of Workers, Autoscale Max Workers, Autoscale Min Workers, AWS Attributes, Tags |
|
Task |
Task Key, Type of Task (Notebook, dbt, Spark jar, Python script, Python wheel, Pipeline task, SQL), Task timeout, Retry interval, Cluster used by the task, Max retries, Depends on, Libraries, Notifications (On start, On success, On failure), Notebook File Path, Notebook Source, Notebook Parameters, Spark Jar Main Class Name, Spark Jar Parameters, Python Script File path, Python Script Parameters, Spark Submit Parameters, Pipeline ID, Pipeline Full Refresh, Python Wheel Package Name, Python Wheel Entry Point, Python Wheel Parameters, SQL Warehouse, SQL Query ID, SQL Dashboard ID, SQL Alert ID, Dbt Project Directory, Dbt Profiles Directory, Dbt warehouse, Dbt catalog, Dbt schema, Dbt commands |
|
Pipeline |
ID, Name, Pipeline Continuous, Run As Username, Creator Username, Last Modified, Pipeline State, Pipeline Type, Pipeline Libraries (file path, notebook path), Pipeline Notifications (email recipient, alert type)Note: Pipeline State, Pipeline Libraries, and Pipeline Notifications are only cataloged if the connected account has direct access to the pipeline. If not, the pipeline is still cataloged without those three fields, and a warning is logged. |
|
External Location |
Name, External URL, Description, Data Source Type, Created Date, Created By, Owner |
|
Storage Credential |
Name, Description, Credential, Created Date, Created By, Owner |
|
Volume |
Name, Description, Type, Owner, Created By, Created At, Last Modified By, Last Modified At, Metastore ID |
|
Materialized View |
Name, SQL Definition, Created, Last Modified |
|
Metric View |
Name, Description, YAML Definition, Source Table, Source Table Type, Filter, Created, Last Modified, Materialization Enabled |
|
Unity Catalog Metastore |
Metastore ID, Name, Region, Cloud, External Access Enabled, Predictive Optimization Enabled, Global Metastore ID, Delta Sharing Scope, Storage Root, Owner, Created By, Last Modified By, Created At, Updated At |
|
Metric View Dimension |
Name, SQL Expression |
|
Metric View Measure |
Name, SQL Expression |
|
Object |
Information Cataloged |
|---|---|
|
Row Filter Access Control |
Name |
|
Column Mask Access Control |
Name |
|
Attribute Based Access Control |
Name, Description, Created by, Created at, Modified by, Modified at, On securable type, For securable type, To principals, Except principals |
|
Workspace bindings |
Workspace ID, Binding type |
|
Privileges |
Granted to, Granted by, Privilege type, Granted on object, Inherited from |
The harvested metadata includes catalog pages for the following data asset types. Each catalog page has a relationship to the other related data asset types.
|
Data asset page |
Relationships |
|---|---|
|
Table |
Columns contained in Table |
|
Schema |
|
|
Database |
|
|
Server |
Database hosted on this server |
|
Columns |
|
|
Table Indexes |
Columns |
|
Job |
|
|
Cluster |
|
|
Task |
|
|
Notebook |
|
|
Folder |
|
|
External Location |
|
|
Storage Credential |
|
|
Model |
|
|
Volume |
|
|
Materialized View |
|
|
Metric View |
|
|
Pipeline |
Note: Ingested data is only available if the connected account has direct access to the pipeline. |
|
Unity Catalog Metastore |
Databases contained in metastore |
|
Row Filter Access Control |
|
|
Column Mask Access Control |
|
|
Attribute Based Access Control |
|
|
Catalog |
|
The following lineage information is collected by the Databricks collector.
Note: The collector does not support lineage for SQL statements defined via variable statements.
|
Object |
Lineage available |
|---|---|
|
Column in view |
The collector identifies the associated column in an upstream view or table for both Hive metastore and Unity Catalog: - Where the data is sourced from
Note: Deprecated columns and any lineage related to these deprecated columns are not cataloged. |
|
Notebook |
Tasks that reference Notebook. (Only if Databricks Unity Catalog is enabled). |
|
Table |
|
|
Databricks table/ Database Schema |
Sourced data from Database Table/Database Schema |
|
Metric View |
Identifies the table a metric view sources its data from. Always available, even if the metric view has never been queried |
The Databricks collector supports Personal access token authentication and Oauth service principal authentication.
- Prepare to run the Databricks collector
Complete authentication and permission configuration tasks before running the Databricks metadata collector. - Create a Databricks metadata collector
Create a collector to import metadata from Databricks.
Parent Topic:Configuring metadata collectors
--- title: Databricks metadata collector description: The Databricks metadata collector provides read-only access to metadata from an external Databricks account. locale: en-US canonical_url: https://www.servicenow.com/docs/r/integrate-applications/databricks-metadata-collector.html release: brazil topic_type: concept last_updated: "2026-09-10" reading_time_minutes: 5 breadcrumb: [Configuring metadata collectors, Data Catalog, Workflow Data Fabric] --- # Databricks metadata collector The Databricks metadata collector provides read-only access to metadata from an external Databricks account. The collector harvests metadata from data assets in Databricks Hive Metadata, Unity Catalog \(including Delta Lake\), Workflows, and Notebooks. ## Metadata cataloged The Databricks collector catalogs the following information. <table id="table_bqt_dpz_qhc"><thead><tr><th> Object </th><th> Information cataloged </th></tr></thead><tbody><tr><td> Server </td><td> Host </td></tr><tr><td> Columns </td><td> Name, Description, JDBC type, Column Type, Is Nullable, Default Value, Column size, Column index Extended metadata: Tags **Note:** Deprecated columns and any lineage related to these deprecated columns are not cataloged. </td></tr><tr><td> Table </td><td> Name, Description, Schema, Primary key, Foreign key Extended metadata: Tags, Owner, Type, Creation date, Last Modified, Location, Provider, Version, Size, File Count, Partition Columns, Properties </td></tr><tr><td> Model </td><td> Name, Owner, Description, Created By, Created At, Last Modified By, Last Modified At, Securable Kind, Securable Type </td></tr><tr><td> Views </td><td> Name, Description, SQL definition, Tags </td></tr><tr><td> Schema </td><td> Name Extended metadata: Tags </td></tr><tr><td> Database </td><td> Type, Name, Server, Port, Environment, JDBC URL Extended metadata: Tags </td></tr><tr><td> Notebook </td><td> Notebook ID, Path, Language Type \(SQL, Python, Scala, R\) </td></tr><tr><td> Function </td><td> Name, Description, Function Type </td></tr><tr><td> Job </td><td> Title, Description, Creator, Created At, Job run as, Format, Max Concurrent Runs, Notification On Start, Timeouts \(sec\), Notification On Success, Schedule, Git Source, Notification on Failure, Tags, List of tasks, List of clusters </td></tr><tr><td> Cluster </td><td> Name, Description, Node Type ID, Driver Node Type ID, Spark Version, Number of Workers, Autoscale Max Workers, Autoscale Min Workers, AWS Attributes, Tags </td></tr><tr><td> Task </td><td> Task Key, Type of Task \(Notebook, dbt, Spark jar, Python script, Python wheel, Pipeline task, SQL\), Task timeout, Retry interval, Cluster used by the task, Max retries, Depends on, Libraries, Notifications \(On start, On success, On failure\), Notebook File Path, Notebook Source, Notebook Parameters, Spark Jar Main Class Name, Spark Jar Parameters, Python Script File path, Python Script Parameters, Spark Submit Parameters, Pipeline ID, Pipeline Full Refresh, Python Wheel Package Name, Python Wheel Entry Point, Python Wheel Parameters, SQL Warehouse, SQL Query ID, SQL Dashboard ID, SQL Alert ID, Dbt Project Directory, Dbt Profiles Directory, Dbt warehouse, Dbt catalog, Dbt schema, Dbt commands </td></tr><tr><td> Pipeline </td><td> ID, Name, Pipeline Continuous, Run As Username, Creator Username, Last Modified, Pipeline State, Pipeline Type, Pipeline Libraries \(file path, notebook path\), Pipeline Notifications \(email recipient, alert type\)**Note:** Pipeline State, Pipeline Libraries, and Pipeline Notifications are only cataloged if the connected account has direct access to the pipeline. If not, the pipeline is still cataloged without those three fields, and a warning is logged. </td></tr><tr><td> External Location </td><td> Name, External URL, Description, Data Source Type, Created Date, Created By, Owner </td></tr><tr><td> Storage Credential </td><td> Name, Description, Credential, Created Date, Created By, Owner </td></tr><tr><td> Volume </td><td> Name, Description, Type, Owner, Created By, Created At, Last Modified By, Last Modified At, Metastore ID </td></tr><tr><td> Materialized View </td><td> Name, SQL Definition, Created, Last Modified </td></tr><tr><td> Metric View </td><td> Name, Description, YAML Definition, Source Table, Source Table Type, Filter, Created, Last Modified, Materialization Enabled </td></tr><tr><td> Unity Catalog Metastore </td><td> Metastore ID, Name, Region, Cloud, External Access Enabled, Predictive Optimization Enabled, Global Metastore ID, Delta Sharing Scope, Storage Root, Owner, Created By, Last Modified By, Created At, Updated At </td></tr><tr><td> Metric View Dimension </td><td> Name, SQL Expression </td></tr><tr><td> Metric View Measure </td><td> Name, SQL Expression </td></tr></tbody> </table>Following additional information is cataloged when you run the collector with the **Enable Governance Metadata Collection** option. <table id="table_governance_policies"><thead><tr><th> Object </th><th> Information Cataloged </th></tr></thead><tbody><tr><td> Row Filter Access Control </td><td> Name </td></tr><tr><td> Column Mask Access Control </td><td> Name </td></tr><tr><td> Attribute Based Access Control </td><td> Name, Description, Created by, Created at, Modified by, Modified at, On securable type, For securable type, To principals, Except principals </td></tr><tr><td> Workspace bindings </td><td> Workspace ID, Binding type </td></tr><tr><td> Privileges </td><td> Granted to, Granted by, Privilege type, Granted on object, Inherited from </td></tr></tbody> </table>## Relationships between objects The harvested metadata includes catalog pages for the following data asset types. Each catalog page has a relationship to the other related data asset types. <table id="table_dqf_nxl_rhc"><thead><tr><th> Data asset page </th><th> Relationships </th></tr></thead><tbody><tr><td> Table </td><td> - Columns contained in Table - Table Indexes - Has privileges Columns contained in Table </td></tr><tr><td> Schema </td><td> - Database that contains Schema - Table that is part of Schema - Has privileges </td></tr><tr><td> Database </td><td> - Schema contained in Database - Server that this database is hosted on </td></tr><tr><td> Server </td><td> Database hosted on this server </td></tr><tr><td> Columns </td><td> - Table Indexes - Table containing Column </td></tr><tr><td> Table Indexes </td><td> Columns </td></tr><tr><td> Job </td><td> - Clusters used by tasks in Job - Tasks contained within Job </td></tr><tr><td> Cluster </td><td> - Cluster contained in job - Task using Cluster </td></tr><tr><td> Task </td><td> - Job containing Task - Cluster used by Task - Tasks depending on Task </td></tr><tr><td> Notebook </td><td> - Folder containing Notebook - Task sourcing data from Notebook </td></tr><tr><td> Folder </td><td> - Folders contained in Folder - Notebooks contained in Folder </td></tr><tr><td> External Location </td><td> - Uses storage credential - Connects to datasource \(S3 bucket, S3 Object, Azure container or Azure blob\) - Has workspace bindings </td></tr><tr><td> Storage Credential </td><td> - Has workspace bindings - Used by External Location </td></tr><tr><td> Model </td><td> - Registered in schema - Stored in data assets \(S3 Bucket, S3 Object\) </td></tr><tr><td> Volume </td><td> - Contained within schema - Stored in data assets \(S3 Bucket, S3 Object\) </td></tr><tr><td> Materialized View </td><td> - Schema that contains Materialized Views - Columns that are part of Materialized Views </td></tr><tr><td> Metric View </td><td> - Schema that contains Metric Views - Columns that are part of Metric Views </td></tr><tr><td> Pipeline </td><td> - Copies data to Databricks Table/Database Schema - Ingests data from Database Table/Database Schema **Note:** Ingested data is only available if the connected account has direct access to the pipeline. </td></tr><tr><td> Unity Catalog Metastore </td><td> Databases contained in metastore </td></tr><tr><td> Row Filter Access Control </td><td> - Applies to table - Uses function - Using column - Contained within schema </td></tr><tr><td> Column Mask Access Control </td><td> - Applies to column - Uses function - Contained within schema </td></tr><tr><td> Attribute Based Access Control </td><td> - Applies to catalog, schema and table - Defined on catalog, schema and table - Uses function </td></tr><tr><td> Catalog </td><td> - Has workspace bindings - Has privileges </td></tr></tbody> </table>## Lineage for Databricks The following lineage information is collected by the Databricks collector. **Note:** The collector does not support lineage for SQL statements defined via variable statements. <table id="table_esh_q54_13c"><thead><tr><th> Object </th><th> Lineage available </th></tr></thead><tbody><tr><td> Column in view </td><td> The collector identifies the associated column in an upstream view or table for both Hive metastore and Unity Catalog: - Where the data is sourced from - That sort the rows via ORDER BY - That filter the rows via WHERE/HAVING - That aggregate the rows via GROUP BY **Note:** Deprecated columns and any lineage related to these deprecated columns are not cataloged. </td></tr><tr><td> Notebook </td><td> Tasks that reference Notebook. \(Only if Databricks Unity Catalog is enabled\). </td></tr><tr><td> Table </td><td> - The collector identifies the upstream and downstream tables and their external locations \(S3 and ADLS Gen2 data assets\) along with the intermediate Job. - The collector harvests the Databricks table lineage to S3 object. </td></tr><tr><td> Databricks table/ Database Schema </td><td> Sourced data from Database Table/Database Schema </td></tr><tr><td> Metric View </td><td> Identifies the table a metric view sources its data from. Always available, even if the metric view has never been queried </td></tr></tbody> </table>## Authentication supported The Databricks collector supports Personal access token authentication and Oauth service principal authentication. - **[Prepare to run the Databricks collector](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/brazil/markdown/integrate-applications/prepare-to-run-the-databricks-collector.md)** Complete authentication and permission configuration tasks before running the Databricks metadata collector. - **[Create a Databricks metadata collector](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/brazil/markdown/integrate-applications/create-databricks-metadata-collector.md)** Create a collector to import metadata from Databricks. **Parent Topic:**[Configuring metadata collectors](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/brazil/markdown/integrate-applications/configure-metadata-collectors-dc.md)
| title | Databricks metadata collector |
| description | The Databricks metadata collector provides read-only access to metadata from an external Databricks account. |
| locale | en-US |
| canonical_url | https://www.servicenow.com/docs/r/integrate-applications/databricks-metadata-collector.html |
| release | brazil |
| topic_type | concept |
| last_updated | 2026-09-10 |
| reading_time_minutes | 5 |
| breadcrumb | Configuring metadata collectorsData CatalogWorkflow Data Fabric |