> For the complete documentation index, see [llms.txt](https://docs.seemoredata.io/external-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/permissions-and-data-handling.md).

# Permissions and Data Handling

Review every permission the SeeMore Data service principal holds in Databricks, what SeeMore Data reads with it, and how that data is handled.

This page is the reference for a security review of the Databricks integration. It lists every permission the service principal holds, what SeeMore Data reads with it, and how that data is stored. [Create Databricks Service Principal and Privileges](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md) and [Automated Setup](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/automated-setup.md) explain how to set these permissions up.

### At a glance <a href="#at-a-glance" id="at-a-glance"></a>

| Item                                                                   | SeeMore Data                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Rows in your tables, volumes or files                                  | **Doesn't read.** It asks for no `SELECT` on your catalogs. See [inherited access](#account-users)                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Model prompts and responses                                            | **Doesn't read.** It isn't granted inference tables or payload logs                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Notebook and file contents, secret values                              | **Doesn't read**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| Catalog, schema and table names, owners, comments, tags and view SQL   | **Reads**, on the catalogs you grant `BROWSE` on                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| Column names and types                                                 | **Doesn't read.** They're visible to `BROWSE`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| Cost, usage and prices                                                 | **Reads**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Warehouse, cluster, job and pipeline configuration and runs            | **Reads**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Who ran each query, when, for how long, and its error message          | **Reads**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Full SQL text                                                          | **Reads**, through the `databricks_pii_access` group that the setup adds the service principal to. You can skip that step                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Audit events, with their request parameters                            | **Reads**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| AI request metadata: requester, client IP, user agent, tokens, latency | **Reads**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| MLflow runs, with their parameters, tags and metrics                   | **Reads**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Workspace user names and email addresses                               | **Reads**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Changes to your account                                                | **Only on SQL warehouses you grant** [**Can manage**](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#grant-can-manage) **on.** There, SeeMore Data changes size, cluster limits or auto stop when one of your users saves a change. It doesn't start, stop or delete warehouses, or change their permissions. Without **Can manage**, it changes nothing. **Can monitor** would allow starting a warehouse; SeeMore Data doesn't |

### Permissions <a href="#permissions" id="permissions"></a>

The permissions are grouped by the [level](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#overview) they're granted at: once for the account, once in each region's metastore, and in each workspace.

| Level     | Permission                                                                                                | Granted on                           | What it allows                                                                                                                                                                                                                                           | Needed for                                                                                                                                                                                                                                       |
| --------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Account   | Service principal with an OAuth secret                                                                    | Account                              | Signing in. It holds no account role                                                                                                                                                                                                                     | Everything                                                                                                                                                                                                                                       |
| Account   | `databricks_pii_access` membership (default; you can skip it)                                             | Account group                        | Reading the full text of every query in the account                                                                                                                                                                                                      | Similar-query grouping, query tagging and lineage, and the SQL shown for each query, including live queries                                                                                                                                      |
| Region    | `USE CATALOG`                                                                                             | `system` catalog                     | Reaching the system schemas                                                                                                                                                                                                                              | Everything                                                                                                                                                                                                                                       |
| Region    | `USE SCHEMA` and `SELECT`                                                                                 | The nine system schemas              | Reading the system tables listed [below](#what-seemore-reads)                                                                                                                                                                                            | Everything; `tags`, `serving`, `ai_gateway` and `mlflow` are optional                                                                                                                                                                            |
| Region    | `BROWSE`                                                                                                  | Each catalog you choose              | Metadata only: no row of data                                                                                                                                                                                                                            | The asset catalog                                                                                                                                                                                                                                |
| Region    | Can use                                                                                                   | `seemore_warehouse` in each region   | Running SeeMore Data's reads                                                                                                                                                                                                                             | Everything                                                                                                                                                                                                                                       |
| Workspace | Workspace membership, as a user, with the **Workspace access** and **Databricks SQL access** entitlements | Each workspace you add it to         | Signing in to that workspace and using its SQL warehouses. It doesn't need cluster or pool creation                                                                                                                                                      | Reading through the workspace's `seemore_warehouse`, if it holds one. Live queries, warehouse automation, the user list and AI Services endpoint details for that workspace. Not query history, which is read through the region's SQL warehouse |
| Workspace | Can monitor                                                                                               | Each warehouse you want watched live | Seeing the warehouse, its query history and query profiles. Databricks also lets it start the warehouse and run queries; SeeMore Data does neither                                                                                                       | Real Time Monitoring                                                                                                                                                                                                                             |
| Workspace | Can view                                                                                                  | Each model serving endpoint          | Seeing the endpoint's configuration and state. It can't query or change the endpoint                                                                                                                                                                     | AI Services endpoint details                                                                                                                                                                                                                     |
| Workspace | Can manage (optional)                                                                                     | SQL warehouses you choose            | Starting, stopping, editing and deleting the warehouse, and changing its permissions. SeeMore Data uses it only to read the warehouse's permission list, to confirm it holds the grant, and to change the warehouse's size, cluster limits and auto stop | Editing a warehouse's settings from SeeMore Data. See [Can manage](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#grant-can-manage) |

#### OAuth scopes <a href="#oauth-scopes" id="oauth-scopes"></a>

Scope the OAuth secret to the seven Databricks APIs SeeMore Data calls:

| Scope               | What SeeMore Data uses it for                                                      |
| ------------------- | ---------------------------------------------------------------------------------- |
| `sql`               | Running its reads on the SQL warehouse, and reading and editing warehouse settings |
| `query-history`     | Live queries, for Real Time Monitoring                                             |
| `access-management` | Reading a warehouse's permission list, to confirm it holds Can manage              |
| `scim`              | The workspace's user list                                                          |
| `model-serving`     | Model serving endpoint details, for AI Services                                    |
| `vector-search`     | The list of vector search endpoints, for AI Services                               |
| `unity-catalog`     | The workspace's metastore, cloud and region                                        |

A token issued from a scoped secret can't call any other Databricks API, such as jobs, clusters, notebooks or workspace files. If the secret is missing one of the seven, the connection test names it. A secret with **All APIs** also works, but reaches every API the service principal is allowed to use.

Scopes limit which APIs the secret can call, not what it can read. That is still bounded by the permissions above and the [inherited access](#account-users) below.

SeeMore Data never asks for `SELECT`, `MODIFY` or `ALL PRIVILEGES` on your catalogs, schemas or tables, or for workspace admin, metastore admin or any account role.

#### Inherited access <a href="#account-users" id="account-users"></a>

Like every identity in your account, the service principal is a member of the `account users` group, and of the `users` group of each workspace you add it to. It holds anything you've granted to those groups:

* **Data** granted to `account users`. To see it, run this for each catalog:

  ```sql
  SHOW GRANTS `account users` ON CATALOG <catalog_name>;
  ```
* **Workspace objects open to all users**, such as the `/Workspace/Shared` folder, where Databricks gives every user full permissions by default, and clusters, warehouses or notebooks shared with all users.

SeeMore Data never queries your own tables or reads your notebooks and files. With a [scoped secret](#oauth-scopes), notebooks, files and clusters are out of the secret's reach, but data granted to `account users` and warehouses shared with all users are still reachable through SQL. With an **All APIs** secret, all of this access is reachable. Either way, store the secret as carefully as any other credential.

### What SeeMore Data reads <a href="#what-seemore-reads" id="what-seemore-reads"></a>

| System schema       | What it's used for                                                                                                                                                                                                                |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `system.access`     | Workspace inventory, account identification, table lineage, Databricks Assistant usage, and the audit log — each event's action, who made it and its full request parameters                                                      |
| `system.billing`    | Consumption per SKU and workspace, and the list and negotiated rates behind it                                                                                                                                                    |
| `system.compute`    | Warehouse and cluster configuration, start/stop/scale events, per-node CPU and memory utilization, and how each cluster node launched and ended                                                                                   |
| `system.lakeflow`   | Jobs, tasks and pipelines, and the timeline of every run, so scheduled work can be costed                                                                                                                                         |
| `system.query`      | Per-query cost attribution and queue pressure, including who ran each query and the error message of a failed one                                                                                                                 |
| `system.tags`       | The governed tag keys defined on your account, used to group spend by tag                                                                                                                                                         |
| `system.serving`    | Which model served each inference request, for whom, at what token count, and the usage context your callers attach                                                                                                               |
| `system.ai_gateway` | Every request through a Unity Gateway model service — destination, requester, client IP address, URL, user agent, your request tags, tokens and latency — and Databricks' estimate of what external model providers bill for them |
| `system.mlflow`     | MLflow experiments and runs, with each run's parameters, tags and metrics                                                                                                                                                         |

Some of these fields are free text that your teams write: query error messages, audit request parameters (which can include job and notebook parameters, file paths and cluster settings), MLflow parameters and tags, and request tags. SeeMore Data copies them as they are, so they can carry any value your teams put in them.

SeeMore Data never reads model prompts or responses. Nothing in `system.serving` or `system.ai_gateway` holds request or response bodies.

Beyond the system schemas, SeeMore Data reads `system.information_schema` for the asset catalog, on the catalogs you granted `BROWSE` on: catalog, schema and table names, table types, owners, comments and timestamps, their tags (keys and values), and the SQL definition of each view, which it uses for view lineage. Through the Databricks API, it also lists your workspace users (names and email addresses), your SQL warehouses and the queries running on them, your model serving endpoints, and your Vector Search endpoints.

Account-wide tables are read once, through the connected region's SQL warehouse. Everything else is read region by region — see [what SeeMore Data reads through each SQL warehouse](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/integration-with-seemore.md#sql-warehouse-per-region).

#### What `BROWSE` exposes <a href="#browse" id="browse"></a>

Through `information_schema`, a principal holding `BROWSE` can see catalog, schema and table names, comments, owners and timestamps, column names and data types, tags, constraints, and the SQL definition of views. It can't read a row of table data, and `DESCRIBE TABLE` is refused. SeeMore Data reads the subset above and not the columns. If your view definitions or tag values hold sensitive literals, grant `BROWSE` only on the catalogs you're comfortable sharing them from.

#### Query text and error messages <a href="#query-text" id="query-text"></a>

Databricks returns `statement_text` in `system.query.history` as `<REDACTED>` to everyone outside the account-level group `databricks_pii_access`. This is platform behavior that applies to every tool — see [Databricks' query history reference](https://docs.databricks.com/aws/en/admin/system-tables/query-history). The setup adds the service principal to the group by default, so SeeMore Data stores the full text of your SQL as it is. It doesn't remove literals from Databricks queries.

Error messages aren't masked. Databricks returns `error_message` in full whether or not the service principal is in the group, and a message can quote values from the failed statement.

If your workspaces use **customer-managed keys**, Databricks encrypts `statement_text` and `error_message`, and no group membership changes that. An account admin can make them readable by adding a key configuration to the `system` catalog. Doing so resets the grants on `system.query` across the metastore, so run the [grants](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#grant-privileges), or the setup script, again afterwards.

#### Tables read <a href="#optional-tables" id="optional-tables"></a>

Five tables must be readable: `system.access.workspaces_latest`, `system.billing.usage`, `system.billing.list_prices`, `system.query.history` and `system.compute.warehouse_events`. The setup's region check also reads the optional tables below. If it can't read one, the region shows **Ready with gaps** and a **Missing** line. A missing one leaves only its feature's views empty. If your policy requires table-level grants rather than schema-level ones, grant all of them, and revisit the list when SeeMore Data starts reading a new table.

{% code overflow="wrap" lineNumbers="true" expandable="true" %}

```sql
-- Prices, machine types and tags: account-wide, checked once.
SELECT 1 AS account_prices_ok   FROM system.billing.account_prices            LIMIT 1;
SELECT 1 AS node_types_ok       FROM system.compute.node_types                LIMIT 1;
SELECT 1 AS governed_tags_ok    FROM system.tags.governed_tags                LIMIT 1;

-- Clusters and SQL warehouses.
SELECT 1 AS warehouses_ok       FROM system.compute.warehouses                LIMIT 1;
SELECT 1 AS clusters_ok         FROM system.compute.clusters                  LIMIT 1;
SELECT 1 AS node_timeline_ok    FROM system.compute.node_timeline             LIMIT 1;
SELECT 1 AS instance_events_ok  FROM system.compute.instance_events           LIMIT 1;

-- Jobs and pipelines.
SELECT 1 AS jobs_ok             FROM system.lakeflow.jobs                     LIMIT 1;
SELECT 1 AS job_tasks_ok        FROM system.lakeflow.job_tasks                LIMIT 1;
SELECT 1 AS job_runs_ok         FROM system.lakeflow.job_run_timeline         LIMIT 1;
SELECT 1 AS task_runs_ok        FROM system.lakeflow.job_task_run_timeline    LIMIT 1;
SELECT 1 AS pipelines_ok        FROM system.lakeflow.pipelines                LIMIT 1;
SELECT 1 AS pipeline_updates_ok FROM system.lakeflow.pipeline_update_timeline LIMIT 1;

-- Lineage and audit.
SELECT 1 AS lineage_ok          FROM system.access.table_lineage              LIMIT 1;
SELECT 1 AS audit_ok            FROM system.access.audit                      LIMIT 1;

-- AI.
SELECT 1 AS serving_usage_ok    FROM system.serving.endpoint_usage            LIMIT 1;
SELECT 1 AS serving_entities_ok FROM system.serving.served_entities           LIMIT 1;
SELECT 1 AS gateway_usage_ok    FROM system.ai_gateway.usage                  LIMIT 1;
SELECT 1 AS gateway_spend_ok    FROM system.ai_gateway.external_model_spend   LIMIT 1;
SELECT 1 AS experiments_ok      FROM system.mlflow.experiments_latest         LIMIT 1;
SELECT 1 AS runs_ok             FROM system.mlflow.runs_latest                LIMIT 1;
SELECT 1 AS run_metrics_ok      FROM system.mlflow.run_metrics_history        LIMIT 1;
SELECT 1 AS assistant_ok        FROM system.access.assistant_events           LIMIT 1;
```

{% endcode %}

An empty result isn't a missing grant — a workspace with no job runs returns no rows either. A permission problem fails with `TABLE_OR_VIEW_NOT_FOUND` or `INSUFFICIENT_PERMISSIONS`.

What some gaps cost you:

* **`node_timeline`** gives per-node CPU and memory for classic job and all-purpose clusters. Jobs on serverless compute or SQL warehouses show statement metrics from `system.query` instead, with or without it.
* **`instance_events`** is in Public Preview at Databricks. Without it, cluster pages show no spot evictions or start-up time.
* **`audit`** is where cluster starts, stops and notebook attaches come from. Databricks records no reason when a cluster stops, so SeeMore Data infers it from these events. Without them, notebook time on a cluster can't be attributed to anyone.

### How SeeMore Data handles your Databricks data <a href="#data-handling" id="data-handling"></a>

SeeMore Data copies the system-table rows and API results listed above into its own storage on AWS. The first import goes back between 14 and 180 days, depending on the table: for example, 30 days for query history and lineage, 90 days for audit events, Databricks Assistant events and job runs, and 180 days for billing. Price lists are imported in full.

* **Encryption.** Everything is encrypted in transit (TLS) and at rest, including your connection credentials.
* **Deletion.** To have imported data deleted, contact the SeeMore Data team.

### Optional cloud billing access

The optional [AWS integration](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/cloud-cost/aws.md) adds AWS billing. It uses a separate read-only role to read billing reports from your S3 bucket; your Databricks service principal does not grant this access. The guide explains the report scope, AWS permissions, and how to revoke access and request deletion of imported cloud-cost data.

See [Security Model & Architecture](https://docs.seemoredata.io/external-docs/security/security/model.md) for certifications.
