For the complete documentation index, see llms.txt. This page is also available as Markdown.

Permissions and Data Handling

Review every permission the SeeMore Data service principal holds in Databricks, what SeeMore Data reads with it, and how that data is handled.

This page is the reference for a security review of the Databricks integration. It lists every permission the service principal holds, what SeeMore Data reads with it, and how that data is stored. Create Databricks Service Principal and Privileges and Automated Setup explain how to set these permissions up.

At a glance

Item
SeeMore Data

Rows in your tables, volumes or files

Doesn't read. It asks for no SELECT on your catalogs. See inherited access

Model prompts and responses

Doesn't read. It isn't granted inference tables or payload logs

Notebook and file contents, secret values

Doesn't read

Catalog, schema and table names, owners, comments, tags and view SQL

Reads, on the catalogs you grant BROWSE on

Column names and types

Doesn't read. They're visible to BROWSE

Cost, usage and prices

Reads

Warehouse, cluster, job and pipeline configuration and runs

Reads

Who ran each query, when, for how long, and its error message

Reads

Full SQL text

Reads, through the databricks_pii_access group that the setup adds the service principal to. You can skip that step

Audit events, with their request parameters

Reads

AI request metadata: requester, client IP, user agent, tokens, latency

Reads

MLflow runs, with their parameters, tags and metrics

Reads

Workspace user names and email addresses

Reads

Changes to your account

Only on SQL warehouses you grant Can manage on. There, SeeMore Data changes size, cluster limits or auto stop when one of your users saves a change. It doesn't start, stop or delete warehouses, or change their permissions. Without Can manage, it changes nothing. Can monitor would allow starting a warehouse; SeeMore Data doesn't

Permissions

Permission
Granted on
What it allows
Needed for

Service principal with an OAuth secret

Account

Signing in. It holds no account role

Everything

Workspace membership, as a user, with the Workspace access and Databricks SQL access entitlements

Each workspace you add it to

Signing in to that workspace and using its SQL warehouses. It doesn't need cluster or pool creation

Query history, live queries and user lists for that workspace

USE CATALOG

system catalog

Reaching the system schemas

Everything

USE SCHEMA and SELECT

The nine system schemas

Reading the system tables listed below

Everything; tags, serving, ai_gateway and mlflow are optional

BROWSE

Each catalog you choose

Metadata only: no row of data

The asset catalog

Can use

seemore_warehouse in each region

Running SeeMore Data's reads

Everything

Can monitor

Each warehouse you want watched live

Seeing the warehouse, its query history and query profiles. Databricks also lets it start the warehouse and run queries; SeeMore Data does neither

Real Time Monitoring

Can view

Each model serving endpoint

Seeing the endpoint's configuration and state. It can't query or change the endpoint

AI Services endpoint details

Can manage (optional)

SQL warehouses you choose

Starting, stopping, editing and deleting the warehouse, and changing its permissions. SeeMore Data uses it only to read the warehouse's permission list, to confirm it holds the grant, and to change the warehouse's size, cluster limits and auto stop

Editing a warehouse's settings from SeeMore Data. See Can manage

databricks_pii_access membership (default; you can skip it)

Account group

Reading the full text of every query in the account

Similar-query grouping, query tagging and lineage, and the SQL shown for each query, including live queries

OAuth scopes

Scope the OAuth secret to the seven Databricks APIs SeeMore Data calls:

Scope
What SeeMore Data uses it for

sql

Running its reads on the SQL warehouse, and reading and editing warehouse settings

query-history

Live queries, for Real Time Monitoring

access-management

Reading a warehouse's permission list, to confirm it holds Can manage

scim

The workspace's user list

model-serving

Model serving endpoint details, for AI Services

vector-search

The list of vector search endpoints, for AI Services

unity-catalog

The workspace's metastore, cloud and region

A token issued from a scoped secret can't call any other Databricks API, such as jobs, clusters, notebooks or workspace files. If the secret is missing one of the seven, the connection test names it. A secret with All APIs also works, but reaches every API the service principal is allowed to use.

Scopes limit which APIs the secret can call, not what it can read. That is still bounded by the permissions above and the inherited access below.

SeeMore Data never asks for SELECT, MODIFY or ALL PRIVILEGES on your catalogs, schemas or tables, or for workspace admin, metastore admin or any account role.

Inherited access

Like every identity in your account, the service principal is a member of the account users group, and of the users group of each workspace you add it to. It holds anything you've granted to those groups:

  • Data granted to account users. To see it, run this for each catalog:

  • Workspace objects open to all users, such as the /Workspace/Shared folder, where Databricks gives every user full permissions by default, and clusters, warehouses or notebooks shared with all users.

SeeMore Data never queries your own tables or reads your notebooks and files. With a scoped secret, notebooks, files and clusters are out of the secret's reach, but data granted to account users and warehouses shared with all users are still reachable through SQL. With an All APIs secret, all of this access is reachable. Either way, store the secret as carefully as any other credential.

What SeeMore Data reads

System schema
What it's used for

system.access

Workspace inventory, account identification, table lineage, Databricks Assistant usage, and the audit log — each event's action, who made it and its full request parameters

system.billing

Consumption per SKU and workspace, and the list and negotiated rates behind it

system.compute

Warehouse and cluster configuration, start/stop/scale events, per-node CPU and memory utilization, and how each cluster node launched and ended

system.lakeflow

Jobs, tasks and pipelines, and the timeline of every run, so scheduled work can be costed

system.query

Per-query cost attribution and queue pressure, including who ran each query and the error message of a failed one

system.tags

The governed tag keys defined on your account, used to group spend by tag

system.serving

Which model served each inference request, for whom, at what token count, and the usage context your callers attach

system.ai_gateway

Every request through a Unity Gateway model service — destination, requester, client IP address, URL, user agent, your request tags, tokens and latency — and Databricks' estimate of what external model providers bill for them

system.mlflow

MLflow experiments and runs, with each run's parameters, tags and metrics

Some of these fields are free text that your teams write: query error messages, audit request parameters (which can include job and notebook parameters, file paths and cluster settings), MLflow parameters and tags, and request tags. SeeMore Data copies them as they are, so they can carry any value your teams put in them.

SeeMore Data never reads model prompts or responses. Nothing in system.serving or system.ai_gateway holds request or response bodies.

Beyond the system schemas, SeeMore Data reads system.information_schema for the asset catalog, on the catalogs you granted BROWSE on: catalog, schema and table names, table types, owners, comments and timestamps, their tags (keys and values), and the SQL definition of each view, which it uses for view lineage. Through the Databricks API, it also lists your workspace users (names and email addresses), your SQL warehouses and the queries running on them, your model serving endpoints, and your Vector Search endpoints.

Account-wide tables are read once, through the connected region's SQL warehouse. Everything else is read region by region — see what SeeMore Data reads through each SQL warehouse.

What BROWSE exposes

Through information_schema, a principal holding BROWSE can see catalog, schema and table names, comments, owners and timestamps, column names and data types, tags, constraints, and the SQL definition of views. It can't read a row of table data, and DESCRIBE TABLE is refused. SeeMore Data reads the subset above and not the columns. If your view definitions or tag values hold sensitive literals, grant BROWSE only on the catalogs you're comfortable sharing them from.

Query text and error messages

Databricks returns statement_text in system.query.history as <REDACTED> to everyone outside the account-level group databricks_pii_access. This is platform behavior that applies to every tool — see Databricks' query history reference. The setup adds the service principal to the group by default, so SeeMore Data stores the full text of your SQL as it is. It doesn't remove literals from Databricks queries.

Error messages aren't masked. Databricks returns error_message in full whether or not the service principal is in the group, and a message can quote values from the failed statement.

If your workspaces use customer-managed keys, Databricks encrypts statement_text and error_message, and no group membership changes that. An account admin can make them readable by adding a key configuration to the system catalog. Doing so resets the grants on system.query across the metastore, so run the grants, or the setup script, again afterwards.

Tables read

Five tables must be readable: system.access.workspaces_latest, system.billing.usage, system.billing.list_prices, system.query.history and system.compute.warehouse_events. Permission checks also reads the optional tables below and reports any it can't read as Ready with optional gaps. A missing one leaves only its feature's views empty. If your policy requires table-level grants rather than schema-level ones, grant all of them, and revisit the list when SeeMore Data starts reading a new table.

An empty result isn't a missing grant — a workspace with no job runs returns no rows either. A permission problem fails with TABLE_OR_VIEW_NOT_FOUND or INSUFFICIENT_PERMISSIONS.

What some gaps cost you:

  • node_timeline gives per-node CPU and memory for classic job and all-purpose clusters. Jobs on serverless compute or SQL warehouses show statement metrics from system.query instead, with or without it.

  • instance_events is in Public Preview at Databricks. Without it, cluster pages show no spot evictions or start-up time.

  • audit is where cluster starts, stops and notebook attaches come from. Databricks records no reason when a cluster stops, so SeeMore Data infers it from these events. Without them, notebook time on a cluster can't be attributed to anyone.

How SeeMore Data handles your Databricks data

SeeMore Data copies the system-table rows and API results listed above into its own storage on AWS. The first import goes back between 14 and 180 days, depending on the table: for example, 30 days for query history and lineage, 90 days for audit events, Databricks Assistant events and job runs, and 180 days for billing. Price lists are imported in full.

  • Encryption. Everything is encrypted in transit (TLS) and at rest, including your connection credentials.

  • Deletion. To have imported data deleted, contact the SeeMore Data team.

Optional cloud billing access

The optional AWS integration adds AWS billing. It uses a separate read-only role to read billing reports from your S3 bucket; your Databricks service principal does not grant this access. The guide explains the report scope, AWS permissions, and how to revoke access and request deletion of imported cloud-cost data.

See Security Model & Architecture for certifications.

Last updated