> For the complete documentation index, see [llms.txt](https://docs.seemoredata.io/external-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/automated-setup.md).

# Automated Setup

Run one script to create the Databricks service principal, warehouse, system schemas and grants that SeeMore Data needs — then verify them before you connect.

The script below does the steps of [Create Databricks Service Principal and Privileges](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md) one region at a time: the service principal and a secret [scoped](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/permissions-and-data-handling.md#oauth-scopes) to the APIs SeeMore Data calls, query text access, and, in the workspace you pass, membership with the **Workspace access** and **Databricks SQL access** entitlements, `seemore_warehouse`, the system schemas and grants, and the warehouse and endpoint permissions. It then **verifies the result by reading the five required tables as the service principal, not as you**, and prints the values to enter in SeeMore Data.

### What each run sets up <a href="#what-each-run-sets-up" id="what-each-run-sets-up"></a>

Run the script once in each region SeeMore Data should fetch, with `--workspace-url` set to that region's [home workspace](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#home-workspace). Each run covers these [levels](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#overview) of the manual steps:

| Level                 | What the script does                                                                                                                                                                                                                                                                                                                                                                                                                                     | On which run                                                                           |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| **Service principal** | Creates it and a scoped secret ([step 1](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#create-service-principal))                                                                                                                                                                                                                          | The first. Later runs reuse both, and generate a new secret only with `--new-secret`   |
| **Account**           | Adds it to `databricks_pii_access` ([step 2](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#account-access)), unless you pass `--no-query-text-access`                                                                                                                                                                                      | The first. Later runs find it there                                                    |
| **Region**            | Adds it to the workspace you pass, with the entitlements, creates `seemore_warehouse` with **Can use**, enables the system schemas and runs the grants ([steps 3 to 5](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#home-workspace))                                                                                                      | Every run, for that region                                                             |
| **Workspace**         | Grants **Can monitor** on the workspace's other warehouses, or **Can manage** with `--can-manage-warehouses`, and **Can view** on its serving endpoints. With `--allowlist-ips`, it also adds SeeMore Data's addresses to the workspace's IP access list ([steps 6 and 7](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#workspace-access)) | Every run, for the workspace you pass only. Do step 6 by hand in your other workspaces |

{% hint style="info" %}
**The defaults cover everything you have.** The script grants `BROWSE` on all your catalogs except Databricks' own, **Can monitor** on every other SQL warehouse, and **Can view** on every model serving endpoint. It also adds the service principal to `databricks_pii_access`, so SeeMore Data can read your query text; pass `--no-query-text-access` to skip that. `--dry-run` lists every object it would touch. To inventory only some catalogs, pass `--no-asset-catalog` and [grant `BROWSE` by hand](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#browse-grant). [Permissions and Data Handling](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/permissions-and-data-handling.md) explains what each permission allows.
{% endhint %}

It's yours to read before you run it. Nothing is sent to SeeMore Data, nothing is written to disk, and every call it makes is one you could make yourself in the Databricks console.

{% hint style="success" %}
**Start with `--dry-run`.** It prints the full plan — every resource it would create and every `GRANT` it would run — and changes nothing. Read the plan, then run the same command without the flag.
{% endhint %}

{% hint style="info" %}
**Safe to re-run.** The service principal and `seemore_warehouse` are matched by name and reused, existing workspace membership is left as it is, and every permission is added rather than replaced. No new secret is generated unless you pass `--new-secret`, and verification is skipped on a run that doesn't generate one — the script says so. If a check fails, fix the cause and run it again.
{% endhint %}

### Before you start

* **Python 3.10 or later**, which the Databricks SDK requires, and the SDK itself, version 0.152.0 or later, the first that creates scoped secrets. On macOS, `/usr/bin/python3` is older than that; check with `python3 --version`.

```bash
pip install "databricks-sdk>=0.152.0"
```

* **Account admin** access. The script uses the account login to create the service principal and its secret, add it to the workspace and add it to `databricks_pii_access`, and those calls need an account admin. Without one, follow the [manual steps](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md) instead: there, a workspace admin can create the service principal.
* **Workspace admin** and **metastore admin** access on the workspace you prepare, for everything else.
* A workspace attached to a **Unity Catalog metastore**. The script stops with a clear message if it isn't — SeeMore Data reads Unity Catalog system tables.

### Authenticating

The script never asks for your credentials. It uses the Databricks SDK's standard authentication, which reads the same environment variables and `~/.databrickscfg` profiles as the Databricks CLI.

Two logins are needed, because the account and the workspace are separate control planes:

| Context       | Used for                                                                                                           | Provide it as                                                                                       |
| ------------- | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- |
| **Account**   | Creating the service principal and its OAuth secret, and adding it to the workspace and to `databricks_pii_access` | `DATABRICKS_ACCOUNT_ID`, `DATABRICKS_CLIENT_ID`, `DATABRICKS_CLIENT_SECRET`, or `--account-profile` |
| **Workspace** | Entitlements, warehouse, system schemas, grants                                                                    | `DATABRICKS_HOST`, `DATABRICKS_TOKEN`, or `--workspace-profile`                                     |

If you already use the Databricks CLI with named profiles, pass them directly and set nothing else:

```bash
python seemore_databricks_setup.py \
  --account-id <account-id> \
  --workspace-url https://dbc-xxxx.cloud.databricks.com \
  --account-profile my-account \
  --workspace-profile my-workspace \
  --dry-run
```

The cloud is worked out from your workspace URL, so there's no separate flag for AWS, Azure or GCP.

### Run it

Save the script as `seemore_databricks_setup.py`, then:

```bash
python seemore_databricks_setup.py \
  --account-id <account-id> \
  --workspace-url https://dbc-xxxx.cloud.databricks.com \
  --dry-run
```

Read the plan. When it looks right, run the same command without `--dry-run`.

A successful run ends with what to enter in SeeMore Data:

```
[12] Verification (running as the service principal)
  [ok]   system.access.workspaces_latest readable
  [ok]   system.billing.usage readable
  [ok]   system.billing.list_prices readable
  [ok]   system.query.history readable
  [ok]   system.compute.warehouse_events readable
  [ok]   asset catalog: 2 catalog(s) visible -- analytics, raw

========================================================================
In SeeMore Data (Marketplace -> Add integration -> Databricks), sign in on the Connected region step:

  Workspace URL                https://dbc-xxxx.cloud.databricks.com
  Service principal client ID  8f2c…
  Service principal secret     dose…

Then choose seemore_warehouse as the SQL warehouse for this
workspace's region. In the region you sign in to, enter this
workspace's URL on Connected region and choose it there. In any other
region, choose it on Other regions. Running this for another region?
Keep the credentials you entered first, and use the same --sp-name.
```

{% hint style="danger" %} <mark style="color:$danger;">**The secret is printed once and stored nowhere.**</mark> Copy it into SeeMore Data, then clear your terminal scrollback. If you lose it, re-run the script with `--new-secret` — it generates a fresh secret and leaves the service principal alone.
{% endhint %}

### Options

| Flag                      | Default             | What it does                                                                                                                                                                                                                                                                                                                                                                   |
| ------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--account-id`            | required            | Your Databricks account ID, from the account console profile menu                                                                                                                                                                                                                                                                                                              |
| `--workspace-url`         | required            | The workspace to prepare: the one you enter in SeeMore Data, or one in another region you want fetched                                                                                                                                                                                                                                                                         |
| `--dry-run`               | off                 | Print the plan and change nothing                                                                                                                                                                                                                                                                                                                                              |
| `--sp-name`               | `seemore`           | Display name of the service principal                                                                                                                                                                                                                                                                                                                                          |
| `--warehouse-name`        | `seemore_warehouse` | Name of the SQL warehouse to create or reuse                                                                                                                                                                                                                                                                                                                                   |
| `--secret-lifetime-days`  | `730`               | OAuth secret lifetime. SeeMore Data stops reading when it expires — record the date                                                                                                                                                                                                                                                                                            |
| `--new-secret`            | off                 | Generate a fresh secret even though the principal already has one. Needed to rotate, or to run verification on a re-run                                                                                                                                                                                                                                                        |
| `--no-serverless`         | off                 | Create a pro warehouse instead of serverless, where serverless isn't available                                                                                                                                                                                                                                                                                                 |
| `--no-query-text-access`  | off                 | Don't add the service principal to the account group that unmasks query text. The step then only reports                                                                                                                                                                                                                                                                       |
| `--no-asset-catalog`      | off                 | Skip `BROWSE` on your own catalogs. SeeMore Data then inventories no assets — costing is unaffected                                                                                                                                                                                                                                                                            |
| `--allowlist-ips`         | off                 | Add SeeMore Data's two egress IPs to the workspace IP access list (Enterprise tier). Only needed if your workspace restricts inbound access; context-based ingress is set separately — see [step 7](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#allowlist-ips) |
| `--can-manage-warehouses` | off                 | Grant **Can manage** instead of **Can monitor** on all your other warehouses, for editing warehouse settings from SeeMore Data                                                                                                                                                                                                                                                 |
| `--account-profile`       | env                 | `~/.databrickscfg` profile for the account login                                                                                                                                                                                                                                                                                                                               |
| `--workspace-profile`     | env                 | `~/.databrickscfg` profile for the workspace login                                                                                                                                                                                                                                                                                                                             |

{% hint style="warning" %}
`--allowlist-ips` adds an **allow** list to the workspace. If the workspace had none, adding one restricts everyone who isn't on a list — check that your other allow lists still cover your users first.
{% endhint %}

**Query text** is unmasked by default: the script creates the `databricks_pii_access` group if it's missing and adds the service principal. That shows SeeMore Data the full text of every query in the account — see [Query text access](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#query-text-access). Pass `--no-query-text-access` if your policy doesn't allow it.

**Can manage** is granted only with `--can-manage-warehouses`, instead of **Can monitor**, and then on every other warehouse in the workspace. Editing a warehouse's settings from SeeMore Data needs it. To grant it on only some warehouses, leave the flag off and add it by hand — see [Can manage](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#grant-can-manage).

**More than one workspace.** The script adds the service principal to the workspace you pass, then lists the account's other workspaces and whether it's a member of each. It never adds it to them, because that changes who can see what in a workspace you didn't nominate. Add the ones you want covered by hand, as described in [step 6](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#workspace-access). A workspace it isn't a member of still contributes its usage, and its query history if it's in a region SeeMore Data fetches; only its live queries, users, warehouse automation and AI Services endpoint details are missing.

**More than one region.** Each region has its own metastore, so run the script again with `--workspace-url` and `--workspace-profile` set to a workspace in that region. The run reuses the service principal and its secret, and prepares that region's warehouse, schemas and grants. If you set `--sp-name` on the first run, pass the same value again, or the run creates a second service principal. Then choose that SQL warehouse for the region on the **Other regions** step of the setup, where its **Check** is the verification. A region you don't prepare still counts toward your total spend; only its query, compute and lineage detail is missing.

{% hint style="warning" %}
**Re-run this after creating a warehouse or a serving endpoint.** Its permissions are granted per object, to the ones that exist at the time, so one created later stays hidden until you run the script again or grant it by hand.
{% endhint %}

### If a check fails

Verification runs as the service principal, so it tests what SeeMore Data will actually be able to read — a metastore admin can read the system tables whether or not the grants worked. It signs in with the service principal's new secret only, even if `DATABRICKS_AUTH_TYPE` or your profile names another way to sign in.

When a check fails, the script prints Databricks' own error alongside the exact `GRANT` statements that fix it. Grants can take a few seconds to propagate, so if a run finishes just after the grants were made, re-running the script is often all that's needed.

If the script stops before it does anything with *no account credentials found* or *no workspace credentials found*, it hasn't found a login for that context. The account and the workspace are separate logins — a workspace profile can't stand in for the account one. Run `databricks auth profiles` to see what you already have.

Two failures have causes the grants can't fix:

* **A system schema reported `UNAVAILABLE`** — Databricks hasn't offered it to your account. Contact Databricks. A schema missing from Databricks' list but present in the `system` catalog is granted as usual, because the list leaves out some schemas Databricks has already enabled. The optional `tags`, `serving`, `ai_gateway` and `mlflow` are skipped without failing when they're missing from both — see [step 4](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/create-databricks-service-principal-and-privileges.md#enable-system-schemas) for what that costs.
* **`GRANT` refused** — the workspace login isn't a metastore admin. Grants on the `system` catalog can only be made by one.

One line in the verification output is a warning rather than a failure:

* **`asset catalog: no catalogs visible`** — SeeMore Data will import no assets, so lineage and unused-table insights come out empty. Costing, query history and warehouse automation are unaffected, which is why this never fails the run. Either a `BROWSE` grant didn't land, or the account holds only Databricks' own catalogs; the script can't tell the two apart. A `could not grant BROWSE on ...` warning earlier in the run names the catalog that refused it.

### The script

{% code title="seemore\_databricks\_setup.py" overflow="wrap" lineNumbers="true" expandable="true" %}

```python
#!/usr/bin/env python3
"""Prepare a Databricks account for SeeMore Data.

Does everything in "Create Databricks Service Principal and Privileges", then
verifies it by reading the five tables SeeMore Data's setup check requires --
as the service principal, not as you. Prints what to enter in SeeMore Data.

SeeMore Data reads each region through a SQL warehouse in that region. For an
account in several regions, run this once more per region, with --workspace-url
set to a workspace in it.

Safe to re-run: every step reuses what already exists, and permissions are added
rather than replaced.

    pip install "databricks-sdk>=0.152.0"
    python seemore_databricks_setup.py --account-id <id> \
        --workspace-url https://dbc-xxxx.cloud.databricks.com --dry-run

Authentication uses the standard Databricks SDK sources, so the script never
handles your credentials. Two contexts are needed, and they are separate logins:

  * Account (account admin) -- creates the service principal and its secret,
    and adds it to the workspace and to databricks_pii_access.
    Set DATABRICKS_ACCOUNT_ID / DATABRICKS_CLIENT_ID / DATABRICKS_CLIENT_SECRET,
    or pass --account-profile for a ~/.databrickscfg profile.
  * Workspace (workspace admin + metastore admin) -- everything else.
    Set DATABRICKS_HOST / DATABRICKS_TOKEN, or pass --workspace-profile.

Drop --dry-run once the plan looks right.
"""

import argparse
import sys
import time

from databricks.sdk import AccountClient, WorkspaceClient
from databricks.sdk.errors import DatabricksError
from databricks.sdk.service.iam import ComplexValue, Patch, PatchOp, PatchSchema, WorkspacePermission
from databricks.sdk.service.serving import (
    ServingEndpointAccessControlRequest,
    ServingEndpointPermissionLevel,
)
from databricks.sdk.service.settings import ListType
from databricks.sdk.service.sql import (
    CreateWarehouseRequestWarehouseType,
    StatementState,
    WarehouseAccessControlRequest,
    WarehousePermissionLevel,
)

# The system schemas SeeMore Data reads. Four are optional and skipped when the
# metastore does not offer them: `tags` is still in preview at Databricks and
# powers governed tags only, and `serving`, `ai_gateway` and `mlflow` are
# regional -- they carry the AI request telemetry, the external-provider spend
# estimate and the MLflow experiment and run activity, while AI spend on the
# Databricks bill comes from `billing` either way.
SCHEMAS = ("access", "billing", "compute", "lakeflow", "query", "tags", "serving", "ai_gateway", "mlflow")
OPTIONAL_SCHEMAS = ("tags", "serving", "ai_gateway", "mlflow")

# The Databricks APIs SeeMore Data calls. The secret is bound to these, so a token
# issued from it cannot reach any other API.
SCOPES = ["sql", "query-history", "access-management", "scim", "model-serving", "vector-search", "unity-catalog"]

# Databricks-provided catalogs, never inventoried. `samples` and the default
# `workspace` catalog are readable without any grant, so leaving them in the
# preflight below would make a failed BROWSE grant look like a success.
EXCLUDED_CATALOGS = ("system", "samples", "__databricks_internal")

# The five tables the setup check requires, and the schema grant behind each.
CHECKS = (
    ("system.access.workspaces_latest", "system.access"),
    ("system.billing.usage", "system.billing"),
    ("system.billing.list_prices", "system.billing"),
    ("system.query.history", "system.query"),
    ("system.compute.warehouse_events", "system.compute"),
)

# States that mean the schema is already readable. MANAGED is what Databricks
# reports for schemas it enables and maintains itself -- most accounts show it
# for all four required schemas, and trying to enable one is refused.
READY_STATES = ("ENABLE_COMPLETED", "MANAGED")

# Databricks masks system.query.history.statement_text for anyone outside this
# account-level group. The name is fixed and case-sensitive.
PII_GROUP = "databricks_pii_access"

ENTITLEMENTS = ("workspace-access", "databricks-sql-access")

SEEMORE_IPS = ("3.78.110.132", "3.73.171.134")

ACCOUNTS_HOSTS = {
    "azuredatabricks.net": "https://accounts.azuredatabricks.net",
    "gcp.databricks.com": "https://accounts.gcp.databricks.com",
    "cloud.databricks.com": "https://accounts.cloud.databricks.com",
}

DRY_RUN = False
_step = 0


def step(msg):
    global _step
    _step += 1
    print(f"\n[{_step}] {msg}", flush=True)


def ok(msg):
    print(f"  [ok]   {msg}")


def kept(msg):
    print(f"  [have] {msg}")


def warn(msg):
    print(f"  [warn] {msg}")


def plan(msg):
    print(f"  [plan] would {msg}")


def fail(msg):
    print(f"\nSetup stopped: {msg}", file=sys.stderr)
    sys.exit(1)


def accounts_host(workspace_url):
    for suffix, host in ACCOUNTS_HOSTS.items():
        if workspace_url.endswith(suffix):
            return host
    fail(
        f"{workspace_url} is not a recognised Databricks workspace URL. "
        "Expected it to end in cloud.databricks.com, azuredatabricks.net or "
        "gcp.databricks.com."
    )


def connect(label, client, flag, **kwargs):
    """Build a client, turning the SDK's default-auth error into a usable one."""
    try:
        return client(**kwargs)
    except ValueError as e:
        if "default auth" not in str(e):
            raise
        fail(
            "\n  ".join(
                [
                    f"no {label} credentials found for {kwargs['host']}.",
                    f"Pass {flag} <name> for a ~/.databrickscfg profile, or set the",
                    "matching DATABRICKS_* environment variables.",
                    "Run 'databricks auth profiles' to see the profiles you have.",
                    "The account and the workspace are separate logins: a workspace",
                    "profile cannot be used for the account, or the other way round.",
                ]
            )
        )


def run_sql(w, warehouse_id, statement):
    """Run one statement and raise with the server's message if it fails."""
    r = w.statement_execution.execute_statement(
        statement=statement, warehouse_id=warehouse_id, wait_timeout="50s"
    )
    if r.status and r.status.state != StatementState.SUCCEEDED:
        detail = r.status.error.message if r.status.error else r.status.state
        raise RuntimeError(f"{statement}\n         {detail}")
    return r


def find_service_principal(a, name):
    escaped = name.replace('"', '\\"')
    return next(iter(a.service_principals.list(filter=f'displayName eq "{escaped}"')), None)


def ensure_service_principal(a, name, lifetime_days, force_new):
    step(f"Service principal {name!r}")
    sp = find_service_principal(a, name)
    if sp:
        kept(f"exists (client ID {sp.application_id})")
    elif DRY_RUN:
        plan("create it at account level")
        return None, None
    else:
        sp = a.service_principals.create(display_name=name, active=True)
        ok(f"created (client ID {sp.application_id})")

    # Only mint a secret when there is none, or when one is asked for. Databricks
    # shows plaintext once, so a secret this run cannot reuse is still a secret
    # somebody holds -- rotating on every run just piles up live credentials.
    existing = list(a.service_principal_secrets.list(service_principal_id=sp.id))
    if existing and not force_new:
        kept(
            f"{len(existing)} existing secret(s), none generated -- "
            "pass --new-secret if you need a fresh one"
        )
        return sp, None
    if DRY_RUN:
        plan(f"generate an OAuth secret valid for {lifetime_days} days")
        return sp, None

    created = a.service_principal_secrets.create(
        service_principal_id=sp.id, lifetime=f"{lifetime_days * 86400}s", scopes=SCOPES
    )
    ok(f"generated an OAuth secret, expires {created.expire_time}")
    if existing:
        warn(
            f"{len(existing)} older secret(s) on this service principal are still "
            "valid -- delete any you no longer use"
        )
    return sp, created.secret


def sp_permissions(a, workspace_id, application_id):
    """The service principal's permissions on one workspace, or None."""
    return next(
        (
            pa.permissions
            for pa in a.workspace_assignment.list(workspace_id=workspace_id)
            if pa.principal and pa.principal.service_principal_name == application_id
        ),
        None,
    )


def assign_one(a, workspace_id, sp, name):
    # An existing assignment is left exactly as it is. update() replaces the
    # principal's whole permission set rather than adding to it, so re-asserting
    # USER would strip ADMIN from a principal that had been given it.
    current = sp_permissions(a, workspace_id, sp.application_id)
    if current:
        levels = ", ".join(str(getattr(x, "value", x)) for x in current)
        kept(f"{name} ({workspace_id}): already a member ({levels})")
    elif DRY_RUN:
        plan(f"add the service principal to {name} ({workspace_id}) as a user")
    else:
        a.workspace_assignment.update(
            workspace_id=workspace_id,
            principal_id=int(sp.id),
            permissions=[WorkspacePermission.USER],
        )
        ok(f"{name} ({workspace_id}): added as a user")


def assign_to_workspace(a, w, sp):
    step("Workspace membership")
    connected = w.get_workspace_id()
    if sp is None:
        plan(f"add the service principal to workspace {connected} as a user")
        return
    try:
        names = {x.workspace_id: x.workspace_name for x in a.workspaces.list()}
    except DatabricksError as e:
        warn(f"could not list the account's workspaces: {e}")
        names = {}
    assign_one(a, connected, sp, names.get(connected) or "connected workspace")

    # Listed, never touched: adding it changes who can see what in a workspace you
    # did not nominate, so that stays your decision.
    others = sorted(((i, n) for i, n in names.items() if i != connected), key=lambda x: x[1] or "")
    if not others:
        return
    print(f"\n  Other workspaces in this account ({len(others)}):")
    missing = 0
    for wid, name in others:
        if sp_permissions(a, wid, sp.application_id):
            kept(f"{name} ({wid}): member")
        else:
            missing += 1
            warn(
                f"{name} ({wid}): not a member -- no live queries, users,"
                " warehouse automation or AI endpoint details"
            )
    if missing:
        print(
            "\n  Nothing here is changed for you. To cover a workspace above, add the\n"
            "  service principal to it in Databricks: Settings -> Identity and access\n"
            "  -> Service principals. Its usage is counted either way, because billing\n"
            "  is account-scoped, and its query history arrives through its region's\n"
            "  SQL warehouse if SeeMore Data fetches that region. What is missing is\n"
            "  its live queries, users, warehouse automation and AI endpoint details."
        )


def grant_entitlements(w, sp):
    step("Workspace entitlements")
    if sp is None:
        plan(f"grant {' and '.join(ENTITLEMENTS)} to the service principal")
        return
    # A workspace's `users` group no longer passes these on by default, and
    # without them every API call the service principal makes is refused.
    # A principal just added to the workspace can take a moment to appear in it.
    escaped = sp.application_id.replace('"', '\\"')
    for _ in range(10):
        ws_sp = next(iter(w.service_principals.list(filter=f'applicationId eq "{escaped}"')), None)
        if ws_sp or DRY_RUN:
            break
        time.sleep(3)
    have = set()
    if ws_sp:
        have = {e.value for e in (w.service_principals.get(id=ws_sp.id).entitlements or [])}
    missing = [e for e in ENTITLEMENTS if e not in have]
    if not missing:
        kept(f"has {' and '.join(ENTITLEMENTS)}")
        return
    if DRY_RUN:
        plan(f"grant {' and '.join(missing)} to the service principal")
        return
    if not ws_sp:
        fail(
            "the service principal is not visible in the workspace yet, so its "
            "entitlements could not be granted. Re-run the script in a minute."
        )
    w.service_principals.patch(
        id=ws_sp.id,
        operations=[Patch(op=PatchOp.ADD, path="entitlements", value=[{"value": e} for e in missing])],
        schemas=[PatchSchema.URN_IETF_PARAMS_SCIM_API_MESSAGES_2_0_PATCH_OP],
    )
    ok(f"granted {' and '.join(missing)}")


def ensure_warehouse(w, name, serverless):
    step(f"SQL warehouse {name!r}")
    existing = next((e for e in w.warehouses.list() if e.name == name), None)
    if existing:
        kept(f"exists (ID {existing.id})")
        return existing.id
    if DRY_RUN:
        plan(f"create a 2X-Small {'serverless' if serverless else 'pro'} warehouse")
        return None
    created = w.warehouses.create(
        name=name,
        cluster_size="2X-Small",
        warehouse_type=CreateWarehouseRequestWarehouseType.PRO,
        enable_serverless_compute=serverless,
        # Pro and classic warehouses can't stop sooner than 10 minutes.
        auto_stop_mins=1 if serverless else 10,
        min_num_clusters=1,
        max_num_clusters=1,
    ).result()
    ok(f"created (ID {created.id})")
    return created.id


def grant_warehouse(w, warehouse_id, client_id, level, label):
    if DRY_RUN or not warehouse_id or not client_id:
        plan(f"grant {level.value} on {label}")
        return
    # update_permissions adds to the ACL; set_permissions would replace it.
    w.warehouses.update_permissions(
        warehouse_id=warehouse_id,
        access_control_list=[
            WarehouseAccessControlRequest(
                service_principal_name=client_id, permission_level=level
            )
        ],
    )
    ok(f"{level.value} on {label}")


def grant_serving_endpoints(w, client_id):
    step("Serving endpoint permissions")
    # The pay-per-token Foundation Model endpoints carry no ID and are visible to the
    # service principal already, so only the endpoints you run need Can view.
    endpoints = [e for e in w.serving_endpoints.list() if e.id]
    if not endpoints:
        kept("no serving endpoints of your own")
        return
    for e in endpoints:
        if DRY_RUN or not client_id:
            plan(f"grant CAN_VIEW on {e.name}")
            continue
        try:
            # update_permissions adds to the ACL; set_permissions would replace it.
            w.serving_endpoints.update_permissions(
                serving_endpoint_id=e.id,
                access_control_list=[
                    ServingEndpointAccessControlRequest(
                        service_principal_name=client_id,
                        permission_level=ServingEndpointPermissionLevel.CAN_VIEW,
                    )
                ],
            )
            ok(f"CAN_VIEW on {e.name}")
        except DatabricksError as err:
            # Not fatal: cost and requests come from the system tables either way;
            # only this endpoint's configuration stays hidden.
            warn(f"could not grant CAN_VIEW on {e.name} -- its configuration stays hidden\n  {err}")


def enable_system_schemas(w):
    step("System schemas")
    assignment = w.metastores.current()
    if not assignment or not assignment.metastore_id:
        fail(
            "this workspace has no Unity Catalog metastore. SeeMore Data reads "
            "Unity Catalog system tables, so connect through a workspace that has one."
        )
    metastore_id = assignment.metastore_id
    states = {
        s.schema: str(getattr(s.state, "value", s.state))
        for s in w.system_schemas.list(metastore_id=metastore_id)
    }
    # The list leaves out some schemas Databricks has already enabled, so a schema
    # missing from it is looked for in the system catalog before it is skipped.
    try:
        in_catalog = {s.name for s in w.schemas.list(catalog_name="system")}
    except DatabricksError:
        in_catalog = set()
    available = []
    for schema in SCHEMAS:
        state = states.get(schema, "UNAVAILABLE")
        if state in READY_STATES:
            kept(f"{schema} already available ({state})")
            available.append(schema)
        elif schema not in states and schema in in_catalog:
            kept(f"{schema} already available (in the system catalog)")
            available.append(schema)
        elif state == "UNAVAILABLE":
            if schema in OPTIONAL_SCHEMAS:
                warn(f"{schema} not offered on this metastore yet -- skipping, it is optional")
            else:
                warn(f"{schema} is UNAVAILABLE -- ask Databricks to enable it for your account")
        elif DRY_RUN:
            plan(f"enable {schema} (currently {state})")
            available.append(schema)
        else:
            try:
                w.system_schemas.enable(metastore_id=metastore_id, schema_name=schema)
                ok(f"{schema} enabled")
            except DatabricksError as e:
                # Databricks manages some schemas itself; that is not a failure.
                if "can only be enabled by Databricks" not in str(e):
                    raise
                kept(f"{schema} managed by Databricks, already available")
            available.append(schema)
    return metastore_id, available


def report_metastore_coverage(a, connected_metastore):
    """Say which workspaces sit in another region, which needs a run of its own."""
    step("Metastore coverage")
    try:
        names = {x.workspace_id: x.workspace_name for x in a.workspaces.list()}
        metastores = list(a.metastores.list())
    except DatabricksError as e:
        warn(f"could not check the account's metastores: {e}")
        return
    others = []
    for m in metastores:
        if m.metastore_id == connected_metastore:
            continue
        try:
            assigned = list(a.metastore_assignments.list(metastore_id=m.metastore_id))
        except DatabricksError:
            assigned = []
        if assigned:
            others.append((m, assigned))
    if not others:
        ok("one metastore in this account, so every system table is covered")
        return
    for m, assigned in others:
        for wid in assigned:
            warn(f"{names.get(wid, wid)} ({wid}) is on {m.name}, region {m.region}")
    print(
        "\n  A metastore is regional, grants apply to one, and SeeMore Data reads\n"
        "  each region only through a SQL warehouse in that region. To fetch a\n"
        "  region above, run this script again with --workspace-url set to one of\n"
        "  its workspaces, then choose its SQL warehouse in SeeMore Data.\n"
        "\n  Spend for the workspaces above is complete either way: system.billing.usage\n"
        "  and the workspace inventory are account-scoped. Until their region is set\n"
        "  up, their query history, compute events and lineage are not read."
    )


def grant_system_catalog(w, warehouse_id, client_id, schemas):
    step("Grants on the system catalog")
    client_id = client_id or "<client-id>"
    statements = [f"GRANT USE CATALOG ON CATALOG system TO `{client_id}`"]
    for schema in schemas:
        statements.append(f"GRANT USE SCHEMA ON SCHEMA system.{schema} TO `{client_id}`")
        statements.append(f"GRANT SELECT ON SCHEMA system.{schema} TO `{client_id}`")
    for statement in statements:
        if DRY_RUN or not warehouse_id or not client_id:
            plan(f"run: {statement}")
            continue
        try:
            run_sql(w, warehouse_id, statement)
            ok(statement)
        except RuntimeError as e:
            fail(f"grant failed. You must be a metastore admin to run it.\n  {e}")


def grant_asset_catalog_browse(w, warehouse_id, client_id, skip):
    step("Browse on your own catalogs (asset catalog)")
    if skip:
        kept("skipped -- SeeMore Data will inventory no assets")
        return
    client_id = client_id or "<client-id>"
    catalogs = [
        c.name for c in w.catalogs.list() if c.name and c.name not in EXCLUDED_CATALOGS
    ]
    if not catalogs:
        warn("no catalogs of your own to grant on -- the asset catalog will be empty")
        return
    for catalog in catalogs:
        # BROWSE is metadata only and catalog-level; Unity Catalog has no wildcard,
        # so it is one statement per catalog.
        statement = f"GRANT BROWSE ON CATALOG `{catalog}` TO `{client_id}`"
        if DRY_RUN or not warehouse_id or client_id == "<client-id>":
            plan(f"run: {statement}")
            continue
        try:
            run_sql(w, warehouse_id, statement)
            ok(statement)
        except RuntimeError as e:
            # Not fatal: assets are observability, so a catalog that refuses the
            # grant costs its own inventory rather than the whole setup.
            warn(f"could not grant BROWSE on `{catalog}` -- it will not be inventoried\n  {e}")


def allowlist_ips(w, label):
    step("IP access list")
    try:
        existing = next((l for l in w.ip_access_lists.list() if l.label == label), None)
    except DatabricksError as e:
        warn(f"IP access lists are unavailable on this workspace: {e}")
        return
    if existing:
        kept(f"{label!r} already exists")
        return
    if DRY_RUN:
        plan(f"create allow list {label!r} with {', '.join(SEEMORE_IPS)}")
        return
    w.ip_access_lists.create(label=label, list_type=ListType.ALLOW, ip_addresses=list(SEEMORE_IPS))
    ok(f"created allow list {label!r}")
    warn(
        "adding an allow list to a workspace that had none restricts everyone else too "
        "-- check your other allow lists still cover your users"
    )


def query_text_access(a, sp, enable):
    """Grant access to unmasked query text, or only report it when skipped."""
    step("Query text access")
    if sp is None:
        plan(f"add the service principal to {PII_GROUP}")
        return
    group = next(iter(a.groups.list(filter=f'displayName eq "{PII_GROUP}"')), None)
    # list() never returns members, even asked for them explicitly, so the group
    # has to be fetched by id to see who is in it.
    members = a.groups.get(id=group.id).members if group else None
    if any(m.value == sp.id for m in (members or [])):
        kept(f"in {PII_GROUP} -- full query text is readable")
        return
    if not enable:
        warn(f"not in {PII_GROUP} -- statement_text will read <REDACTED>")
        print(
            "\n  Databricks masks query text for everyone outside that account-level\n"
            "  group. Cost and timing are unaffected, but every query then arrives\n"
            "  with identical text, so similar-query grouping, query tagging, lineage\n"
            "  and the SQL shown for each query have nothing to work with.\n"
            "\n  Re-run without --no-query-text-access to create the group and join it.\n"
            "  It grants sight of the full text of every query run in the account."
        )
        return
    if DRY_RUN:
        plan(f"{'add the service principal to' if group else 'create'} {PII_GROUP}")
        return
    if group:
        a.groups.patch(
            id=group.id,
            operations=[Patch(op=PatchOp.ADD, path="members", value=[{"value": sp.id}])],
            schemas=[PatchSchema.URN_IETF_PARAMS_SCIM_API_MESSAGES_2_0_PATCH_OP],
        )
        ok(f"added to {PII_GROUP}")
    else:
        # Created through the account API on purpose: a group created from
        # workspace admin settings gives its creator Manage on it.
        a.groups.create(display_name=PII_GROUP, members=[ComplexValue(value=sp.id)])
        ok(f"created {PII_GROUP} and added the service principal")


def verify(workspace_url, client_id, client_secret, warehouse_id):
    step("Verification (running as the service principal)")
    if DRY_RUN or not warehouse_id:
        plan("read each of the five tables the setup check requires")
        plan("list the catalogs the asset catalog will inventory")
        return True
    if not client_secret:
        kept("skipped -- no new secret this run, so these checks cannot sign in")
        return True
    # Pinned, or DATABRICKS_AUTH_TYPE or a profile's auth_type would sign in as
    # you instead and every check would pass. The SDK asks for all-apis unless
    # told otherwise, which a scoped secret refuses.
    sp_w = WorkspaceClient(
        host=workspace_url,
        client_id=client_id,
        client_secret=client_secret,
        auth_type="oauth-m2m",
        scopes=SCOPES,
    )
    failures = []
    for table, schema in CHECKS:
        try:
            run_sql(sp_w, warehouse_id, f"SELECT 1 FROM {table} LIMIT 1")
            ok(f"{table} readable")
        except Exception as e:  # noqa: BLE001 -- report every failure, not just the first
            failures.append((table, schema, e))
            warn(f"{table} NOT readable")
    verify_asset_catalog(sp_w, warehouse_id, client_id)

    if not failures:
        return True
    print("\n  Fix each failure with:")
    for table, schema, e in failures:
        print(f"    -- {table}: {str(e).strip().splitlines()[-1].strip()}")
        print(f"    GRANT USE SCHEMA ON SCHEMA {schema} TO `{client_id}`;")
        print(f"    GRANT SELECT ON SCHEMA {schema} TO `{client_id}`;")
    print("\n  Grants can take a few seconds to propagate. Re-run this script to check again.")
    return False


def verify_asset_catalog(sp_w, warehouse_id, client_id):
    """What the asset fetch itself will see. Never fails the run -- assets are
    observability rather than costing, so an empty catalog does not block onboarding."""
    excluded = ", ".join(f"'{name}'" for name in EXCLUDED_CATALOGS)
    try:
        r = run_sql(
            sp_w,
            warehouse_id,
            "SELECT catalog_name FROM system.information_schema.catalogs "
            f"WHERE catalog_name NOT IN ({excluded}) ORDER BY catalog_name",
        )
    except Exception as e:  # noqa: BLE001 -- report, never block
        warn(f"asset catalog preflight could not run\n  {e}")
        return
    rows = (r.result.data_array if r.result and r.result.data_array else []) or []
    names = [row[0] for row in rows]
    if names:
        ok(f"asset catalog: {len(names)} catalog(s) visible -- {', '.join(names)}")
        return
    warn(
        "asset catalog: no catalogs visible -- SeeMore Data will import no assets.\n"
        "         Either a BROWSE grant did not land, or this account holds only\n"
        "         Databricks' own catalogs. Everything else still works."
    )


def main():
    global DRY_RUN
    p = argparse.ArgumentParser(
        description="Prepare a Databricks account for SeeMore Data.",
        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
    )
    p.add_argument("--account-id", required=True, help="Databricks account ID")
    p.add_argument(
        "--workspace-url",
        required=True,
        help="URL of the workspace to prepare -- run once per region",
    )
    p.add_argument("--account-profile", help="~/.databrickscfg profile for the account login")
    p.add_argument("--workspace-profile", help="~/.databrickscfg profile for the workspace login")
    p.add_argument("--sp-name", default="seemore", help="service principal display name")
    p.add_argument("--warehouse-name", default="seemore_warehouse", help="SQL warehouse name")
    p.add_argument("--secret-lifetime-days", type=int, default=730, help="OAuth secret lifetime")
    p.add_argument(
        "--new-secret",
        action="store_true",
        help="generate a fresh OAuth secret even though the principal already has one",
    )
    p.add_argument(
        "--no-serverless",
        action="store_true",
        help="create a pro warehouse instead of serverless",
    )
    p.add_argument(
        "--allowlist-ips",
        action="store_true",
        help="add SeeMore Data's egress IPs to the workspace allow list",
    )
    p.add_argument(
        "--can-manage-warehouses",
        action="store_true",
        help="also grant Can manage on your other warehouses, for settings edits",
    )
    p.add_argument(
        "--query-text-access",
        action=argparse.BooleanOptionalAction,
        default=True,
        help=f"add the service principal to {PII_GROUP} so query text is not masked",
    )
    p.add_argument(
        "--no-asset-catalog",
        action="store_true",
        help="skip BROWSE on your own catalogs; SeeMore Data will inventory no assets",
    )
    p.add_argument("--dry-run", action="store_true", help="print the plan, change nothing")
    args = p.parse_args()

    DRY_RUN = args.dry_run
    workspace_url = args.workspace_url.rstrip("/")
    host = accounts_host(workspace_url)

    if DRY_RUN:
        print("DRY RUN -- nothing below is applied.", flush=True)

    a = connect("account", AccountClient, "--account-profile",
                host=host, account_id=args.account_id, profile=args.account_profile)
    w = connect("workspace", WorkspaceClient, "--workspace-profile",
                host=workspace_url, profile=args.workspace_profile)

    sp, secret = ensure_service_principal(
        a, args.sp_name, args.secret_lifetime_days, args.new_secret
    )
    client_id = sp.application_id if sp else None
    assign_to_workspace(a, w, sp)
    grant_entitlements(w, sp)

    warehouse_id = ensure_warehouse(w, args.warehouse_name, not args.no_serverless)
    step(f"Warehouse permissions for {args.sp_name!r}")
    grant_warehouse(
        w, warehouse_id, client_id, WarehousePermissionLevel.CAN_USE, args.warehouse_name
    )
    # Can monitor is what makes a warehouse visible to Real Time Monitoring, so it is the
    # default on every other warehouse. Can manage is a superset and replaces it when asked.
    other_level = (
        WarehousePermissionLevel.CAN_MANAGE
        if args.can_manage_warehouses
        else WarehousePermissionLevel.CAN_MONITOR
    )
    for e in w.warehouses.list():
        if e.name != args.warehouse_name:
            grant_warehouse(w, e.id, client_id, other_level, e.name)
    grant_serving_endpoints(w, client_id)

    metastore_id, schemas = enable_system_schemas(w)
    grant_system_catalog(w, warehouse_id, client_id, schemas)
    grant_asset_catalog_browse(w, warehouse_id, client_id, args.no_asset_catalog)
    report_metastore_coverage(a, metastore_id)

    if args.allowlist_ips:
        allowlist_ips(w, "seemore-data")

    query_text_access(a, sp, args.query_text_access)

    verified = verify(workspace_url, client_id, secret, warehouse_id)

    print("\n" + "=" * 72)
    if DRY_RUN:
        print("Dry run complete. Re-run without --dry-run to apply.")
        return
    print("In SeeMore Data (Marketplace -> Add integration -> Databricks), sign in on the Connected region step:\n")
    print(f"  Workspace URL                {workspace_url}")
    print(f"  Service principal client ID  {client_id}")
    print(f"  Service principal secret     {secret or '[unchanged - reuse your existing secret]'}")
    print(f"\nThen choose {args.warehouse_name} as the SQL warehouse for this")
    print("workspace's region. In the region you sign in to, enter this")
    print("workspace's URL on Connected region and choose it there. In any other")
    print("region, choose it on Other regions. Running this for another region?")
    print("Keep the credentials you entered first, and use the same --sp-name.")
    print("\nThe secret is shown once and is not written to disk. Store it securely,")
    print("and clear your terminal scrollback when you are done.")
    if not verified:
        sys.exit(1)


if __name__ == "__main__":
    main()
```

{% endcode %}

### Next step

{% content-ref url="/pages/tavYfi9vb01RuziE5oFG" %}
[Integration with Seemore](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/integration-with-seemore.md)
{% endcontent-ref %}
