> For the complete documentation index, see [llms.txt](https://docs.seemoredata.io/external-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/cloud-cost/aws.md).

# AWS billing

Connect your AWS billing to see the infrastructure cost of your Databricks workloads in SeeMore Data.

An AWS integration adds the AWS infrastructure cost of your Databricks workloads to SeeMore Data. It is optional, and it links to one Databricks integration on AWS. For Databricks on Azure, see [Azure billing](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/cloud-cost/azure.md).

AWS writes a Cost and Usage Report 2.0 (CUR 2.0) export to an S3 bucket in your account. SeeMore Data reads the report through a read-only IAM role that a CloudFormation stack creates. You do not share access keys.

{% hint style="info" %}
**Required role to configure:** [Admin](https://docs.seemoredata.io/external-docs/fundamentals/settings/user-roles.md) or higher.

Only Admins and Owners see AWS integration cards and setup. Other users see attributed AWS costs within their workspace permissions.
{% endhint %}

## What SeeMore Data can access

* **Only the report.** Your read-only role can list and read the `seemore/` folder of one bucket, and decrypt with your KMS key if you supply one. It cannot write or delete objects, change billing settings, or activate tags.
* **Only the accounts you list.** For an account that runs only Databricks, the report includes all its usage. For an account that also runs other workloads, it includes only usage tagged `Vendor=Databricks`.
* **The tags on those charges.** Each charge includes every active cost allocation tag on its resource, not only the Databricks keys, and its AWS cost categories. Review your active tags before you connect.
* **The report stays in your bucket.** SeeMore Data copies the billing rows into its own storage. Deleting the stack ends SeeMore Data's access to the bucket; see [Access and removal](#access-and-removal). For retention, see [Security Model & Architecture](https://docs.seemoredata.io/external-docs/security/security/model.md).

Your AWS account pays the standard S3 charges for the report bucket, including SeeMore Data's reads.

## Before you start

| Step                                        | Who                                                                                       | How long                                  |
| ------------------------------------------- | ----------------------------------------------------------------------------------------- | ----------------------------------------- |
| Create the AWS integration                  | A SeeMore Data Admin or Owner                                                             | A few minutes                             |
| Deploy the stack                            | An AWS administrator of the billing account                                               | A few minutes                             |
| [Activate tags](#prepare-and-activate-tags) | An administrator of your AWS Organizations management account, or of a standalone account | Up to 48 hours                            |
| Deliver the first report                    | AWS                                                                                       | Up to 24 hours after the stack is created |
| Import the first report                     | SeeMore Data                                                                              | The next daily sync                       |

You also need:

* A [Databricks integration](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/integration-with-seemore.md) whose setup is finished and whose workspaces are imported.
* The IDs of the AWS accounts where your Databricks workloads run, and the workspaces that run in each.
* Room for one more CUR 2.0 export in the billing account. AWS allows five. See [AWS quotas](https://docs.aws.amazon.com/cur/latest/userguide/dataexports-quotas.html).

The AWS administrator needs permission to create an IAM role, an S3 bucket, and a data export, including `cur:PutReportDefinition`. See [AWS Data Exports permissions](https://docs.aws.amazon.com/cur/latest/userguide/bcm-data-exports-access.html).

Always deploy the stack in **US East (N. Virginia), `us-east-1`**, even when the report bucket is in another region. Only AWS commercial regions are supported.

### Decide which accounts to include <a href="#decide-which-accounts-to-include" id="decide-which-accounts-to-include"></a>

**Usage accounts** are the AWS accounts where your Databricks workloads run. For each one, choose what runs in it:

| What runs in this account                    | What goes into the report             |
| -------------------------------------------- | ------------------------------------- |
| **Only Databricks**                          | All usage in that account             |
| **Databricks and other workloads** (default) | Only usage tagged `Vendor=Databricks` |

If you are not sure, keep **Databricks and other workloads**. A wrong choice then only leaves out untagged usage. An AWS integration takes 1 to 300 usage accounts.

The **billing account** is where you deploy the stack. Its report must cover every usage account:

| Your usage accounts                                       | Billing account                                                                                                                                       |
| --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| One standalone account                                    | That account. SeeMore Data fills it in.                                                                                                               |
| One member account of an AWS organization                 | Usually that account. If you know that it cannot create its own export, select **Change** under **Billing export**, and enter the management account. |
| Several accounts in one AWS organization                  | The management (payer) account. A member account's report covers only that member account.                                                            |
| Accounts that a reseller or another organization pays for | That payer's account. Create one AWS integration per payer.                                                                                           |

You can create one AWS integration per billing account.

### Prepare and activate tags <a href="#prepare-and-activate-tags" id="prepare-and-activate-tags"></a>

Without active tags, SeeMore Data can import costs but cannot attribute them. Only an AWS Organizations management account, or a standalone account, can activate cost allocation tags. The root user is not required. See [Activating cost allocation tags](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/activating-tags.html).

In **AWS Billing and Cost Management → Cost allocation tags**, activate these keys:

| Tag key         | What SeeMore Data uses it for                             | Activate                                                 |
| --------------- | --------------------------------------------------------- | -------------------------------------------------------- |
| `Vendor`        | Selects Databricks usage. The value must be `Databricks`. | Required for **Databricks and other workloads** accounts |
| `Name`          | Identifies the workspace of an instance                   | Recommended                                              |
| `ClusterId`     | Links an instance's cost to its cluster                   | Recommended                                              |
| `SqlEndpointId` | Links an instance's cost to its classic SQL warehouse     | If you use classic SQL warehouses                        |
| `JobId`         | Identifies the job on a job cluster                       | If you use jobs compute                                  |

Databricks applies these keys to the instances it launches. In **Databricks and other workloads** accounts, also tag the infrastructure that Databricks uses with `Vendor=Databricks`: the NAT gateway, its Elastic IP and network interface, the workspace and Unity Catalog buckets, and VPC endpoints.

A new key can take up to 48 hours to appear and activate. Activation does not tag earlier charges. See [History and backfill](#history-and-backfill).

### Choose the report bucket <a href="#choose-the-report-bucket" id="choose-the-report-bucket"></a>

* **A new bucket (recommended):** the stack creates a private bucket in `us-east-1`, with S3-managed encryption and the policy that AWS needs to deliver the report. The bucket stays if you delete the stack.
* **An existing bucket:** select **Use an existing S3 bucket**, and enter the **Bucket name** and **Bucket region**. With one usage account, select **Change** first. The bucket must belong to the billing account. Before you deploy the stack, add the two statements that SeeMore Data shows to the bucket policy. See [Bucket delivery policy](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/cloud-cost/aws-permissions.md#bucket-delivery-policy).

If an existing bucket uses SSE-KMS, enter the **KMS key ARN**. The key policy must let AWS Data Exports encrypt the report and let your read-only role decrypt it. If you use service control policies (SCPs), they must allow the export and the role's S3 and KMS access.

You cannot change the billing account or the bucket after you save.

## 1. Create the AWS integration

Start in one of these places:

* In **Marketplace**, select **Add integration**, and then **AWS** in the **Cloud** section.
* On a Databricks integration card, select **Connect cloud provider billing**.
* After you save a Databricks integration, select **Connect cloud provider billing**.

The setup opens in a side panel with three steps:

1. **Databricks:** enter an **Integration name** if asked, and choose the **Databricks integration**. You cannot change it later.
2. **Accounts:** add each usage account with its **AWS account ID**, **What runs in this account**, and **Databricks workspaces**. To add several accounts, paste a list of IDs into one field. Then check **Billing export**. See [Decide which accounts to include](#decide-which-accounts-to-include).
3. Select **Save and get AWS setup**. The panel opens **Connect**.

<figure><img src="https://3620459840-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnSbIoBjUxWTGNWa9gGw7%2Fuploads%2Fgit-blob-8b978392e73994a16116d5cbc9a07088efd4efd4%2Faws-setup-accounts.png?alt=media" alt="The Accounts step with two AWS accounts: one set to Only Databricks with one workspace, one set to Databricks and other workloads with two workspaces, and the billing account entered under Billing export"><figcaption><p>Two usage accounts and the billing account that covers them</p></figcaption></figure>

Costs that match more than one mapped workspace stay unallocated. Only the workspaces that you can access in the Databricks integration are listed.

Saving turns on imports. Nothing is deployed in AWS yet. You can close the panel at any time. To return, select **Edit** on the AWS integration card.

## 2. Deploy the stack

**Connect** shows the values that your AWS administrator needs, each with a copy button. To send the instructions and values to your administrator, select **Copy setup for your AWS admin**.

<figure><img src="https://3620459840-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnSbIoBjUxWTGNWa9gGw7%2Fuploads%2Fgit-blob-d74f6f1eb1371c15a001941aa3e184906e749942%2Faws-setup-connect.png?alt=media" alt="The Connect step of a new AWS integration with the status Waiting for setup: the Download template button, the stack name and parameters with copy buttons, and the Test connection button"><figcaption><p>Everything your AWS administrator needs to deploy the stack</p></figcaption></figure>

1. Select **Download template**.
2. Sign in to the billing account. Check that the account ID in the console's account menu matches **Connect**. Then open **CloudFormation** in `us-east-1`.
3. Create a stack from the template file. Enter the stack name and the parameters exactly as shown. Leave the values marked **Leave empty** blank.
4. Acknowledge that the stack creates an IAM role, create the stack, and wait for **CREATE\_COMPLETE**.
5. [Activate the cost allocation tags](#prepare-and-activate-tags).

Use only the values from this AWS integration. The `ExternalId` ties your read-only role to this integration. It is not a secret. Do not change the export in AWS later: SeeMore Data checks each report against the export's settings.

The stack creates the CUR 2.0 export, which writes hourly, resource-level files to the `seemore/` folder, and your read-only IAM role. For a new bucket, it also creates the bucket and its delivery policy. It does not change an existing bucket. For the exact policies, see [AWS billing permissions](https://docs.seemoredata.io/external-docs/fundamentals/getting-set-up/setting-integrations/databricks/cloud-cost/aws-permissions.md).

## 3. Test the connection

When the stack shows **CREATE\_COMPLETE**, select **Test connection**. The test only reads. It imports nothing.

<figure><img src="https://3620459840-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnSbIoBjUxWTGNWa9gGw7%2Fuploads%2Fgit-blob-f17ed3d044dba31488353bac94e800ef12835557%2Faws-connection-test.png?alt=media" alt="Test the connection with all six checks passed: four connection checks, then the two report checks with the billing period and delivery time of the latest report"><figcaption><p>A passed test. The report checks pass once AWS delivers the first report.</p></figcaption></figure>

The first four checks confirm access to your read-only role and the bucket. The two report checks pass after AWS delivers the first report, which can take up to 24 hours. You can close the panel while you wait.

The test stops at the first check that fails or is pending, and shows what to do. The checks after it show **Not checked yet**. See [Connection test](#connection-test).

## 4. Imports and status

The AWS integration imports once a day with your account's daily sync, and it keeps importing when the Databricks integration is paused. After AWS delivers a report, the next sync imports it. Each import picks up new months and months that AWS has refreshed. A refreshed month replaces its earlier data, so nothing is counted twice.

To pause imports, turn off **Auto sync** on the AWS integration card.

<figure><img src="https://3620459840-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FnSbIoBjUxWTGNWa9gGw7%2Fuploads%2Fgit-blob-7e66d40fd0abc1b35ffeee5cd001c65c3c332443%2Faws-integration-card.png?alt=media" alt="A Databricks integration card that lists its AWS integration and a Connect cloud provider billing button, next to the AWS integration card with its Databricks integration, last import attempt, last successful import, and Auto sync"><figcaption><p>The Databricks card lists its AWS integrations. The AWS card shows the last imports.</p></figcaption></figure>

The AWS integration card shows a status only when something needs attention. **Connect** always shows the status. After imports, it also shows the last delivery, the last import, and the newest month's rows and cost.

| Status                        | Meaning and next step                                                                                                                                         |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Waiting for setup**         | AWS has not delivered a report yet. Deploy the stack, test the connection, and allow up to 24 hours.                                                          |
| **No report yet**             | There is still no report a few days after setup. Select **Test connection** to find the missing step. If every check passes, contact the SeeMore Data team.   |
| **Report rejected**           | A report failed SeeMore Data's checks, or access was lost after an earlier delivery. **Connect** shows the reason. See [Rejected reports](#rejected-reports). |
| **Report out of date**        | No new report was imported for a few days. Check that the stack still exists and that **Auto sync** is on. Then select **Test connection**.                   |
| **No Databricks costs found** | The reports contain no costs. Check the account list and the `Vendor` tag.                                                                                    |
| **Imported**                  | Costs are imported.                                                                                                                                           |

## 5. Confirm attribution

1. **Test connection** passes, and **Connect** shows **Imported**.
2. The newest month's cost in **Connect** is close to AWS Cost Explorer for the same accounts. For **Databricks and other workloads** accounts, filter Cost Explorer on `Vendor=Databricks`.
3. A classic cluster that ran after you activated the tags shows an AWS part next to its DBU cost on its compute unit.
4. On the Databricks cost overview, **Cloud infrastructure (AWS)** splits AWS cost into **Compute units**, **Networking**, **Storage**, and **Other**.

## How AWS costs are attributed

SeeMore Data combines your active tags with your Databricks data to assign AWS usage costs to compute units: SQL warehouses, jobs, pipelines, and all-purpose clusters.

* Credits, refunds, taxes, and commitment fees are left out. Usage that Savings Plans or Reserved Instances cover counts at its effective cost.
* NAT gateways, data transfer, and storage belong to no single compute unit. They stay with their workspace when it is known, and are never split across jobs.
* Instances from a pool may not carry cluster tags. SeeMore Data links them where your Databricks data allows.
* Serverless compute runs in the Databricks account, so it is not in your AWS report.
* Cost without an identifiable workspace or cluster stays unallocated.

## History and backfill

SeeMore Data imports every month that the export delivers. A new export may not include earlier months. To get them, open an AWS Support case in the billing account, and ask for a backfill of the export. Give its `ExportName` from **Connect** and the first month you need. To attribute a month, SeeMore Data also needs your Databricks data for that month.

Tags apply only from activation. To tag up to twelve earlier months, your billing-tag administrator can request a cost allocation tag backfill in the management account. A backfill applies the current status of **all** your tag keys, across your organization, to those months. SeeMore Data does not need one. AWS then refreshes the reports, and SeeMore Data imports them. See [Backfilling cost allocation tags](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-allocation-backfill.html).

## Troubleshooting

### Connection test

| Check that fails                         | What to do                                                                                                                                                                                            |
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Seemore can sign in to the read role** | Usually the stack is not deployed yet. If it is, check that it completed and that its `ExternalId` matches **Connect**. Compare the role ARN shown under the check with the stack's `RoleArn` output. |
| **The role requires the external ID**    | Update the stack from the SeeMore Data template, with this integration's values.                                                                                                                      |
| **The bucket region matches**            | The message names the bucket's actual region. You cannot change the region of a saved AWS integration. Delete it, and create a new one with the correct region.                                       |
| **Seemore can read the bucket**          | Check that the stack created the bucket or, for your own bucket, that its policy has the statements from **Connect**. Also check for an SCP that denies access.                                       |
| **AWS has delivered a report** (pending) | Allow up to 24 hours. Then check the export in **AWS Billing and Cost Management → Data Exports**, and check for files in the bucket's `seemore/` folder.                                             |
| **Seemore can read the report**          | Check that the KMS key policy lets your read-only role decrypt the report.                                                                                                                            |

### Stack creation fails

Read the failed resource's status reason in CloudFormation. Fix the cause, delete the failed stack, and create it again.

| Error                                                                                          | What to do                                                                                                                                                                                                                                                     |
| ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| An unrecognized resource type                                                                  | The stack is not in `us-east-1`. Create it there.                                                                                                                                                                                                              |
| "Give at least one account ID…", "…set BucketRegion to us-east-1", or "…leave KmsKeyArn empty" | A parameter differs from **Connect**. Enter the parameters exactly as shown.                                                                                                                                                                                   |
| A role or export with that name already exists                                                 | A stack for this integration already exists in the account. Use it, or delete it first.                                                                                                                                                                        |
| The data export fails in a member account                                                      | Your management account or a reseller bills this account. You cannot change the billing account of a saved AWS integration. Delete the failed stack and the AWS integration, and create a new AWS integration with the payer's account as the billing account. |

### Rejected reports

**Connect** shows the reason below the status. Fix the cause. The next import tries the report again.

| The reason starts with                               | What to do                                                                                                                   |
| ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| "AWS access or report validation failed"             | Access was lost after an earlier delivery. Select **Test connection**.                                                       |
| "usage accounts not approved"                        | The report includes an account that this integration does not list. See [Change account coverage](#change-account-coverage). |
| "export owner" or "export name"                      | The report comes from a different export. Update the stack with this integration's values.                                   |
| "data files outside"                                 | Do not move or rename report files.                                                                                          |
| "required columns missing" or "column types changed" | The export was changed in AWS. Update the stack from the SeeMore Data template.                                              |
| Any other reason                                     | Contact the SeeMore Data team.                                                                                               |

### Missing or late costs

| Symptom                               | What to check                                                                                                          |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| No costs after setup                  | AWS can take up to 24 hours to deliver the first report. The next daily sync imports it.                               |
| The last day or two are missing       | AWS refreshes the report about once a day, so recent costs can lag by one to two days.                                 |
| **No Databricks costs found**         | Check that `Vendor=Databricks` is on the resources and that the `Vendor` key is active.                                |
| Most costs are unallocated            | Check that `Name` and `ClusterId` are active and on the instances, and that each account maps to the right workspaces. |
| Tag keys look different in the report | CUR 2.0 renames tag keys, for example `ClusterId` to `user_cluster_id`. Activate the original names.                   |
| Older months are unallocated          | The tags were activated after that usage. See [History and backfill](#history-and-backfill).                           |

## Change account coverage <a href="#change-account-coverage" id="change-account-coverage"></a>

Saving accounts in SeeMore Data does not update the stack. To edit them, select **Edit** on the AWS integration card, go to **Accounts**, and select **Save changes**. **Connect** then shows **Update your AWS stack** with new `DedicatedAccountIds` and `SharedAccountIds` values.

* **Add accounts:** save them in SeeMore Data first. Then update the stack.
* **Remove accounts:** update the stack first. After AWS delivers a report without those accounts, save the shorter list.

To update the stack, update it in CloudFormation with its current template, and change only those two parameters. Then select **Test connection**. Workspace changes need no stack update.

## Access and removal

* **Pause imports:** turn off **Auto sync** on the AWS integration card. Your setup and imported costs stay.
* **Delete the AWS integration:** on its card, open the menu and select **Delete**. Imports stop, and its AWS costs no longer appear. Your AWS account does not change, and imported billing data is not deleted. To have it deleted, contact the SeeMore Data team.
* **Revoke access:** delete the CloudFormation stack. This removes your read-only role and the export. A bucket that the stack created stays, with its files.

To delete a Databricks integration, first delete its linked AWS integrations.
