clock-rotate-leftFrequency Optimization

Identify tables that are refreshed far more often than they are read and reduce pipeline frequency to save compute.

Overview

Frequency Optimization surfaces tables whose write frequency significantly exceeds their read frequency. When a table is refreshed every hour but only queried once a day, the excess pipeline runs are wasted compute. Reducing the refresh cadence saves money without impacting consumers.


How It Works

Detection Logic

Seemore analyzes query history to build a read/write frequency profile for each table. A table is flagged when:

  1. High write frequency — the table receives regular INSERT, MERGE, or COPY operations.

  2. Low read frequency — SELECT activity is significantly less frequent than writes.

  3. Sustained gap — there are sequences of 4 or more consecutive hours with write activity but no reads, indicating a pattern rather than a one-time spike.


Source and Destination Analysis

Seemore evaluates tables in two roles:

  • As a source (reads) — how often the table is queried by downstream consumers.

  • As a destination (writes) — how often the table is refreshed by upstream pipelines.

The ratio between these two frequencies determines the optimization opportunity.


Savings Estimation

Each insight includes:

  • Current refresh frequency and recommended frequency based on actual read patterns.

  • Estimated annual compute savings from reducing pipeline cadence.

  • Storage savings from fewer intermediate writes.


Remediation

  1. Review the suggested frequency reduction in the insight detail.

  2. Adjust the pipeline schedule (dbt job, Airflow DAG, Fivetran sync frequency) to match the recommended cadence.

  3. Monitor downstream consumers for any impact.

  4. Mark the insight as Done once the schedule is updated.


Configuration

Parameter
Default
Description

Analysis window

21 days

How far back Seemore analyzes read/write patterns.

Minimum write gap

4 hours

Consecutive hours of writes without reads required to flag a table.

circle-info

If you have configured Exclude Users from Usage, excluded users are filtered out of both the source (read) and destination (write) queries used for frequency analysis. This ensures that automated service account activity does not mask a table's true consumption pattern.

Last updated