dbt Core

DBT Core integration to Seemore>data

Seemore supports dbt core by using custom tags for classifying data assets that run through DBT core.

1. In order to integrate you would need to add the following tags package - dbt snowflake query tags package.

2. In order to add more informative metadata such as data orchestration tools that execute the DBT Queries add the following code.

Quickstart

1. Add this package to your packages.yml file, then install it with dbt deps.

packages:
  - package: get-select/dbt_snowflake_query_tags
    version: [">=2.0.0", "<3.0.0"]

1. Adding the query tags

Option 1: If running dbt >= 1.2, simply configure the dispatch search order in dbt_project.yml.

dispatch:
  - macro_namespace: dbt
    search_order:
      - <YOUR_PROJECT_NAME>
      - dbt_snowflake_query_tags
      - dbt

Option 2: If running dbt < 1.2, create a folder named macros in your dbt project's top level directory (if it doesn't exist). Inside, make a new file called query_tags.sql with the following content:

{% macro set_query_tag() -%}
{% do return(dbt_snowflake_query_tags.set_query_tag()) %}
{% endmacro %}

{% macro unset_query_tag(original_query_tag) -%}
{% do return(dbt_snowflake_query_tags.unset_query_tag(original_query_tag)) %}
{% endmacro %}

1. To configure the query comments, add the following config to dbt_project.yml.

query-comment:
  comment: '{{ dbt_snowflake_query_tags.get_query_comment(node) }}'
  append: true # Snowflake removes prefixed comments.

That's it! All dbt-issued queries will now be tagged.

Once the dbt package to automatically tag dbt-issued queries with informative metadata has been added you can add more informative information such as Orchestration tools used to execute DBT queries.

Airflow Example

query-comment:
  comment: >
    {{ dbt_snowflake_query_tags.get_query_comment(
      node,
      extra={
        'workflow_id': env_var('AIRFLOW_CTX_DAG_ID', 'default_dag_id'),
        'job_id': env_var('AIRFLOW_CTX_TASK_ID', 'default_taks_id'),
        'scheduler': 'airflow'
      }
    ) }}
  append: true # Snowflake removes prefixed comments.

Github Actions

query-comment:
  comment: >
    {{ dbt_snowflake_query_tags.get_query_comment(
      node,
      extra={
        'workflow_id': env_var('GITHUB_WORKFLOW', 'default_dag_id'),
        'job_id': env_var('GITHUB_JOB', 'default_taks_id'),
        'scheduler': 'github_actions'
      }
    ) }}
  append: true # Snowflake removes prefixed comments.

Gitlab

query-comment:
  comment: >
    {{ dbt_snowflake_query_tags.get_query_comment(
      node,
      extra={
        'workflow_id': env_var('CI_PIPELINE_ID', 'default_dag_id'),
        'job_id': env_var('CI_JOB_NAME', 'default_taks_id'),
        'scheduler': 'gitlab'
      }
    ) }}
  append: true # Snowflake removes prefixed comments.

Good luck!