Why I Decided to Give dbt Another Look
Introduction
Back in 2019, I dismissed dbt as another abstraction layer on top of SQL.
At the time, I just didn’t see the value.
Looking back, I think I underestimated where analytics engineering was headed. As I’ve spent more time working on data architecture and thinking about how LLMs interact with data warehouses, a lot of the things I brushed off back then have become much more relevant.
For those less familiar, dbt (Data Build Tool) is an open-source analytics engineering framework that transforms raw warehouse data into tested, documented, version-controlled tables and views using SQL, while automatically managing dependencies, lineage, documentation, and data quality.
Enterprise SQL Environments Without dbt
If you’ve worked in a decent-sized data warehouse, you’ve probably seen metric sprawl happen.
One team defines revenue one way. Another team defines it slightly differently. Before long there are multiple versions of the same KPI floating around the organization, and nobody is completely sure which one should be used.
Eventually leadership starts asking why two dashboards don’t match. The data team then shifts from building new things to explaining why different reports return different numbers. Very frustrating.
I’ve spent a lot of time dealing with that over the years. Looking back, I wish I had adopted something like dbt earlier to manage transformations from raw data into curated marts while keeping documentation close to the code.
Working with dbt
This weekend I finally sat down and built a small analytics stack using DuckDB, Python, dbt, and Prefect.
Python handled loading data into a raw schema. dbt handled the transformations into staging models and marts. Prefect orchestrated the pipeline.
One thing that immediately clicked for me was how dbt tracks dependencies between models. Instead of hardcoding table names all over the place, relationships are defined once and carried throughout the project. I can see how much easier that would make maintaining a warehouse with hundreds of models.
I also liked that testing is part of the normal development workflow instead of being something you remember to do later. Having data quality checks run alongside transformations feels like the right default.
The documentation was probably the biggest surprise. Since models, sources, tests, and descriptions all live together, dbt generates documentation that actually stays current with the project.
I also kept thinking about LLMs while looking through it. Instead of asking a model to figure out your warehouse by reading raw SQL and table names, you can give it documentation that explains what the models are, how they’re related, and what the business definitions actually mean. That’s a much better starting point for an AI than making it guess.
After spending a weekend with it, I understand why dbt has become so common. I don’t really think of it as another SQL abstraction anymore. I think of it as a way to bring some software engineering discipline into analytics.
Conclusion
I think tools like dbt are only going to become more valuable as analytics teams start building more AI applications that integrate with data warehouses.
LLMs don’t just need data. They need context. They need documentation. They need consistent business definitions. Without those things, they’re just generating answers from whatever they can infer.
I will now consider dbt for my projects in the future!