Any experienced data engineer will tell you that efficiency and resource optimization are always top priorities. One powerful feature that can significantly optimize your dbt CI/CD workflow is dbt Slim CI. However, despite its benefits, some limitations have persisted. Fortunately, the recent addition of the --empty flag in dbt 1.8 addresses these issues. In this article, we will share a GitHub Action Workflow and demonstrate how the new --empty flag can save you time and resources.
What is dbt Slim CI?
dbt Slim CI is designed to make your continuous integration (CI) process more efficient by running only the models that have been changed and their dependencies, rather than running all models during every CI build. In large projects, this feature can lead to significant savings in both compute resources and time.
Key Benefits of dbt Slim CI
- Speed Up Your Workflows: Slim CI accelerates your CI/CD pipelines by skipping the full execution of all dbt models. Instead, it focuses on only the modified models and their dependencies and uses the defer flag to pull the unmodified models from production. So, if we have model A, B and C yet only make changes to C, then only model C will be run during the CI/CD process.
- Save Time, Snowflake Credits, and Money: By running only the necessary models, Slim CI helps you save valuable build time and Snowflake credits. This selective approach means fewer computational resources are used, leading to cost savings.
dbt Slim CI flags explained
dbt Slim CI is implemented efficiently using these flags:
--select state:modified: The state:modified selector allows you to choose the models whose "state" has changed (modified) to be included in the run/build. This is done using the state:modified+ selector which tells dbt to run only the models that have been modified and their downstream dependencies.
--state <path to production manifest>: The --state flag specifies the directory where the artifacts from a previous dbt run are stored ie) the production dbt manifest. By comparing the current branch's manifest with the production manifest, dbt can identify which models have been modified.
--defer: The --defer flag tells dbt to pull upstream models that have not changed from a different environment (database). Why rebuild something that exists somewhere else? For this to work, dbt will need access to the dbt production manifest.
You may have noticed that there is an additional flag in the command above.
--fail-fast: The --fail-fast flag is an example of an optimization flag that is not essential to a barebones Slim CI but can provide powerful cost savings. This flag stops the build as soon as an error is encountered instead of allowing dbt to continue building downstream models, therefore reducing wasted builds. To learn more about these arguments you can use have a look at our dbt cheatsheet.
dbt Slim CI with Github Actions before dbt 1.8
The following sample Github Actions workflow below is executed when a Pull Request is opened. ie) You have a feature branch that you want to merge into main.
Workflow Steps
Checkout Branch: The workflow begins by checking out the branch associated with the pull request to ensure that the latest code is being used.
Set Secure Directory: This step ensures the repository directory is marked as safe, preventing potential issues with Git operations.
List of Files Changed: This command lists the files changed between the PR branch and the base branch, providing context for the changes and helpful for debugging.
Install dbt Packages: This step installs all required dbt packages, ensuring the environment is set up correctly for the dbt commands that follow.
Create PR Database: This step creates a dedicated database for the PR, isolating the changes and tests from the production environment.
Get Production Manifest: Retrieves the production manifest file, which will be used for deferred runs and governance checks in the following steps.
Run dbt Build in Slim Mode or Run dbt Build Full Run: If a manifest is present in production, dbt will be run in slim mode with deferred models. This build includes only the modified models and their dependencies. If no manifest is present in production we will do a full refresh.
Grant Access to PR Database: Grants the necessary access to the new PR database for end user review.
Generate Docs Combining Production and Branch Catalog: If a dbt test is added to a YAML file, the model will not be run, meaning it will not be present in the PR database. However, governance checks (dbt-checkpoint) will need the model in the database for some checks and if not present this will cause a failure. To solve this, the generate docs step is added to merge the catalog.json from the current branch with the production catalog.json.
Run Governance Checks: Executes governance checks such as SQLFluff and dbt-checkpoint.
Problems with the dbt CI/CD Workflow
As mentioned in the beginning of the article, there is a limitation to this setup. In the existing workflow, governance checks need to run after the dbt build step. This is because dbt-checkpoint relies on the manifest.json and catalog.json. However, if these governance checks fail, it means that the dbt build step will need to run again once the governance issues are fixed. As shown in the diagram below, after running our dbt build, we proceed with governance checks. If these checks fail, we need to resolve the issue and re-trigger the pipeline, leading to another dbt build. This cycle can lead to unnecessary model builds even when leveraging dbt Slim CI.
Leveraging the --empty Flag for Efficient dbt CI/CD Workflows
The solution to this problem is the --empty flag in dbt 1.8. This flag allows dbt to perform schema-only dry runs without processing large datasets. It's like building the wooden frame of a house—it sets up the structure, including the metadata needed for governance checks, without filling it with data. The framework is there, but the data itself is left out, enabling you to perform governance checks without completing an actual build.
Let’s see how we can rework our Github Action:
Workflow Steps
Checkout Branch: The workflow begins by checking out the branch associated with the pull request to ensure that the latest code is being used.
Set Secure Directory: This step ensures the repository directory is marked as safe, preventing potential issues with Git operations.
List of Files Changed: This step lists the files changed between the PR branch and the base branch, providing context for the changes and helpful for debugging.
Install dbt Packages: This step installs all required dbt packages, ensuring the environment is set up correctly for the dbt commands that follow.
Create PR Database: This command creates a dedicated database for the PR, isolating the changes and tests from the production environment.
Get Production Manifest: Retrieves the production manifest file, which will be used for deferred runs and governance checks in the following steps.
*NEW* Governance Run of dbt (Slim or Full) with EMPTY Models: If there is a manifest in production, this step runs dbt with empty models using slim mode and using the empty flag. The models will be built in the PR database with no data inside and we can now use the catalog.json to run our governance checks since the models. Since the models are empty and we have everything we need to run our checks, we have saved on compute costs as well as run time.
Generate Docs Combining Production and Branch Catalog: If a dbt test is added to a YAML file, the model will not be run, meaning it will not be present in the PR database. However, governance checks (dbt-checkpoint) will need the model in the database for some checks and if not present this will cause a failure. To solve this, the generate docs step is added to merge the catalog.json from the current branch with the production catalog.json.
Run Governance Checks: Executes governance checks such as SQLFluff and dbt-checkpoint.
Run dbt Build: Runs dbt build using either slim mode or full run after passing governance checks.
Grant Access to PR Database: Grants the necessary access to the new PR database for end user review.
By leveraging the dbt --empty flag, we can materialize models in the PR database without the computational overhead, as the actual data is left out. We can then use the metadata that was generated during the empty build. If any checks fail, we can repeat the process again but without the worry of wasting any computational resources doing an actual build. The cycle still exists but we have moved our real build outside of this cycle and replaced it with an empty or fake build. Once all governance checks have passed, we can proceed with the real dbt build of the dbt models as seen in the diagram below.
Conclusion
dbt Slim CI is a powerful addition to the dbt toolkit, offering significant benefits in terms of speed, resource savings, and early error detection. However, we still faced an issue of wasted models when it came to failing governance checks. By incorporating dbt 1.8’s --empty flag into your CI/CD workflows we can reduce wasted model builds to zero, improving the efficiency and reliability of your data engineering processes.
🔗 Watch the vide where Noel explains the --empty flag implementation in Github Actions:
FAQ
How do you handle incremental models in a Slim CI pipeline?
Incremental models require extra care. If an incremental model does not exist yet in the PR-specific schema, the is_incremental flag will be false, causing it to run in full-refresh mode rather than incremental mode. A reliable fix is to clone the relevant incremental models into the PR schema as a first step using dbt clone, so the models already exist and will behave correctly during the build.
How does dbt Slim CI work with dbt Core vs. dbt Cloud?
In dbt Cloud, Slim CI is largely built-in. You configure a PR job to defer to a production environment, and dbt Cloud handles artifact storage automatically. With dbt Core, you have to manage artifact persistence yourself. dbt Core users need to store the production manifest.json in remote object storage like S3 and retrieve it at the start of each CI pipeline run. Datacoves provides pre-built CI/CD templates for GitHub Actions that handle this automatically, removing the manual setup burden.
What are the key flags used to implement dbt Slim CI?
Three flags do the heavy lifting:--select state:modified+ selects only modified models and their downstream dependencies.--state points to the production manifest.json so dbt can identify what changed.--defer tells dbt to pull unmodified upstream models from production instead of rebuilding them.An optional --fail-fast flag stops the build at the first error, preventing wasted builds from running downstream models unnecessarily.
What does the --defer flag do in a Slim CI workflow?
When --defer is enabled, dbt resolves ref() calls using the state manifest instead of the current target, but only when the model does not exist in the current environment. This means a developer can test a single changed model without rebuilding every upstream parent. Unbuilt upstream models fall back to their production versions automatically unless they exist in the current database.When --favor-state is passed, dbt prioritizes node definitions from the --state directory even if the models exist in the current database.
What happens when Slim CI triggers a false positive and rebuilds unchanged models?
This was a known issue before dbt 1.9. Dynamic configurations, like setting a database name based on the environment, could make dbt perceive model changes that did not actually occur. In dbt 1.9, the state:modified selector was improved to compare unrendered configuration values, eliminating false positives and ensuring only genuinely modified models are selected for rebuilding.
What is dbt Slim CI?
dbt Slim CI is a CI/CD optimization that builds and tests only the dbt models that changed in a pull request, along with their downstream dependencies. Instead of rebuilding every model on every PR, it uses the --select state:modified+ flag to compare the current branch against a production manifest and run only what actually changed. On large projects this leads to significant savings in both compute resources and time.
What is the dbt production manifest and why does Slim CI need it?
The manifest.json is a file generated every time dbt runs. It captures the full state of your project: models, tests, macros, and how they connect. Slim CI compares a current branch manifest against a production one to determine which parts of the project have changed. Without a production manifest, dbt has no baseline to diff against and cannot know which models are modified.
What problem does the --empty flag solve in dbt 1.8+?
Before dbt 1.8, governance checks had to run after a full dbt build, because tools like dbt-checkpoint need the catalog.json to run. If a governance check failed, the entire build had to re-run, wasting compute. The --empty flag allows dbt to perform schema-only dry runs without processing large datasets, setting up the structure and metadata needed for governance checks without filling models with data. This breaks the costly rebuild cycle and brings wasted model builds to zero.
Where should the production manifest be stored for dbt Core Slim CI?
The manifest.json needs to be accessible to your CI pipeline at runtime. The most common approach is to store it in a cloud storage bucket such as AWS S3 or Google Cloud Storage after each production run, then retrieve it at the start of each PR pipeline. In a CI/CD invocation context, you typically will not be running jobs on a machine with persistent storage across builds, making remote object storage the standard approach. Datacoves also has a dbt-api where dbt artifacts may be stored and later retreived for deferal and Airflow runs.
