Datacoves blog

Companies are investing heavily to become data-driven and to democratize data access. However, many are not achieving the transformative outcomes they expected.

The core issue? A lack of trust.

This mistrust stems from a lack of focus on core aspects that ensure a robust data-driven culture and critical mistakes in these areas.

Fortunately, these mistakes are self-inflicted which means they can be fixed, and this article aims to help highlight and address these pitfalls. By understanding and adhering to the core pillars of a data-driven culture and avoiding the common mistakes, organizations can develop and maintain a data-driven culture that people can trust.

What is data-driven culture?

It is no secret that there is power and opportunity in data, and data-driven culture is the approach which aims to take advantage of that.

A data-driven culture is not about hastily adopting the latest tools or technologies in the hope of resolving data challenges. This common mistake often leads to a focus on immediate results or 'shiny objects', such as acquiring cutting-edge technology or hiring new talent. Unfortunately, this approach tends to overlook essential priorities and gradually erodes the foundation of a data-driven culture: Trust in the data.

Many companies struggle with effectively using analytics because they overemphasize these immediate goals – the 'destination' – rather than appreciating the foundational journey necessary for impactful analytics. This journey involves more than just technology; it requires a shift in mindset and approach.

Data-driven culture represents an organizational approach where data is the cornerstone of decision-making processes. In such a culture, decisions are primarily informed by data analysis, rather than relying exclusively on intuition or past experiences. This approach involves strategically employing data at every level of the organization. It fosters an environment where data is not just an asset but the main driver of strategy, innovation, and operational choices. By harnessing the power and opportunities offered by data, a data-driven culture ensures that decisions across the organization are grounded in solid evidence and analytical insight, enhancing the overall decision-making quality and efficacy.

Key features of data-driven culture include:

Empowered Decision Making: Decisions are based on data analysis, leading to objective and impactful outcomes.

Accessibility of Data: Data is accessible across the organization, breaking down silos and empowering all employees.

Investment in Technology: Adequate tools and technologies are provided for effective data collection and analysis.

Data Literacy: Continuous training is provided to enhance the workforce's understanding and use of data.

Quality and Governance: High standards of data accuracy and security are maintained.

Agility: The organization adapts quickly to insights derived from data.

Collaborative Integration: Data insights are shared and integrated across various functions.

Outcome-Focused: Emphasis on measurable results driven by data insights.

Building a data-driven culture: Core pillars

All of that sounds great, but how do we achieve a data-driven culture?

Like mentioned earlier in the article, true success in analytics comes not from merely chasing new tools or methodologies but from establishing three core pillars as part of a Data-Driven Culture:

• Fundamental Alignment: It's essential to align analytics strategies with core business objectives, ensuring everyone involved shares a common vision and understanding.
• User-Focused Solutions: The end goal of analytics should be to serve the user's needs. This involves designing solutions that are practical, add real value, and enhance the decision-making process.
• Efficient Data Management: Implementing robust data processes is key. This involves ensuring data accuracy, accessibility, and understandability, which are crucial for informed decision-making.

By refocusing on these foundational elements, businesses can drive more meaningful and sustainable results from their analytic endeavors, leading to overall project success and satisfaction.  

Let's dive deeper into the core pillars and examine the common pitfalls within each pillar that I have observed lead to challenges.

Lack of alignment reduces faith in the solution: Fundamental alignment

Fundamental alignment is about synchronizing analytics strategies with the organization's core business objectives. This ensures everyone involved, from executives to frontline employees, share a common vision and understanding of what analytics aims to achieve. This alignment is crucial for creating a unified direction in data-driven initiatives and ensuring that every analytics effort contributes meaningfully to the overall business strategy.

This sounds great right? So much so that every project I've participated in began with high hopes and enthusiasm. Initially, there was a sense of unity – funding secured, partnerships with vendors established, and the latest technology acquired. This honeymoon phase of the data driven transformation, filled with optimism, had everyone working diligently, with management receiving regular updates and a general belief that we were on the right track.

Pitfall

The real test emerged when critical decisions were required. This was the point where the honeymoon phase often faded, revealing a lack of true alignment. Meetings became prolonged discussions where the team struggled to reach consensus. This challenge stemmed from either not spending enough time initially to ensure everyone was on the same page or not conducting a discovery phase at the start of the project.

Although we agreed on high-level objectives like digital transformation and self-service analytics, there was a misalignment in our deeper understanding and perspectives. We were each influenced by our varied backgrounds and expertise in different aspects of the project.

‍

Solution

This led me to a crucial fact: the importance of alignment before action. In projects where we dedicated time upfront for structured alignment and thorough product discovery, we not only achieved better estimations but also greater overall satisfaction. It became evident that successful analytics projects require a deep understanding of business objectives, the current state, potential risks, and a clear prioritization of features. This was because we developed a clear understanding and set up expectations that people could rely on throughout the course of implementation.

Crucially, alignment also involves clarity on what the project will not address, alongside the criteria for prioritization such as quality, completeness of features, and usability. Embracing agility does not mean forgoing thorough planning.

Ultimately, building trust in any project begins with listening, creating a shared vision, and setting the right expectations from the start. A well-defined and achievable plan, understood and agreed upon by all, is the foundation of success.

Bad user experiences erode confidence: User-focused solutions  

The end goal of analytics should be to serve the user's needs and involve designing practical solutions that add real value and enhance decision-making processes. This means creating analytics tools and processes that are intuitively aligned with how users work and make decisions, ensuring that these tools are not just technically proficient but also practically useful.

Pitfall

There are two pitfalls to avoid.

1. Trying to please everyone often leads to pleasing no one.

This is a common scenario in many companies where technical teams strive to meet all demands. Despite their efforts to deliver on projects, dissatisfaction with the end results is frequent.

2. Not addressing the actual user pain points.

This happens when the user does not actually get a good working solution out of the process.

Solution

During discovery it is important to discuss what is in scope, out of scope, essential, and nice to have. By categorizing this way you can better understand the needs of the group and use it to guide the process. With this process done, you can move forward with confidence that you are addressing the most important pain points.

Now that we have defined the pain points, the next step is to fully understand. The key is to not only understand their needs but the reasons behind them. What are the goals they're trying to achieve? What are the shortcomings of their current methods? Is the new process or tool genuinely an improvement over what they currently have?  For example, if users need to navigate a tool quickly, finding ways to reduce unnecessary clicks and simplifying access becomes important. Sometimes, it's more practical to keep an existing process unchanged until other parts are enhanced. By bringing the most critical information to the forefront, the solution becomes more user centric.

It is important to have these needs in mind at the beginning of the project and strive to truly understand. If not, you risk investing time, money, and resources in a tool that users don't need, and this can have a detrimental effect on the overall culture.

More importantly, this approach allows you to justify your decisions and explain why certain aspects are prioritized over others. When users see that their needs and challenges are understood and addressed, they are more likely to trust and accept the solutions provided. This trust is built through consistently demonstrating that their best interests are at heart.  

Data driven transformation: Efficient data management

Efficient data management involves implementing robust processes to ensure data accuracy, accessibility, and understandability. This pillar is key to informed decision-making as it underpins the reliability of data-driven insights. Effective data management includes organizing, storing, and safeguarding data to make it readily available and useful for users across the organization.

Pitfall

Let's face it, your data processes get no love. This is usually because they are "too technical." Users often do not concern themselves with databases, schemas, tables, or columns, let alone the process that turns raw facts into business-ready insights. It is easy for management to get excited about a fancy dashboard and the potential of Machine Learning and Gen AI, but when it comes to the actual data, interest tends to wane.

It makes sense; most people don't understand how the power grid works. We take it for granted that we flip a switch and expect the lights to turn on. We move on without a second thought. No one really cares about electricity until something goes wrong. Similarly, in many organizations, data issues often go unnoticed until a failure occurs. Sometimes these issues are immediately apparent, but other times they are silent. When a failure does happen, there is a scramble to fix it. Meetings are held, issues are identified, and patches are implemented to "prevent" future failures. However, the best time to think about potential problems isn't after they happen, but before — building systems that anticipate and are designed for resilience.

The real issue is that the process from raw data to insights isn't often viewed as a single system. It is all interconnected and should be treated as such. In the world of analytics, it sometimes feels like companies are trying to build a mansion on a foundation of quicksand. Initially, everything seems fine, and everyone is busy with their tasks, but when the foundation starts to give way, the focus shifts to propping up the weak points. You can't effectively build on quicksand; you need solid, repeatable processes from the start.

Solution

The focus should be on building systems that anticipate challenges and are designed for resilience. This involves integrating data management practices into the company's culture from the start, ensuring users trust the data and the processes that generate insights. If you want effective collaboration and impact analysis, these are difficult to retrofit later — they need to be part of the initial plan. Documented analytics isn't a magical solution; it needs to be ingrained in the culture and process from the beginning. The good news is that there are many examples and best practices from those who have navigated these challenges successfully.

For users to truly trust in analytics, they need to have faith in the data and the processes that generate it. They need to see and believe in a robust system built on a solid foundation.

Conclusion  

To achieve a data-driven culture, companies must refocus on three core pillars: fundamental alignment, user-focused solutions, efficient data management, and avoid common mistakes in these areas. Success in analytics isn't about chasing new tools or methodologies but about building a robust system from the ground up, aligning everyone's vision, and creating practical, value-added solutions. Prioritizing foundational elements over immediate shiny objects will lead to more meaningful, sustainable results and will build trust in the analytics process.

As the world of data management continues to grow, terms and new concepts are constantly popping up. It's important for data professionals to stay up to date with terms such as Data Mesh and data observability. For those coming into the field from other areas, it’s also good to understand terminology to communicate more effectively with others.

In this blog post, we've put together an extensive table that breaks down and explains the essential terms in modern data engineering, analytics, and architecture. This resource is designed to help both experienced data professionals and newcomers alike to navigate and understand the ever-evolving language of data.

Glossary

We've covered basic concepts like data warehouses and ETL pipelines and advanced ideas like Data Mesh. Each of these terms is crucial in shaping today's data ecosystems. Think about how these terms apply to your business and can enhance your understanding. Have we missed any terms that you were hoping to see defined, or do you think we could improve the definitions of some of the terms already defined? Please share your thoughts with us by providing feedback through our contact page.

Interested in modern data solutions? Accelerate your journey to a modern data stack with Datacoves' managed solution, designed to streamline your data processes and implement best practices efficiently. Discover how Datacoves can help you quickly add value and transform your data strategy, ensuring you make the most informed decisions for your specific needs, by scheduling a demo.

dbt is wildly popular and has become a fundamental part of many data stacks. While it’s easy to spin up a project and get things running on a local machine, taking the next step and deploying dbt to production isn’t quite as simple.  

In this article we will discuss options for deploying dbt to production, comparing some high, medium, and low effort options so that you can find which works best for your business and team. You might be deploying dbt using one of these patterns already; if you are, hopefully this guide will help highlight some improvements you can make to your existing deployment process.  

We're going to assume you know how to run dbt on your own computer (aka your local dbt setup).  We’re also going to assume that you either want to or need to run dbt in a “production” environment – a place where other tools and systems make use of the data models that dbt creates in your warehouse.

Enhancing understanding of dbt deployment

The deployment process for dbt jobs extends beyond basic scheduling and involves a multifaceted approach. This includes establishing various dbt environments with distinct roles and requirements, ensuring the reliability and scalability of these environments, integrating dbt with other tools in the (EL)T stack, and implementing effective scheduling strategies for dbt tasks. By focusing on these aspects, a comprehensive and robust dbt deployment strategy can be developed. This strategy will not only address current data processing needs but also adapt to future challenges and changes in your data landscape, ensuring long-term success and reliability.

dbt environments

In deploying dbt you have the creation and management of certain dbt environments. The development environment is the initial testing ground for creating and refining dbt models. It allows for experimentation without impacting production data. Following this, the testing environment, including stages like UAT and regression testing, rigorously evaluates the models for accuracy and performance. Finally, the production environment is where these models are executed on actual data, demanding high stability and performance.

Data reliability and scalability

Reliability and scalability of data models are also important. Ensuring that the data models produce accurate and consistent results is essential for maintaining trust in your data. As your data grows, the dbt deployment should be capable of scaling, handling increased volumes, and maintaining performance.

End-to-End data pipeline integration

Integration with other data tools and systems is another key aspect. A seamless integration of dbt with EL tools, data visualization platforms, and data warehouses ensures efficient data flow and processing, making dbt a harmonious component of your broader data stack.

dbt job scheduling

Effective dbt scheduling goes beyond mere time-based scheduling. It involves context-aware execution, such as triggering jobs based on data availability or other external events. Managing dependencies within your data models is critical to ensure that transformations occur in the correct sequence. Additionally, adapting to varying data loads is necessary to scale resources effectively and maintain the efficiency of dbt job executions.

The main flavors for dbt deployment are:  

• Cron Job
• Cloud Runner Service (like dbt Cloud)
• Integrated Platform (like Datacoves)
• Fully Custom (like Airflow, Astronomer, etc)

They each have their place, and the trade-offs between setup costs and long-term maintainability is important to consider when you’re choosing one versus another.  

We can compare these dbt deployment options across the following criteria:

• Ease of Use / Implementation
• Required Technical Ability
• Configurability
• Customization
• Best for End-to End Deployment

Cron job

Cron jobs are scripts that run at a set schedule. They can be defined in any language. For instance, we can use a simple bash script to run dbt. It’s just like running the CLI commands, but instead of you running them by hand, a computer process would do it for you.  

Here’s a simple cron script:

In order to run on schedule, you’ll need to add this file to your system’s crontab.

As you can tell, this is a very basic dbt run script; we are doing the bare minimum to run the project. There is no consideration for tagged models, test, alerting, or more advanced checks.  

Even though Cron jobs are the most basic way to deploy dbt there is still a learning curve. It requires some technical skills to set up this deployment. Additionally, because of its simplicity, it is pretty limited. If you are thinking of using crons for multi-step deployments, you might want to look elsewhere.  

While it's relatively easy to set up a cron job to run on your laptop this defeats the purpose of using a cron altogether. Crons will only run when the daemon is running, so unless you plan on never turning off your laptop, you’ll want to set up the cron on an EC2 instance (or another server). Now you have infrastructure to support and added complexity to keep in mind when making changes. Running a cron on an EC2 instance is certainly doable, but likely not the best use of resources. Just because it can be done does not mean it should be done. At this point, you’re better off using a different deployment method.

The biggest downside, however, is that your cron script must handle any edge cases or errors gracefully.  If it doesn’t, you might wind up with silent failures – a data engineer’s worst enemy.  

Who should use Cron for dbt deployment?

Cron jobs might serve you well if you have some running servers you can use, have a strong handle on the types of problems your dbt runs and cron executions might run into, and you can get away with a simple deployment with limited dbt steps. It is also a solid choice if you are running a small side-project where missed deployments are probably not a big deal.  

Use crons for anything more complex, and you might be setting yourself up for future headaches.  

Ease of Use / Implementation – You need to know what you’re doing

Required Technical Ability – Medium/ High

Configurability – High, but with the added complexity of managing more complex code

Customization – High, but with a lot of overhead. Best to keep things very simple  

Best for End-to-End Deployment - Low.

Cloud Service Runners (like dbt Cloud)

Cloud Service Runners like dbt Cloud are probably the most obvious way to deploy your dbt project without writing code for those deployments, but they are not perfect.  

dbt Cloud is a product from dbt Labs, the creators of dbt. The platform has some out-of-the-box integrations, such as Github Actions and Webhooks, but anything more will have to be managed by your team. While there is an IDE (Integrated Developer Experience) that allows the user to write new dbt models, you are adding a layer of complexity by orchestrating your deployments in another tool. If you are only orchestrating dbt runs, dbt Cloud is a reasonable choice – it's designed for just that.  

However, when you want to orchestrate more than just your dbt runs – for instance, kickoff multiple Extract / Load (EL) processes or trigger other jobs after dbt completes – you will need to look elsewhere.  

dbt Cloud will host your project documentation and provide access to its APIs. But that is the lion’s share of the offering. Unless you spring for the Enterprise Tier, you will not be able to do custom deployments or trigger dbt runs based on incoming data with ease.  

Deploying your dbt project with dbt Cloud is straightforward, though. And that is its best feature. All deployment commands use native dbt command line syntax, and you can create various "Jobs" through their UI to run specific models at different cadences.

Who should use dbt Cloud for dbt deployment?

If you are a data team with data pipelines that are not too complex and you are looking to handle dbt deployments without the need for standing up infrastructure or stringing together advanced deployment logic, then dbt Cloud will work for you. If you are interested in more complex triggers to kickoff your dbt runs - for instance, triggering a run immediately after your data is loaded – there are other options which natively support patterns like that. The most important factor is the complexity of the pieces you need to coordinate, not necessarily the size of your team or organization.  

Overall, it is a great choice if you’re okay working within its limitations and support a simple workflow. As soon as you reach any scale, however, the cost may be too high.  

Ease of Use / Implementation – Very easy

Required Technical Ability – Low

Configurability – Low / Medium

Customization – Low

Best for End-to-End Deployment - Low

Integrated platform (like Datacoves)

The Modern Data Stack is a composite of tools. Unfortunately, many of those tools are disconnected because they specialize in handling one of the steps in the ELT process. Only after working with them do you realize that there are implicit dependencies between these tools. Tools like Datacoves bridge the gaps between the tools in the Modern Data Stack and enable some more flexible dbt deployment patterns. Additionally, they cover the End-to-End solution, from Extraction to Visualization, meaning it can handle steps before and after Transformation.

Coordinating dbt runs with EL processes

If you are loading your data into Snowflake with Fivetran or Airbyte, your dbt runs need to be coordinated with those EL processes. Often, this is done by manually setting the ETL schedule and then defining your dbt run schedule to coincide with your ETL completion. It is not a hard dependency, though. If you’re processing a spike in data or running a historical re-sync, your ETL pipeline might take significantly longer than usual. Your normal dbt run won’t play nicely with this extended ETL process, and you’ll wind up using Snowflake credits for nothing.

This is a common issue for companies moving from early stage / MVP data warehouses into more advanced patterns. There are ways to connect your EL processes and dbt deployments with code, but Datacoves makes it much easier. Datacoves will trigger the right dbt job immediately after the load is complete. No need to engineer a solution yourself. The value of the Modern Data Stack is being able to mix and match tools that are fit for purpose.

Seamless integration and orchestration

Meeting established data freshness and quality SLAs is challenging enough, but with Datacoves, you’re able to skip building custom solutions for these problems. Every piece of your data stack is integrated and working together. If you are orchestrating with Airflow, then you’re likely running a Docker container which may or may not have added dependencies. That’s one common challenge teams managing their own instances of Airflow will meet, but with Datacoves, container / image management and synchronization between EL and dbt executions are all handled on the platform. The setup and maintenance of the scalable Kubernetes infrastructure necessary to run Airflow is handled entirely by the Datacoves platform, which gives you flexibility but with a lower learning curve. And, it goes without saying that this works across multiple environments like development, UAT, and production.

Streamlining the Datacoves experience

With the End-to-End Pipeline in mind, one of the convenient features is that Datacoves provides a singular place to access all the tools within your normal analytics workflow - extraction + load, transformation, orchestration, and security controls are in a single place. The implicit dependencies are now codified; it is clear how a change to your dbt model will flow through to the various pieces downstream.  

Who should use Datacoves for dbt deployment?

Datacoves is for teams who want to introduce a mature analytics workflow without the weight of adopting and integrating a new suite of tools on their own. This might mean you are a small team at a young company, or an established analytics team at an enterprise looking to simplify and reduce platform complexity and costs.

There are some prerequisites, though. To make use of Datacoves, you do need to write some code, but you’ll likely already be used to writing configuration files and dbt models that Datacoves expects. You won't be starting from scratch because best practices, accelerators, and expertise are already provided.

Ease of Use / Implementation – You can utilize YAML to generate DAGS for a simpler approach, but you also have the option to use Python DAGS for added flexibility and complexity in your pipelines.

Required Technical Ability – Medium

Configurability – High

Customization – High. Datacoves is modular, allowing you to embed the tools you already use

Best for End-to-End Deployment - High. Datacoves takes into account all of the factors of dbt Deployment

Fully custom (like Airflow, Dagster, etc)

What do you use to deploy your dbt project when you have a large, complex set of models and dependencies? An orchestrator like Airflow is a popular choice, with many companies opting to use managed deployments through services such as Astronomer.  

For many companies – especially in the enterprise – this is familiar territory. Adoption of these orchestrators is widespread. The tools are stable, but they are not without some downsides.  

These orchestrators require a lot of setup and maintenance. If you’re not using a managed service, you’ll need to deploy the orchestrator yourself, and handle the upkeep of the infrastructure running your orchestrator, not to mention manage the code your orchestrator is executing. It’s no small feat, and a large part of the reason that many large engineering groups have dedicated data engineering and infrastructure teams.  

Running your dbt deployment through Airflow or any other orchestrator is the most flexible option you can find, though. The increase in flexibility means more overhead in terms of setting up the systems you need to run and maintain this architecture. You might need to get DevOps involved, you’ll need to move your dbt project into a Docker image, you’ll want an airtight CI/CD process, and ultimately have well defined SLAs. This typically requires Docker images, container management, and some DevOps work. There can be a steep learning curve, especially if you’re unfamiliar with what’s needed to take an Airflow instance to a stable production release.

There are 3 ways to run Airflow, specifically – deploying on your own, using a managed service, or using an integrated platform like Datacoves. When using a managed service or an integrated platform like Datacoves, you need to consider a few factors:

1. Airflow use cases

2. Ownership and contributing teams

3. Integrations with rest of your stack

‍

Airflow use cases

Airflow is a multi-purpose tool. It’s not just for dbt deployments. Many organizations run complex data engineering pipelines with Airflow, and by design, it is flexible. If your use of Airflow extends well beyond dbt deployments or ELT jobs oriented around your data warehouse, you may be better suited for a dedicated managed service.

Ownership and contributing teams

Similarly, if your organization has numerous teams dedicated to designing, building and maintaining  your data infrastructure, you may want to use a dedicated Airflow solution. However, not every organization is able to stand up platform engineering teams or DevOps squads dedicated to the data infrastructure. Regardless of the size of your team, you will need to make sure that your data infrastructure needs do not outmatch your team’s ability to support and maintain that infrastructure.

Integrations with the rest of your stack

Every part of the Modern Data Stack relies on other tools performing their jobs; data pipelines, transformations, data models, BI tools - they are all connected. Using Airflow for your dbt deployment adds another link in the dependency chain. Coordinating dbt deployments via Airflow can always be done through writing additional code, but this is an additional overhead you will need to design, implement, and maintain. With this approach, you begin to require strong software engineering and design principles. Your data models are only as useful as your data is fresh; meeting your required SLAs will require significant cross-tool integration and customization.  

Who should use a fully custom setup for dbt deployment?

If you are a small team looking to deploy dbt, there are likely better options. If you are a growing team, there are certainly simpler options with less infrastructure overhead. For Data teams with complex data workflows that combine multiple tools and transformation technologies such as Python, Scala, and dbt, however, Airflow and other orchestrators can be a good choice.  

Ease of Use / Implementation – Can be quite challenging starting from scratch

Required Technical Ability – High

Configurability – High

Customization – High, but build time and maintenance costs can be prohibitive ‍

Best for End-to-End Deployment - High, but requires a lot of resources to set up and maintain

dbt Deployment Options Overview

Final thoughts

The way you should deploy your dbt project depends on a handful of factors – how much time you’re willing to invest up front, your level of technical expertise, as well as how much configuration and customization you need.  

Small teams might have high technical acumen but not enough capacity to manage a deployment on their own. Enterprise teams might have enough resources but maintain disparate, interdependent projects for analytics. Thankfully, there are several options to move your project beyond your local and into a production environment with ease. And while specific tools like Airflow have their own pros and cons, it’s becoming increasingly important to evaluate your data stack vendor solution holistically. Ultimately, there are many ways to deploy dbt to production, and the decision comes down to spending time building a robust deployment pipeline or spending more time focusing on analytics.  

Digital transformation is often seen through the lens of technological advancement and process optimization. Most blog posts and guides out there revolve around implementing new software, automating tasks, and digitizing operations. Yet, there's a pivotal element that's frequently overlooked in these discussions, especially when it comes to an enterprise: the mindset and culture within an organization. This article aims to shed light on why this is crucial in achieving true digital transformation. But first, let's investigate what digital transformation is and why it is important.

Digital transformation defined

Digital transformation is the integration of digital technology into all areas of a business, fundamentally changing how it operates and delivers value to customers. It is more than just a technological upgrade; it is a cultural shift that requires organizations to continually challenge the status quo, experiment, and get comfortable with failure. This often means walking away from long-standing business processes that companies were built upon to embrace new ways of working. Most organizations find this part the most challenging.

Why is digital transformation important

• Keeping Up with the Digital Economy: In a world where technology evolves rapidly, businesses must adapt to stay relevant. Digital transformation allows companies to remain competitive in an increasingly digital economy.
• Enhanced Data Collection and Analysis: Digital transformation creates a system for gathering the right data and fully utilizing it for better business decisions, efficiencies, and customer insights.
• Customer Expectations: Today's customers expect a seamless digital experience. Businesses need to engage with customers on their terms, using digital tools and platforms that are convenient and user-friendly.
• Increased Agility and Innovation: Adopting digital solutions empowers organizations to be more agile and responsive to changes in the marketplace or industry. It fosters a culture of innovation, encouraging new ideas and approaches.
• Operational Efficiency: Automation and streamlining of processes reduce operational costs and improve efficiency. This allows employees to focus on more strategic tasks that add value to the business.
• Risk Management and Compliance: With the increasing importance of data security and privacy, digital transformation helps businesses keep up with changing regulations and protect sensitive information.
• Sustainability: Digital processes can reduce waste and improve energy efficiency, contributing to more sustainable business practices.

Enterprise digital transformation

To achieve digital transformation in an enterprise 9 times out of 10 there must be a change in company culture. However, changing a company's culture is a formidable task. It is rare to hear statements like, “We need to fundamentally change our problem-solving approach.” This realization became clear to me through my past experiences as I noticed that managers often lacked the influence to drive change at the highest organizational levels. Additionally, the pressure to deliver quick results within budget cycles frequently hindered genuine cultural transformation.

During my tenure at various companies, under numerous managers, the consistent message was the need for improvement. However, I have come to understand that organizations, much like fireflies, develop their own rhythms. It is this unique rhythm that sets apart innovative and transformative companies from those that merely follow without achieving similar success. What do I mean by this? Let’s turn to nature for an explanation.

Firefly phenomenon - Does it mean conformity or innovation in your organization?

Nature is fascinating, especially when observing how hundreds or thousands of fireflies can synchronize their flashes.

In organizations, a similar phenomenon occurs. People will sync up and follow the status quo, even if it is not what is best for the organization. This dramatically hinders digital transformation because the loudest are not always right and yet they cause others to sync up with them. This will cause innovation to be stopped in its tracks.

In addition to this firefly phenomenon, often action differs from ambition. I recall a staff meeting with a former CIO discussing a future less dependent on Microsoft and more open to non-Windows devices. It was clear that iPhones were going to change the corporate landscape. Despite this, every new tool implemented was still optimized for Internet Explorer. This discrepancy between ambition and action often drives analytical people like me to frustration. To effect change, persistence is key. I have had ideas initially dismissed as “not my job,” only to see one later turn into a patented invention.

This manifests itself in other ways as well; have you ever seen a company advocate for fewer meetings while simultaneously criticizing those who do not include “everyone” in decision-making? I have been in such situations and can attest that decision-making by committee is not inherently superior. In fact, the more people involved in an initiative, the less effective it tends to be. This, I believe, is due to the Dunning-Kruger effect.

The more people you involve in a transformation initiative, the more likely the discussions will deteriorate to bike shedding discussions. When there is a disconnect between what is said and what is done, people take notice, and it breeds discontent.

One firefly can only affect their neighbors

Even in my most successful transformation initiatives, the radius of transformation has been limited to my sphere of influence. Sure, some of my tools and processes got global and cross-functional acceptance, but the underlying principles never took hold because they were too radical for the organization at the time. I was not part of the IT organization so the things I did were typically seen as shadow IT. Instead of focusing on what I should not be doing, it would have been more progressive for them to see how I was practicing Agile principles. They could have inquired about how my project was doing DevOps before that was in style, or how it was that this non-sanctioned product was extremely well received and people sought me out to help them improve their processes.

This means if you want the organization to be more innovative, you need to find the obstacles that hold people back from being innovative. Often politics and bureaucracy impact an initiative more than the solution itself. If you force everyone to comply with existing tools and processes, then you are imposing a constraint on the team that will limit innovation.

A typical way this manifests itself is leadership pushing the idea that one platform or process can solve every need. This can come in the form of imposing that a particular group do data transformation, or a visualization tool be the way that everyone can do analytics. I have never seen one tool that is good at everything, and you end up balancing the single solution with an unmanageable array of tools and processes. A healthy organization is a learning organization that is always open to improvement.  When management encourages pushing boundaries and not taking anything as fact then the company can innovate.  

A great example of driving innovation is seen in the approach of Steve Jobs, co-founder of Apple Inc. Jobs was known for his ability to challenge conventional wisdom and existing standards in the technology industry. He emphasized the importance of understanding the fundamental principles underlying a problem to innovate and create groundbreaking solutions. One notable instance was the development of the iPhone, which revolutionized the smartphone industry. Jobs and his team did not just improve on existing phones; they rethought what a phone could be, focusing on user experience and simplicity. This approach led to a product that dramatically altered how people interact with technology.  

As a leader, you need to look for the fireflies who are using first principles like Steve Jobs to deliver innovative solutions and nurture, or create, a corporate culture that truly challenges what has been done without artificial constraints.

Reasoning by first principles removes the impurity of assumptions and conventions. What remains is the essentials. It’s one of the best mental models you can use to improve your thinking because the essentials allow you to see where reasoning by analogy might lead you astray.

Most fireflies eventually comply, or fly away – Loss of innovators

The transformative and innovative thinkers will either comply or leave, both of which are undesirable. In my case I tended to leave. In every organization where I have worked, I have managed to make a significant impact, often through sheer determination. During my time at one such company, our goal was to introduce a data catalog. By analyzing the problem I was able to discern what was essential for our organization vs an elaborate and idealistic vision which was capable of doing everything. While the IT organization felt it would be better to create a home-grown catalog I understood that our biggest obstacle was getting people to use a catalog in the first place, so time to market was critical. I found that Alation met the needs we had and IT kept to their vision to build an all encompassing catalog, In 3 months I had deployed Alation and 1.5 years later, the home grown solution was a tenth as good. This approach of breaking down the problem to its basic elements and building up from there was critical. It is often underestimated how challenging it is to develop and maintain custom software.  This experience highlights the effectiveness of first principles thinking in deploying practical and efficient solutions.

The reality is that not everyone possesses the tenacity to advocate for change, especially in the face of substantial resistance. Not only that, but I have also witnessed people being ostracized for thinking differently, while others were promoted for fitting in. It is crucial to seek out divergent thinkers and consider the validity of their perspectives, instead of forcing them to conform. This is why true digital transformation necessitates a shift in culture.  

When an individual, much like a firefly that does not flash in unison with the rest, finds themselves out of sync with the collective rhythm, they face a decision: conform and synchronize with the group or venture out to find a new collective that resonates with their unique spark.

How do we change the flash for all? Aligning mindsets for transformation

True transformational change must come from the top. Achieving enterprise digital transformation requires a deep and bold questioning of the status quo. We must critically assess our processes: Is a particular task truly necessary for a certain group? Can we identify and eliminate inefficiencies? Will adding another layer of approval or inspection genuinely enhance outcomes? It is essential to remember that human behavior often has a more profound impact than any technology or process we implement. When decision-making is centralized within one group, solutions are inevitably skewed to reflect their viewpoint. Too often, I have witnessed decisions justified by cost considerations that, upon closer inspection, proved detrimental in the broader context. An effective strategy involves analyzing the entire system, recognizing that optimizing the whole may require accepting lower efficiency in some areas.

The key is to align with the needs of users and the organization and engage leadership in this journey. With a united front, tackling the 'corporate dragons' becomes a more manageable endeavor. One practical approach is employing methodologies like the 'Job to be Done' framework.

Conclusion

Company culture and change management are frequently overlooked in the pursuit of process improvement. Employees operate within their limitations, while management ponders the lack of innovation and agility compared to other companies. The simpler path might seem to be increasing staff or updating technology, but the heart of transformation lies in the mindset of the organization. Leaders aiming for a lasting impact must embrace first principles thinking, ready to scrutinize and challenge established norms. Transformational change rarely stems from incremental improvements; truly innovative companies are those that dare to think and act differently. The organization thus faces a pivotal choice: will it adapt to a new rhythm, or compel its 'new fireflies' to fall in line with the existing order?

Data teams deciding on data transformation tools need to consider various aspects before deciding on how they will develop and orchestrate data pipelines. They also need to accelerate infrastructure deployment to deliver at the pace the business requires. 

The hurdle to overcome is that doing this well requires a lot of rethinking of legacy processes and technology. 

Implementing DataOps, CI/CD, and setting up an ETL or ELT isn’t a straightforward process, which is why teams often go with an incremental approach or set up the basics and end up with technical debt that accumulates substantially over time. 

In this article, we’ll go through a list of 10 data transformation tools that will help you get the job done. If you are in the process of evaluating your next ETL/ELT platforms, this article for you.

Side Note: As data professionals, we’ve been around since the early days of data transformation and noticed many flaws within the entire process. There’s a steep learning curve: adding a single tool to the workflow can quickly multiply into a tech stack with multiple SaaS platforms. That’s why we built Datacoves to help you bring everything together to accelerate time to value. If you’d like to learn more about how Datacoves helps you develop and orchestrate data pipelines, you can schedule a free demo here. 

Data transformation in the End-to End process

Data transformation is the process of converting data from one format or structure to another. It improves the performance of data processing systems and compliance with data governance regulations.  

Data transformation is just one of the steps on the road to deriving value from data.

The end-to-end process includes the following steps:

• Data Extraction: To extract data from sources such as databases and APIs 
• Data Loading:  To load data into a desired destination such as a Data Warehouse.
• Data Transformation: To cleanse and transform data into usable insights based on business needs. 
• Data Orchestration: To schedule and automate the end-to-end process
• Data Delivery: To visualize and support decision-making 
• Data Observability: To view and get alerted when data issues occur

‍

It’s worth taking each of these steps into consideration when determining the best data transformation tool for your organization.

There is a common misconception that the tool alone will solve all the problems. 

However, using the right tools without addressing the underlying processes can lead to a data mess that can exacerbate the underlying issue, costing more time and money. This data mess could easily be avoided in the first place, not just by having the right tools but by also having the modern best practices in place.

What is the difference between ETL and ELT? 

Both help businesses extract, load, and transform data, but the sequence of events is different with their pros and cons.

• ETL Process: The traditional approach to data transformation. Data is Extracted and Transformed before it gets Loaded. 
• ELT Process: The modern approach to data transformation. Data is Extracted and Loaded before it gets Transformed. 

‍

ELT is generally more effective than ETL processes because it removes the uncertainty of not having the necessary data for future use cases and offers more flexibility in the long term. Since storage is typically affordable, it makes more sense to simplify the ingestion process.

10 best data transformation tools

Here’s a list of the top data transformation tools to manage the ETL process: 

• Datacoves
• dbt Cloud
• Apache Airflow
• SAS
• SQLMesh
• Informatica
• Talend
• Azure Data Factory
• Matillion
• Alteryx

‍

Each of these tools falls into one of two categories: code-based or visual/drag-and-drop interface. Both have their own set of pros and cons, which we’ll go through below. 

Code-based tools for data transformation 

Code-based tools allow you to transform data by using SQL or Python to explicitly define the transformation steps. Although it requires knowledge and experience, visual tools don’t negate the need to know SQL. This approach gives users a high degree of flexibility and control, and simplifies the maintainability and validation of work before releasing it to production. 

Moreover, it is simpler to trace each data transformation step without having a disconnected document explaining what the transformation “should” do.

1. Datacoves

After having multiple conversations with data teams at enterprise companies, the challenge of developing and orchestrating dbt pipelines is a topic that has come up on numerous occasions. 

There are a lot of tools to figure out when it comes to implementing the best practices for digital transformations and custom applications. It’s not uncommon for companies to end up with more than one SaaS platform and tool than they had initially planned. We built Datacoves to eliminate this need by providing the following: 

• Managed dbt Core
• Open-source technologies, meaning there is no vendor lock-in
• Managed SaaS or private cloud deployment

‍

Datacoves focuses on helping companies accelerate growth by providing a complete ELT solution, including orchestration and visualization. Therefore, the learning curve for data transformation is minimized because of our best-practice accelerators and the available tool integrations to form an end-to-end platform. 

Top features

• Managed dbt Core: Get full access to dbt through Datacoves, where we provide a structured process for developing dbt pipelines. Configure a dbt environment where you can write data transformations in modular code so it’s easier to test and maintain. 
• Hosted dbt Documentation: Simplified dbt docs deployment
• Managed Airflow: Orchestrate data using Airflow, which is an integration that’s available to give companies the flexibility they need for an end-to-end process from data load to activation.
• VS Code in Browser: No software installation is required, allowing you to write and edit code from anywhere as long as you have access to a web browser.
• Deployable in Private Cloud: Accelerate data transformation and minimize ownership costs while complying with corporate and governmental regulations. 
• Internal Tool Integrations: Integrate with internal tools like Active Directory, Bitbucket, Jenkins, GitLab, and more. 
• CI/CD: Deliver high-quality data to users faster with more reliability and efficiency. 

How data transformation works with Datacoves

Here is the extended version of the ELT process with Datacoves: 

• Extract data
• Load data
• Transform data
• Observe
• Orchestrate
• Analyze 

Develop modular code and track version changes that you and your team can view. You’re also able to validate the quality of data transformations with our built-in testing frameworks and generate documents to leave a record of how you’re transforming data.

You develop in a VS Code environment that can be configured with a vast array of VS Code extensions and Python libraries All the modern data tools you need are provided in a structured workspace: 

• GitHub workflows
• Automation scripts
• Loading
• Scheduling
• Data security
• Data transformation 

Is Datacoves right for you?

It’s suitable for medium and large companies that lack the expertise or don’t want to create and manage complex data processes and need the flexibility that complex enterprise processes require. 

Data teams can use all the components provided within the dbt ecosystem in a structured, methodical way with Datacoves. This means you’ll have a simplified dbt experience, yet you’ll still see the same results of dbt when used to its full potential. 

Smaller companies also gain competitive advantages with Datacoves because they’ll be able to implement DataOps, follow best practices, and get a fully managed VS Code environment accelerating time to value. 

If you would like to know more about how Datacoves can help, you can schedule a demo here. 

2. dbt Cloud

dbt Cloud allows businesses to build and maintain data pipelines. It’s a cloud-based platform with a web-based IDE that allows you to transform data within a cloud data warehouse. They can help you reduce the time spent setting up an end-to-end solution.

Notable features 

• Modular Code: Write data transformations in modular code. 
• Version Control: Monitor changes to your code and seamlessly collaborate with team members. 
• Data Testing: Built-in testing framework for validating the quality of data transformations. 
• Documentation: Generate documents for data transformations so you know how your data is being transformed. 
• CI/CD: Streamline the data transformation process. 

Is dbt Cloud right for you? 

dbt Cloud works well for organizations looking to reduce the time and effort required to transform data pipelines. 

Since dbt Cloud is a web-based IDE, it may feel limited for data teams that would rather use a VS Code environment. Moreover, dbt is not deployable in a company’s private cloud. It also typically requires other SaaS tools for complicated data pipelines, making it more difficult to manage unless you have the necessary integration experience with each of those SaaS tools. 

Most importantly, dbt Cloud is focused solely on the data transformation step of the ELT process. Hence, you are unable to load VS Code extensions nor additional Python libraries. An enterprise with any level of complexity will also need a full-featured orchestrator. 

‍

3. Apache Airflow

Apache Airflow is an open-source platform for workflow management. You can orchestrate and schedule data pipelines. It’s a scalable and flexible platform that’s based on Python. You can also define your own operators with Airflow. 

Notable features 

• ELT pipelines: Airflow is a tool for organizing full ELT pipelines. But, it still requires your expertise to build them using Python. 
• CI/CD tools: You can use CI/CD tools to assist with deployment. 
• Data Extraction and Load: With airflow, you can create Python scripts that extract and load information from other data sources. 
• Python Libraries: Airflow can be paired with Python libraries like Pandas to shape and aggregate data.
• Machine learning: Schedule the training of machine learning models. 

Is Apache Airflow right for you?

Apache Airflow works well for those needing a scalable data transformation tool with an open-source platform. It’s particularly a good choice for businesses mainly using Python to manage their data. 

However, Airflow is primarily an orchestrator. That means you may end up building complex code in your data pipelines. Therefore, developing and maintaining this complexity requires experience and technical expertise. Managing the infrastructure for Airflow is not trivial and also requires an understanding of tools like Docker and Kubernetes.

4. SAS

SAS is a solution that allows you to transform and prepare data for analysis. It offers a wide range of features for data transformation, including data cleaning, data integration, and data mining.

‍

Notable features

• Data Cleaning: Remove duplicate records, correct errors, complete missing values, and so forth. 
• Data Integration: Combine data from different departments or systems into a single dataset. 
• Data Mining: Identify patterns and trends in your data. 
• Data Visualization: Create charts and graphs to visualize data. 

Is SAS right for you?

SAS is ideal for companies with complex datasets, such as those in financial services, healthcare, and retail industries. Additionally, it’s ideal for professionals with advanced skills and knowledge in data transformation. 

With that in mind, there are better solutions than SAS for those less experienced in programming and data management, as SAS licensing can be quite expensive.  

5. SQLMesh

SQLMesh is a complete DataOps solution for data testing and transformation. Teams can use SQLMesh to collaborate on data pipelines when transforming data. 

Notable features

• Semantic Understanding: SQLMesh can understand the SQL written so you can write code efficiently and avoid errors.
• Simplified CI/CD: SQLMesh can identify the changes made to data pipelines and apply only the necessary updates to each environment.‍
• Column-level Lineage: Get a better understanding of the relationships between your data and the transformation process.‍
• Transpilation: Run your SQL on multiple engines so that it’s easier to migrate data into a new platform.

Is SQLMesh right for you?

SQLMesh is well-suited for businesses with SQL and Python expertise that need to collaborate on complex data transformations and pipelines. Although other open-source tools are available, teams can use SQLMesh to maintain data quality and perform unit testing of their transformations.

SQLMesh may not be ideal when you only need to perform simple data transformations. In this case, there are other more straightforward tools available. Moreover, SQLMesh may not be for you when your primary focus is on real-time data processing.

Visual ELT tools for data transformation

Visual tools make the ELT process more straightforward by removing the need to manually write code. It works by dragging and dropping pre-built components into a canvas. This makes them ideal for data teams who aren’t as experienced in programming. 

The biggest advantage of graphical tools for ETL is that people who are less comfortable with code can use them. Conversely, drag-and-drop tools typically don’t offer the same level of flexibility and control as code-based tools, which can complicate the process of debugging data pipelines and long-term maintenance.

6. Informatica

Informatica helps you turn your data into an asset. It’s a cloud-based or on-prem solution for data management with numerous data transformation libraries and APIs available. 

Notable features 

• PowerCenter: Enterprises can use this to manage large and complex data pipelines. 
• Cloud Data Integration: Cloud-based integrations that allow you to move data. 
• Data Engineering Integration: A solution designed to assist data engineers with code development, version control, and CI/CD.
• Data Engineering Streaming: Manage streaming data pipelines with data ingestion, processing, and visualization. 

Is Informatica right for you?

Informatica can be a good choice for large enterprises and data professionals looking to quickly transform large volumes of complex data using an on-premise solution. It can also be a good choice for companies that need to comply with industry-specific data standards. 

However, it may be too complicated to use for some organizations. Informatica requires a team of experienced data engineers with the necessary skills and experience. DataOps can also be a challenge. Since you’ll be dealing with multiple things simultaneously, it’s easy to get lost in the process when you don’t have the full technical expertise.

Moreover, it’s an expensive solution. There are other more affordable alternatives.

7. Talend

Talend is a cloud-native platform deployable on public cloud solutions such as AWS, Azure, and GCP. They also offer an on-prem solution and provide a variety of components and custom connectors for data transformation. 

Notable features 

• Talend Open Studio: An open-source data integration tool for smaller workloads. 
• Talend Data Fabric: Manage the data integration process. Maintain data quality and governance. 
• Cloud Data Integration: Cloud-based data integration service with a graphical user-friendly interface for creating and managing data transformation tasks. 
• Built-in Data Catalog: Discover new data assets across your organization. 

Is Talend right for you?

Talend works for most businesses and data professionals. It’s particularly well-suited for those who need to:

• Transform data from a variety of sources.
• Migrate data to a new system. 
• Build and maintain a data warehouse. 
• Check and resolve data quality issues 

Still, you may want to consider other options when prioritizing DataOps and performing highly specialized data transformations such as machine learning or NLP. Talend enterprise licenses may also be costly.

8. Azure Data Factory

Azure Data Factory helps you simplify the data transformation process at scale. You’re provided with a code-free and code-centric experience for orchestrating data transformation pipelines. 

Notable features 

• Built-in connectors: A variety of built-in connectors for popular data sources are available. 
• Data Orchestration: Schedule your data pipelines.
• Built-in Components: Use built-in components to reshape data.

Is Azure Data Factory right for you?

Azure Data Factory could be the right option for data professionals working within the Azure ecosystem. Azure may be worth considering when you’re looking into data warehousing using Azure Synapse and Azure DataOps and not just ELT. 

However, Azure Data Factory might not be the best option when you’re on a budget. As with any visual ELT tooling, DataOps and pipeline maintainability may be more complex leading to an increased total cost of ownership.

9. Matillion

Matillion is a cloud-based data transformation tool that provides you with on-premises databases, cloud applications, and SaaS platform integrations. 

Notable features

• Cloud-native architecture: Matillion runs in the cloud and allows you to push down transformation logic to leverage the scalability and performance of cloud data platforms such as Amazon Redshift and Snowflake. 
• Visual Interface: Create data pipelines using a graphical interface, reducing the need to write code. 
• Library of Connectors: Access a library of pre-built transformations and connectors across a range of data sources. 
• High-Code and No-Code: Supports both hide-code and no-code development, making it accessible for beginner and intermediate users. 
• dbt Component: With the dbt component, you can embed dbt within a Matillion pipeline.

Is Matillion right for you?

Matillion’s pre-built connectors and visual interface makes it an ideal solution for less experienced data professionals. The disadvantage is that it can be costly for businesses on a budget. Moreover, you must ensure that Matillion supports your specific requirements and how you intend to perform the data transformations. Care must be given to the long-term maintainability of pipelines that are both visual and code-based.

Getting started with Matillion is simple because they use a drag-and-drop interface for building data pipelines. But like with any other visual tool, there is still a learning curve and it’s typical to have a mix of code and visual components in a production data pipeline.

10. Alteryx

Alteryx simplifies the data transformation process. You can automate advanced analytics and prepare data through self-service. It’s an effective solution that makes it easier for teams to collaborate. Unlike the other visual tools above which are typically used by Data Engineers in IT, Alteryx is more widely adopted in less technical departments of an organization. It’s also typically paired with visualization tools like Tableau. 

Notable features 

• Drag-and-drop User Interface: The visual interface makes building and collaborating on data transformation workflows easier.
• Data Loading: Connectors to popular databases and services allow you to integrate different data sources 
• Machine Learning: Alteryx may also be used to create simple machine learning models in a visual way

Is Alteryx right for you?

Alteryx is a good option to help ensure teams are on the same page throughout the data workflow. Data transformation projects can be shared and feedback provided seamlessly, making collaboration easier. 

The downside is that Alteryx is costly compared to the other tools on this list. Moreover, there is still a bit of a learning curve, even if you’re experienced in data analytics. You should also check that Alteryx aligns with teams for effective collaboration.

How Datacoves can help you transform data

Data transformation is a process that’s prone to multiple errors along the way. While many tools listed can help you reduce friction, they must be carefully evaluated. With Datacoves, you’ll be able to implement best data practices and DataOps so that you have a smooth process with a minimized learning curve. 

If you’d like to learn more about how Datacoves helps you accelerate time to value, you can schedule a free demo here.

Jinja is the game changing feature of dbt Core that allows us to create dynamic SQL code. In addition to the standard Jinja library, dbt Core includes additional functions and variables to make working with dbt even more powerful out of the box.

See our original post, The Ultimate dbt Jinja Cheat Sheet, to get started with Jinja fundamentals like syntax, variable assignment, looping and more. Then dive into the information below which covers Jinja additions added by dbt Core.

This cheatsheet references the extra functions, macros, filters, variables and context methods that are specific to dbt Core.

Enjoy!

dbt Core: Functions

These pre-defined dbt Jinja functions are essential to the dbt workflow by allowing you to reference models and sources, write documentation, print, write to the log and more.

select * from {{ ref('model_name') }}
select * from {{ ref('model_name', version=1) }}
select * from {{ ref('project_or_package', 'model_name') }}

select * from {{ source('source_name', 'table_name') }} 

{{ log('Logging something', info=True) }}

{{ print("Print something: " ~ value1 ~ ", " ~ value2) }}

{% docs table_sales_description %}

# docs
- go
- here
 
{% enddocs %}

#schema.yml
version: 2

models:
- name: sales   
  description: '{{ doc("table_sales_description") }}'

{% macro get_list() %}
  {{ return(['column_1', 'column_2', 'column_2']) }}
{% endmacro %}

select 
-- getdata() returns a list!
{% for item in get_list() %}
  {{ item }}
  {% if not loop.last %},{% endif %}
{% endfor %}
from {{ ref('some_model') }}

"{{ env_var('DB_PORT') | as_number }}"

#dbt_project.yml
models:
  my_project:
    +materialized: "{{ env_var('DBT_MATERIALIZATION', 'table') }}"

#dbt_project.yml
vars:
  start_date: '2023-10-01'

select * from orders 
where start_date >'{{ var("start_date") }}'

dbt Core: Macros

These macros are provided in dbt Core to improve the dbt workflow.

{% set results = run_query('select order from orders_table') %}

{% macro my_macro() %}
  {% set result = my_macro() %}
  
  {{ debug() }}
  
{% endmacro %}

dbt Core: Filters

These dbt Jinja filters are used to modify data types.

{{ <my_value> | as_bool }}

{{ <my_value> | as_number }}

{{ <my_value> | as_native }}

dbt Core: Project context variables

These dbt core "variables" such as config, target, source, and others contain values that provide contextual information about your dbt project and configuration settings. They are typically used for accessing and customizing the behavior of your dbt models based on project-specific and environment-specific information.

{% macro ref(modelname) %}
    {% set db_name = builtins.ref(modelname).database | lower %}}
    {% if db_name.startswith('staging') or 
        db_name.endswith('staging')  %}
        {{ return(builtins.ref(modelname).include(database=false)) }}
    {% else %}
        {{ return(builtins.ref(modelname)) }}
    {% endif %}
{% endmacro %}

{%- set unique_key = config.get('unique_key', default='id') -%}

{%- set unique_key = config.require('unique_key') -%}

{% set payment_method_query %}
  SELECT DISTINCT
    payment_method
  FROM {{ ref('raw_payments') }}
  ORDER BY 1
{% endset %}

{% set payment_methods = run_query(payment_method_query) %}

{% if execute %}
  {# Extract the payment methods from the query results #}
  {% set payment_methods_list = payment_methods.columns[0].values() %}
{% else %}
  {% set payment_methods_list = [] %}
{% endif %}

{% if flags.STORE_FAILURES %}
--your logic
{% else %}
--other logic
{% endif %}

modules.<desired_module>.<desired_function> 

{% set now = modules.datetime.datetime.now() %}

dbt Core: Run context variables

These special variables provide information about the current context in which your dbt code is running, such as the model, schema, or project name.

select
    '{{ run_started_at.strftime("%Y-%m-%d") }}' as run_start_utc

dbt Core: Context methods

These methods allow you to retrieve information about models, columns, or other elements of your project.

{% set my_list = ['a', 'b', 'b', 'c'] %}
{% set my_set = set(my_list) %}
{% do print(my_set) %}


{# evaluates to {'a','b', 'c'} #}

{% set my_set = set_strict(my_list) %}

exceptions.raise_compiler_error("Custom Error message")

exceptions.warn("Custom Warning message") 

{%- set hashed_value = local_md5(“String to hash”) -%}
select 
     '{{ hashed_value }}' as my_hashed_value

{% set my_list_a = [1, 2] %}
{% set my_list_b = ['Data', 'Coves'] %}
{% set my_zip = zip(my_list_a, my_list_b) | list %}
{% do print(my_zip) %}  

{# Result [(1, 'Data'), (2, 'Coves')] #}

{% set my_list_a = 123 %}
{% set my_list_b = 'Datacoves' %}
{% set my_zip = zip_strict(my_list_a, my_list_b) | list %}
{# This will fail #}

Please contact us with any errors or suggestions.

If you've taken an interest in dbt (data build tool) and are on the fence about whether to opt for dbt Cloud or dbt Core, you're in the right place. Perhaps you're already using one of the dbt platforms and are considering a change. Regardless of your current position, understanding the differences of these options is crucial for making an informed decision. In this article, we'll delve deep into the key distinctions between dbt Cloud and dbt Core.

dbt, dbt Core, dbt Cloud: What do they mean?

For those new to the dbt community, navigating the terminology can be a tad confusing. "dbt," "dbt Core," and "dbt Cloud" may sound similar but each represents a different facet of the dbt ecosystem. Let's break it down.

What is dbt?

dbt is the generic name for the open-source tool and when people say dbt the features are mainly those in dbt Core. dbt allows users to write, document, and execute SQL-based transformations, making it easier to produce reliable and up-to-date analytics. By facilitating practices like version control, testing, and documentation, dbt enhances the analytics engineering workflow, turning raw data into actionable insights.

Once you decide dbt is right for your organization, the next step is to determine how you'll access dbt. The two most prevalent methods are dbt Core and dbt Cloud. While dbt Cloud offers an enhanced experience with additional features, its abstraction can sometimes limit the desired flexibility and control over the workflow especially when it comes to using dbt with the complexities of an enterprise.

Throughout this article we'll observe that by using dbt Core and incorporating other tools, you can achieve many of the same functionalities as dbt Cloud while maintaining flexibility and control. While this approach offers enhanced flexibility, it consequently introduces increased complexity, maintenance, and an added workload. When adopting a dbt platform it is important to understand the tradeoffs to truly know what will work best for your data team.

What is dbt Core?

dbt Core is an open-source data transformation tool that enables data analysts and engineers to transform and model data to derive business insights. dbt Core is the foundational, open-source version of dbt that provides users with the utmost flexibility. The term "flexible" implies that users have complete autonomy over its implementation, integration, and configuration within their projects.

Even though dbt Core is free, to meet or exceed the functionality of dbt Cloud, it will need to be paired with additional tooling as we will discuss below.These open source solutions may be leverage at no cost, but this increases the platform maintenance overhead and may impact the total cost of ownership and the platform's time to market. Alternatively, managed dbt Core platforms exist, like Datacoves, which simplify this process.

Using and installing dbt Core is done manually. Depending on which data warehouse you are using, you select the appropriate dit adapter such as dit-snowflake, dbt-databricks, dt-redshift, etc. You can see all available dbt adapters on our dbt libraries. If you are using Snowflake you can check out our detailed Snowflake with dbt getting started guide.

Given that you have installed the pre-requisites, installing dbt is just a matter of installing dbt-snowflake.

What is dbt Cloud?

dbt Cloud is a hosted dbt platform to develop and deploy dbt projects. dbt Cloud leverages all the power of dbt Core with some extra features such as a proprietary Web-based UI, a dbt job scheduler, APIs, integration with Continuous Integrations platforms like Github Actions, and a proprietary Semantic layer. dbt Cloud's features are all intended to facilitate the dbt workflow.

dbt cloud pricing has three tiers: Enterprise, Team and Developer. Developer is a free tier meant for a single developer with a hard limit of 3000 model runs per month. The Team Plan pricing starts at $100 per developer for teams up to 8 with 15,000 successful models built per month; any additional models will cost $0.01.

dbt Core vs dbt Cloud: Development - IDE  

When it comes to the Integrated Development Environment (IDE), both dbt Cloud and dbt Core present distinct advantages and challenges. Whether you prioritize flexibility, ease of setup, or a blend of both, your choice will influence how your team develops, tests, and schedules your data transformations. Let's explore how each option handles the IDE aspect and the impact on developers and analytic engineers.

dbt Core

In the instance of IDEs, using dbt Core requires setting up a dev environment on each member's device or a virtual space like AWS workspace. This involves installing a popular dbt IDE such as VS Code, dbt Core, connecting to a data warehouse, and handling dependencies like Python versions.  

Enterprise dbt setups typically include additional dependencies to enhance productivity. Some notable VS Code extensions for this include dbt Power User, SQLFluff, and the official dbt Snowflake VS Code extension.

dbt Cloud

When companies are ramping up with dbt, one of the pain points is setting up and managing dbt IDE environments. Analytic Engineers coming to dbt may not be familiar with concepts like version control with git or using the command line. The dbt Cloud IDE simplifies developer onboarding by providing a web-based SQL IDE to team members so they can easily write, test, and refine data transformation code without having to install anything on their computers. Complexities like starting a git branch are tucked behind a big colorful button so users know that is the first step in their new feature journey.

However, Developers who are accustomed to more versatile local IDEs, such as VS Code, may find the dbt Cloud experience limiting as they cannot leverage extensions such as those from the VS Code Marketplace nor can they extend dbt Core using the vast array of Python libraries.  

It is possible to get the best of both worlds - the flexibility of dbt Core in VS Code and the quick setup that dbt Cloud Offers  - with a Managed dbt Core Platform like Datacoves. In a best-in-class developer setup, new users are onboarded in minutes with simple configuration screens that remove the need to edit text files such as profiles.yml and remove the complexity of creating and managing SSH keys. Version upgrades of dbt or any dependent library should be transparent to users. Spinning up a pristine environment should be a matter of clicks.  

dbt Core vs dbt Cloud: Scheduling jobs  

Scheduling in a dbt project is crucial for ensuring timely and consistent data updates. It's the backbone of reliable and up-to-date analytics in a dbt-driven environment.  

dbt Core

While an orchestrator does not come out of the box with dbt Core, when setting up a deployment environment companies can leverage any orchestration tool, such as Airflow, Dagster, or Prefect. They can connect steps prior to or after the dbt transformations and they can trigger any tool that exists within or outside the corporate network.

dbt Cloud

dbt Cloud makes deploying a dbt Core project simple. It allows you to define custom environment variables and the specific dbt commands (seed, run, test) that you want to run during production runs. The dbt Cloud scheduler can be configured to trigger at specific intervals using an intuitive UI.

dbt Cloud is primarily focused on running dbt projects. Therefore, if a data pipeline has more dependencies, an external orchestration tool may be required. Fortunately, if you do use an external orchestrator, dbt Cloud offers an API to trigger dbt Cloud jobs from your orchestrator.

dbt Core vs dbt Cloud: DataOps - Continuous Integration  

DataOps emphasizes automating the integration of code changes, ensuring that data transformations are consistently robust and reliable. Both platforms approach CI/CD differently. How seamless is the integration? How does each platform handle tool compatibility?

dbt Core  

When using dbt Core for your enterprise data platform, you will need to not only define and configure the automation scripts, but you will also need to ensure that all the components, such as a git server, CI server, CI runners, etc. are all working harmoniously together.

Since dbt Core can be run within the corporate firewall, it can be integrated with any CI tool and internal components such as Jira, Bitbucket, and Jenkins. To do this well, all the project dependencies must be packed into reusable Docker containers. Notifications will also need to be defined across the various components and all of this will take time and money.

dbt Cloud

dbt Cloud has built in CI capabilities which reduce the need for third party tools. dbt Cloud can also be paired with Continuous Integration (CI) tools like GitHub Actions to validate data transformations before they are added to a production environment. Aspects such as code reviews and approvals will occur in the CI/CD tool of choice such as GitHub and dbt Cloud can report job execution status back to GitHub Actions. This allows teams to know when it is safe to merge changes to their code. One item to note is that each successful model run in your CI run will count against the monthly model runs as outlined in the dbt Cloud pricing.

Companies that have tools like Bitbucket, Jira, and Jenkins within their corporate firewall may find it challenging to integrate with dbt Cloud.

dbt Core vs dbt Cloud: Semantic layer  

A semantic layer helps businesses define important metrics like sales, customer churn, and customer activations with the flexibility to aggregate at run time. These metrics can be referenced by downstream tools as if they had been previously computed. End-users benefit from the flexibility to aggregate metrics at diverse grains without the company incurring the cost of pre-computing every permutation. These on-the-fly pivots ensure consistent and accurate insights across the organization.

dbt Core

dbt Core does not come with a built-in semantic layer, but there are open source and proprietary alternatives that allow you to achieve the same functionality. These include cube.dev, and Lightdash.

dbt Cloud

dbt Cloud has been rolling out a proprietary semantic layer which is currently in public preview. This feature is only available to dbt pricing plans Team and Enterprise. When using the dbt Cloud semantic layer your BI tool connects to a dbt Cloud proxy server which sits between the BI tool and your Data Warehouse.

dbt’s semantic layer offers a system where metrics are standardized as dbt metadata, visualized in your DAG, and integrated seamlessly with features like the Metadata API and the dbt proxy server.

dbt Core vs. dbt Cloud: Project Exploration and Lineage Tracking

Understanding your dbt project's structure and data flow is essential for effective data management and collaboration. While dbt Cloud offers dbt Explorer, a tool that visually maps model dependencies and metadata, it is exclusive to dbt Cloud users.

dbt Docs (dbt docs generate) is a built-in feature in dbt Core that generates a static documentation site, providing lineage graphs and detailed metadata for models, columns, and tests. However, for larger projects, dbt Docs can struggle with high memory usage and slow load times, making it less practical for extensive datasets. Also, dbt Docs lacks column-level lineage, which is crucial for impact analysis and debugging.

But no worries—dbt Core users can achieve similar, and even better, functionalities through alternative methods. The answer: a data catalog like DataHub. A Data Catalog can significantly enhance not just your dbt exploration, but your entire data project discovery experience! 

DataHub Offers:

• Searchable Metadata with Comprehensive Integration – DataHub can ingest metadata from various sources, including Apache Airflow and dbt, consolidating everything into a single repository. This creates a unified view of your data ecosystem, surpassing the capabilities of dbt Explorer.
• Column-Level Lineage – See how data moves and transforms at the column level, enabling more precise impact analysis and debugging.
• Collaborative Features – Facilitate teamwork with tools designed for sharing and documenting data knowledge across your organization such as a data glossary.

There is an obvious caveat. Implementing and maintaining an open-source data catalog like DataHub introduces additional complexity. Organizations need to allocate resources to manage, update, and scale the platform effectively. Fortunately, a managed solution like Datacoves simplifies this by providing an integrated offering that includes DataHub, streamlining deployment and reducing maintenance overhead.

dbt Core vs dbt Cloud: API

APIs play a crucial role in streamlining dbt operations and enhancing extensibility.  

dbt Core

With dbt Core, users often rely on external solutions to integrate specific API functionalities.

Administrative API Alternative: There is currently no feature-to-feature alternative for the dbt Cloud administrative API. However, the Airflow API can be leveraged to enqueue runs for jobs which is a primary feature of the dbt Cloud Administrative API.

Discovery API Alternative: This API was formerly known as the dbt Cloud Metadata API. Tools such as Datahub can provide similar functionality. Datahub can consume dbt Core artifacts such as the manifest.json and expose an API for dbt metadata consumption.

Semantic Layer API Alternative: When it comes to establishing and managing the semantic layer, Cube.dev provides a mature, robust, and comprehensive alternative to the dbt Cloud Semantic layer. Cube also has an API tailored for this purpose.

dbt Cloud

dbt Cloud offers three APIs. These APIs are available to Team and Enterprise customers.  

Administrative API: The dbt Cloud Administrative API is designed primarily for tasks like initiating runs from a job, monitoring the progress of these runs, and retrieving artifacts once the jobs have been executed. dbt Cloud is working on additional functionality for this API, such as operational functions within dbt Cloud.

Discovery API: Whenever you run a project in dbt Cloud, it saves details about that project, such as information about your data models, sources, and how they connect. The Discovery API lets you access and understand this saved information. Use cases include: performance, quality, discovery, governance and development.

Semantic Layer API: The dbt Semantic Layer API provides a way for users to interact with their data using a JDBC driver. By using this API, you can easily query metrics values from your data and get insights.

Conclusion

Examining the differences between dbt Core and dbt Cloud reveals that both can lead organizations to similar results. Much of what dbt Cloud offers can be replicated with dbt Core when combined with appropriate additional tools. While this might introduce some complexities, the increased control and flexibility might justify the trade-offs for certain organizations. Thus, when deciding between the two, it's a matter of prioritizing simplicity versus adaptability for the team. This article only covers dbt core vs dbt cloud but you can read more about dbt alternatives in our blog..

As a managed dbt Core solution, the Datacoves platform simplifies the dbt Core experience and retains its inherent flexibility. It effectively bridges the gap, capturing many benefits of dbt Cloud while mitigating the challenges tied to a pure dbt Core setup. See if Datacoves dbt pricing is right for your organization or visit our product page.

In the age of data-driven decision-making, companies grapple with the mammoth task of setting up a robust Modern Data Stack. The on premise legacy systems struggle to keep up, and standing up a Modern Data Stack (MDS) isn't just a tech upgrade; it's an essential pivot, ensuring businesses extract actionable insights from the raw data they encounter. However, the road to achieving this is complex and slower than the line at the DMV.

If the responsibility of establishing a Modern Data Stack falls on your shoulders and you're feeling the weight of its time/resource/knowledge-intensive nature, this post offers insights and solutions.We explore the hurdles businesses encounter while shaping their data infrastructure and how you can streamline and expedite the process.

What is a Modern Data Stack?

A Modern Data Stack refers to a suite of tools and digital technologies specifically designed for data management. Within this stack, some tools specialize in collecting data, while others focus on storing or processing it. As data moves through this system, it's transformed from raw input into actionable insights.

Many of these tools come from various providers and must be seamlessly integrated to ensure optimal performance. Leveraging the latest technologies, the modern data stack efficiently manages the entire data lifecycle, from collection to analysis. This stack is both scalable and flexible, ensuring it can adapt and grow with the ever-evolving demands of a business, and provide consistent performance regardless of data volume or complexity.

Below you can see an example Modern Data Architecture Diagram.

‍

Standing up a Modern Data Stack takes time 

The path to a comprehensive end-to-end enterprise data platform is not without challenges. Embarking on such a journey requires diligent research, because the process of migrating to a Modern Data Stack or establishing it from the ground up is intricate and piecemeal. Since there are many individual tools in the Modern Data Stack, you may have to tackle each tool individually so you can focus on setting it up correctly. Given the complexity of the endeavor, even with a skilled team on board, it can take between 6 to 9 months to build a complete end-to-end data solution. This may be frustrating, but understanding the pain points in setting up a Modern Data Stack can help to make educated decisions that accelerate the process.

High level pain points:

• Hundreds of tools to choose from - While having many options can be beneficial, it can also be overwhelming. The vast selection can lead to what's known as "tool overload," making it hard to pick the best fit.
• Difficult to Integrate various tools - Once you've picked your tools, the next step is integration. But with so many different systems and platforms available, getting them to work together can be like solving a complex puzzle.
• Architecting a secure platform - Everyone agrees that data security is critical. Creating a platform that's not just secure but also easy to replicate and audit is challenging and requires careful planning.
• Implementing best practices - The lack of standard processes in the data world can lead to inconsistencies. Finding and applying the best practices isn't just about knowledge; it's also about experience.

• Hidden Costs - Even though a lot of modern data stack tools are freely available, the hidden cost emerges in the form of time - spent on learning, configuring, testing, and refining. It’s like getting a "free puppy"; while there might be zero upfront expenses, the continuous care, attention, and commitment required are far from zero.

‍

Modern Data Stack: Guiding principles for success

A strong data platform is the backbone of good decision-making. It helps us see clear insights fast and strengthens our data teams. When creating or choosing such a system, keep these principles in mind:

• Trustworthiness - There needs to be trust that the data is always right and true.
• Usability - The system should be easy to use and understand.
• Collaboration - Teams should be able to work together in a secure way.
• Reusability - If one part of the system is good, we should be able to use it in other places too.
• Maintainability - There should be automated process and DataOps in place.
• Reliability - It should always work well, detect errors, not break down often, and keep data safe.

‍

Following these rules can help us get the most from our data and make the best decisions

Simplify the Modern Data Stack

Understanding the challenges and intricacies of setting up a Modern Data Stack makes it clear why we need efficient solutions. In the data world things move fast and speed is imperative. While there are numerous tools available that cater to specific components, Datacoves offers a more comprehensive approach, addressing the end-to-end data stack. Datacoves could reduce the setup of your Enterprise Data Platform from the usual 6-9 months to just 2-3 weeks. But how does it achieve this feat?

Datacoves is:

• A Turnkey Solution - Datacoves doesn't just offer a solution; it provides an all-encompassing package designed meticulously to streamline the entire data-to-insight trajectory. This isn't about starting from scratch; it's about leveraging a fully-equipped platform to jumpstart your journey.
• Guidelines and Expertise - No more searching in the dark. Datacoves ensures its users have a clear path ahead. The challenge of standardizing processes, which once seemed like climbing Everest, is simplified, thanks to the expert guidance provided.
• Scalability At Its Finest - Whether you’re a budding team of 3 just starting out or a robust squad of 300, Datacoves has been engineered to scale with your needs, ensuring consistency and efficiency at every stage.
• State-of-the-Art Tools - With tools like a finely-tuned VS Code in the browser with dbt Core, Datacoves ensures users aren't just walking but sprinting from the get-go. It's about giving you the best gear to make your climb smoother.
• Best Practices at Your Fingertips - Datacoves realizes that in the fast-paced world of data, time is of the essence. That's why, through integrated accelerators, it ensures that adhering to industry best practices isn’t a drawn-out quest but just a matter of configuring to your needs.

Highlighting Datacoves' features

Datacoves is not just another platform; it's a game-changer. Its project-based structure integrates seamlessly with any git repository, and it can be swiftly deployed in a private cloud to connect with existing tools. Each project provides multiple environments, facilitating role-based access and ensuring user-specific needs are met.

Here are just a few ways that Datacoves empowers the Data Engineer and Analytics Engineer to deliver quickly:

• Everything in one place - The objective is to streamline the Data/Analytics Engineer's workflow. By consolidating essential tools and functionalities into a single interface, users can load data, review entries in their data warehouse, configure DAGs, write code in VSCode, and more, all without switching tabs.

• ‍Airflow YML Configuration - With this feature, users can bypass the complexities of Python when working with Airflow. Instead, the YML configuration allows for a more direct way to set up your DAGs.

• ‍dbt-coves Extension - Within your vscode workspace, the dbt-coves extension is integrated, making tasks more efficient. Specifically, the "dbt-coves generate sources" command examines your database, updates files, and integrates them into your yml with ease.

• ‍ChatGPT Integration - Embedded directly in your vscode workspace, ChatGPT offers a hassle-free way to seek answers without changing tabs. This feature is especially handy for tasks like creating model descriptions—simply generate, adjust as needed, and move forward.

Datacoves’ Northstar

Datacoves aims to simplify, reduce friction, enhance collaboration, and inject software engineering practices into data operations. It seeks to empower teams, enabling swift productivity and ensuring teams function cohesively.

Intrigued by Datacoves? Dive deeper by watching the full video below or book a demo to experience its magic first-hand.

‍