Digital Transformation

There's a lot of buzz around Microsoft Fabric these days. Some people are all-in, singing its praises from the rooftops, while others are more skeptical, waving the "buyer beware" flag. After talking with the community and observing Fabric in action, we're leaning toward caution. Why? Well, like many things in the Microsoft ecosystem, it's a jack of all trades but a master of none. Many of the promises seem to be more marketing hype than substance, leaving you with "marketecture" instead of solid architecture. While the product has admirable, lofty goals, Microsoft has many wrinkles to iron out.

In this article, we'll dive into 10 reasons why Microsoft Fabric might not be the best fit for your organization in 2025. By examining both the promises and the current realities of Microsoft Fabric, we hope to equip you with the information needed to make an informed decision about its adoption.

What is Microsoft Fabric?

Microsoft Fabric is marketed as a unified, cloud-based data platform developed to streamline data management and analytics within organizations. Its goal is to integrate various Microsoft services into a single environment and to centralize and simplify data operations.

This means that Microsoft Fabric is positioning itself as an all-in-one analytics platform designed to handle a wide range of data-related tasks. A place to handle data engineering, data integration, data warehousing, data science, real-time analytics, and business intelligence. A one stop shop if you will. By consolidating these functions, Fabric hopes to provide a seamless experience for organizations to manage, analyze, and gather insights from their data.

Core Components of Microsoft Fabric

• OneLake: OneLake is the foundation of Microsoft Fabric, serving as a unified data lake that centralizes storage across Fabric services. It is built on Delta Lake technology and leverages Azure Blob Storage, similar to how Apache Iceberg is used for large-scale cloud data management

• Synapse Data Warehouse: Similar to Amazon Redshift this provides storage and management for structured data. It supports SQL-based querying and analytics, aiming to facilitate data warehousing needs.

• Synapse Data Engineering: Compute engine based on Apache Spark, similar to Databricks' offering. It is built on Apache Spark and is intended to support tasks such as data cleaning, transformation, and feature engineering.

• Azure Data Factory: A tool for pipeline orchestration and data loading which is also part of Synapse Data Engineering

• Synapse Data Science: Similar to Jupiter Notebooks that can only run on Azure Spark. It is designed to support data scientists in developing predictive analytics and AI solutions by leveraging Azure ML and Azure Spark services.

• Synapse Real-Time Analytics: Enables the analysis of streaming data from various sources including Kafka, Kinesis, and CDC sources.

• Power BI: This is a BI (business intelligence) tool like Tableau tool designed to create data visualizations and dashboards.

Fabric presents itself as an all-in-one solution, but is it really? Let’s break down where the marketing meets reality.

10 Reasons It’s Still Not the Right Choice in 2025

While Microsoft positions Fabric is making an innovative step forward, much of it is clever marketing and repackaging of existing tools. Here’s what’s claimed—and the reality behind these claims:

1. Fragmented User Experience, Not True Unification

Claim: Fabric combines multiple services into a seamless platform, aiming to unify and simplify workflows, reduce tool sprawl, and make collaboration easier with a one-stop shop.

Reality:

• Rebranded Existing Services: Fabric is mainly repackaging existing Azure services under a new brand. For example, Fabric bundles Azure Data Factory (ADF) for pipeline orchestration and Azure Synapse Analytics for traditional data warehousing needs and Azure Spark for distributed workloads. While there are some enhancements to synapse to synchronize data from OneLake, the core functionalities remain largely unchanged. PowerBI is also part of Fabric  and this tool has existed for years as have notebooks under the Synapse Data Science umbrella.

• Steep Learning Curve and Complexity: Fabric claims to create a unified experience that doesn’t exist in other platforms, but it just bundles a wide range of services—from data engineering to analytics—and introduces new concepts (like proprietary query language, KQL which is only used in the Azure ecosystem). Some tools are geared to different user personas such as ADF for data engineers and Power BI for business analysts, but to “connect” an end-to-end process, users would need to interact with different tools. This can be overwhelming, particularly for teams without deep Microsoft expertise. Each tool has its own unique quirks and even services that have functionality overlap don’t work exactly the same way to do the same thing. This just complicates the learning process and reduces overall efficiency.

2. Performance Bottlenecks & Throttling Issues

Claim: Fabric offers a scalable and flexible platform.

Reality: In practice, managing scalability in Fabric can be difficult. Scaling isn’t a one‑click, all‑services solution—instead, it requires dedicated administrative intervention. For example, you often have to manually pause and un-pause capacity to save money, a process that is far from ideal if you’re aiming for automation. Although there are ways to automate these operations, setting up such automation is not straightforward. Additionally, scaling isn’t uniform across the board; each service or component must be configured individually, meaning that you must treat them on a case‑by‑case basis. This reality makes the promise of scalability and flexibility a challenge to realize without significant administrative overhead.  

3. Capacity-Based Pricing Creates Cost Uncertainty

Claim: Fabric offers predictable, cost-effective pricing.

Reality: While Fabric's pricing structure appears straightforward, several hidden costs and adoption challenges can impact overall expenses and efficiency:  

• Cost uncertainty: Microsoft Fabric uses a capacity-based pricing model that requires organizations to purchase predefined Capacity Units (CUs).  Organizations need to carefully assess their workload requirements to optimize resource allocation and control expenses. Although a pay-as-you-go (PAYG) option is available, it often demands manual intervention or additional automation to adjust resources dynamically.  This means organizations often need to overprovision compute power to avoid throttling, leading to inefficiencies and increased costs. The problem is you pay for what you think you will use and get a 40% discount. If you don’t use all of the capacity, then there are wasted capacity. If you go over capacity, you can do configure PAYG (pay as you go) but it’s at full price. Unlike true serverless solutions, you pay for allocated capacity regardless of actual usage. This isn’t flexible like the cloud was intended to be.  👎

• Throttling and Performance Degradation: Exceeding purchased capacity can result in throttling, causing degraded performance. To prevent this, organizations might feel compelled to purchase higher capacity tiers, further escalating costs.

• Visibility and Cost Management: Users have reported challenges in monitoring and predicting costs due to limited visibility into additional expenses. This lack of transparency necessitates careful monitoring and manual intervention to manage budgets effectively.  

• Adoption and Training Time: It’s important to note that implementing Fabric requires significant time investment in training and adapting existing workflows. While this is the case with any new platform, Microsoft is notorious for complexity in their tooling and this can lead to longer adoption periods, during which productivity may temporarily decline.

All this to say that the pricing model is not good unless you can predict with great accuracy exactly how much you will spend every single day, and who knows that? Check out this article on the hidden cost of fabric which goes into detail and cost comparisons.

4. Limited Compatibility with Non-Microsoft Tools

Claim: Fabric supports a wide range of data tools and integrations.

Reality: Fabric is built around a tight integration with other Fabric services and Microsoft tools such as Office 365 and Power BI, making it less ideal for organizations that prefer a “best‑of‑breed” approach (or rely on tools like Tableau, Looker, open-source solutions like Lightdash, or other non‑Microsoft solutions), this can severely limit flexibility and complicate future migrations.

While third-party connections are possible, they don’t integrate as smoothly as those in the MS ecosystem like Power BI, potentially forcing organizations to switch tools just to make Fabric work.

5. Poor DataOps & CI/CD Support

Claim: Fabric simplifies automation and deployment for data teams by supporting modern DataOps workflows.

‍
Reality: Despite some scripting support, many components remain heavily UI‑driven. This hinders full automation and integration with established best practices for CI/CD pipelines (e.g., using Terraform, dbt, or Airflow). Organizations that want to mature data operations with agile DataOps practices find themselves forced into manual workarounds and struggle to integrate Fabric tools into their CI/CD processes. Unlike tools such as dbt, there is not built-in Data Quality or Unit Testing, so additional tools would need to be added to Fabric to achieve this functionality.

6. Security Gaps & Compliance Risks

Claim: Microsoft Fabric provides enterprise-grade security, compliance, and governance features.

Reality: While Microsoft Fabric offers robust security measures like data encryption, role-based access control, and compliance with various regulatory standards, there are some concerns organizations should consider.

One major complaint is that access permissions do not always persist consistently across Fabric services, leading to unintended data exposure.

For example, users can still retrieve restricted data from reports due to how Fabric handles permissions at the semantic model level. Even when specific data is excluded from a report, built-in features may allow users to access the data, creating compliance risks and potential unauthorized access. Read more: Zenity - Inherent Data Leakage in Microsoft Fabric.

While some of these security risks can be mitigated, they require additional configurations and ongoing monitoring, making management more complex than it should be. Ideally, these protections should be unified and work out of the box rather than requiring extra effort to lock down sensitive data.

7. Lack of Maturity & Changes that Disrupt Workflow

Claim: Fabric is presented as a mature, production-ready analytics platform.

Reality: The good news for Fabric is that it is still evolving. The bad news is, it's still evolving. That evolution impacts users in several ways:  

• Frequent Updates and Unstable Workflows: Many features remain in preview, and regular updates can sometimes disrupt workflows or introduce unexpected issues. Users have noted that the platform’s UI/UX is continually changing, which can impact consistency in day-to-day operations. Just when you figure out how to do something, the buttons change. 😤

• Limited Features: Several functionalities are still in preview or implementation is still in progress. For example, dynamic connection information, Key Vault integration for connections, and nested notebooks are not yet fully implemented. This restricts the platform’s applicability in scenarios that rely on these advanced features.

• Bugs and Stability Issues: A range of known issues—from data pipeline failures to problems with Direct Lake connections—highlights the platform’s instability. These bugs can make Fabric unpredictable for mission-critical tasks. One user lost 3 months of work!

8. Black Box Automation & Limited Customization

Claim: Fabric automates many complex data processes to simplify workflows.

Reality: Fabric is heavy on abstractions and this can be a double‑edged sword. While at first it may appear to simplify things, these abstractions lead to a lack of visibility and control. When things go wrong it is hard to debug and it may be difficult to fine-tune performance or optimize costs.

For organizations that need deep visibility into query performance, workload scheduling, or resource allocation, Fabric lacks the granular control offered by competitors like Databricks or Snowflake.

9. Limited Resource Governance and Alerting

Claim: Fabric offers comprehensive resource governance and robust alerting mechanisms, enabling administrators to effectively manage and troubleshoot performance issues.  

Reality: Fabric currently lacks fine-grained resource governance features making it challenging for administrators to control resource consumption and mitigate issues like the "noisy neighbor" problem, where one service consumes disproportionate resources, affecting others.  

The platform's alerting mechanisms are also underdeveloped. While some basic alerting features exist, they often fail to provide detailed information about which processes or users are causing issues. This can make debugging an absolute nightmare. For example, users have reported challenges in identifying specific reports causing slowdowns due to limited visibility in the capacity metrics app. This lack of detailed alerting makes it difficult for administrators to effectively monitor and troubleshoot performance issues, often needing the adoption of third-party tools for more granular governance and alerting capabilities. In other words, not so all in one in this case.

10. Missing Features & Gaps in Functionality

Claim: Fabric aims to be an all-in-one platform that covers every aspect of data management.  

Reality: Despite its broad ambitions, key features are missing such as:

• Geographical Availability: Fabric's data warehousing does not support multiple geographies, which could be a constraint for global organizations seeking localized data storage and processing.  

• Garbage Collection: Parquet files that are no longer needed are not automatically removed from storage, potentially leading to inefficient storage utilization.  

While these are just a couple of examples it's important to note that missing features will compel users to seek third-party tools to fill the gaps, introducing additional complexities.  Integrating external solutions is not always straight forward with Microsoft products and often introduces a lot of overhead.  Alternatively, users will have to go without the features and create workarounds or add more tools which we know will lead to issues down the road.  

Conclusion

Microsoft Fabric promises a lot, but its current execution falls short. Instead of an innovative new platform, Fabric repackages existing services, often making things more complex rather than simpler.

That’s not to say Fabric won’t improve—Microsoft has the resources to refine the platform. But as of 2025, the downsides outweigh the benefits for many organizations.

If your company values flexibility, cost control, and seamless third-party integrations, Fabric may not be the best choice. There are more mature, well-integrated, and cost-effective alternatives that offer the same features without the Microsoft lock-in.

Time will tell if Fabric evolves into the powerhouse it aspires to be. For now, the smart move is to approach it with a healthy dose of skepticism.

👉 Before making a decision, thoroughly evaluate how Fabric fits into your data strategy. Need help assessing your options? Check out this data platform evaluation worksheet.  

‍

The reason companies fail at leveraging analytics stems from the fact that people tend to focus on the destination instead of the journey that will lead to the solutions that will have the most impact on the business. Time and time again, I see people focus on the so-called shiny objects, like new tools, new techniques, or even new people, that appear to be the silver bullet everyone needs. The truth is, if you go back to the first principles and start with true alignment, good data processes, and user-centric experiences, project success and satisfaction are achievable.

Lack of Alignment Reduces Faith in the Solution

Every project I have been a part of started with a sense of optimism and excitement. The honeymoon phase was great. Everyone was united; we had gotten the funding, selected vendor partners, and purchased whatever technology was part of the solution. We all spoke the same language, everyone got to work, management started getting progress updates, and everyone thought we were off to a great start.

It wasn't until real decisions needed to be made that we realized the honeymoon was over. In every single instance, an excessive amount of time was spent in meetings  arguing and reaching some level of consensus until the next decision. The reason this happened was because we didn't really spend the time to get on the same page. People assumed that we were aligned because at a high level, we all talking about the key points of the given initiative: digital transformation, self-service analytics, customer mastering, data lakes, etc.

But we were not really thinking the same things. Everyone had different backgrounds and had expertise on different parts of the solution: regulatory requirement, technology limitations, end-user needs, etc. There were also things no one knew at the start, and we didn't have a north star to guide these decisions. We all appeared to be saying the same things, but we were thinking very differently.

‍

I have seen the pressure to get started on a project and show progress lead to delays and ultimate dissatisfaction with the end result. On projects where we have spent a couple of weeks getting aligned using a structured approach to product discovery, we ended up with better estimates and better overall satisfaction.

In any analytics-related project, the same things apply: the team needs to understand the business objectives, the current state (so the new process isn't worse), the risks, and prioritize the high-level features. Most importantly, the team needs to align on what's NOT in the new solution and the prioritizing criteria such as quality, feature completeness, or usability that will be used when making decisions. Agile does not mean no planning.

Trust starts by listening to people and creating a shared vision that sets the right expectations from day one. You can create an achievable plan if everyone knows what you are trying to achieve.

People don't Trust the Data

Let's face it, your data processes get no love. This is usually because this is "too technical." Your users don't care about databases, schemas, tables, or columns, let alone the process of converting raw facts into business-ready insights. It's easy for management to see a fancy dashboard and get excited about the possibility of machine learning, but talk about data and people's eyes gloss over.

It kind of makes sense; most people don't understand how the power grid works. We all take it for granted. We flip a switch, the light turns on, and we move forward. No one cares about electricity until something goes wrong. In a lot of organizations, things go wrong with data more often than you would think. Sometimes people notice right away, but other times failures are silent. When something does go wrong, everyone goes into firefighting mode. Meetings are held, issues are discovered, and patches to "prevent" the failure are put in place. The time to think about the inevitable is not once things break; you need to anticipate failure and design for resilience.

The issue here is that we don't think of the process of going from raw data to insights as a single system. It is all interconnected and needs to be treated as such. When it comes to analytics, sometimes it feels like companies want to build a mansion on a foundation atop quicksand. Initially, all seems fine, and everyone is in the house decorating until someone notices that a corner of the house is sinking. Everyone goes outside, props up the corner, and they happily go back inside to decide what color to paint the next room.

You can't build a house on quicksand; you need to set up repeatable processes with quality built in from the start. If we want collaboration, we have to build it in. If you want to be able to do impact analysis, guess what? You can't retrofit that later if you didn't do it from the start. Having documented analytics is not magic; you need this to be part of the culture and part of the process. The good thing is that many smart people have faced the same issues, and there are examples we can see where people are doing things right.

If you want users to trust data analytics, they need to trust the data, and they need to believe in a solid process that is built on a solid foundation.  

Bad User Experiences Erode Confidence

When you try to please everyone, you please no one, and in many companies, technical teams try to do everything they are asked. They jump through hoops to deliver projects, but it is very common for people to be dissatisfied with the end results. I have also seen new tools used like old ones. Teams sometimes take the approach that the new process is just affecting some part of the current broken process, so they only incrementally change it. I have seen Tableau dashboards that are essentially Excel on the web with some automation.

Instead of asking users what they want, we need to understand what they need and why. What are they trying to accomplish? What's wrong with how they do things today? Is the new process / tool you are putting in place better than what they already have? Sometimes it makes more sense to leave a current process as-is until other parts of the system are improved.

When you understand the real need for an omni-channel dashboard or a sales dashboard, you design the solution to help you achieve that goal. If your users need to quickly get in and out of the tool, you can find ways to reduce the number of clicks it takes them to get there. You simplify access, and you surface the most important information first. You build the solution around them, and more importantly, you are able to justify your decisions and why certain things need to be de-prioritized. When users see that you empathize with them, they trust you. They don't push back on every choice because they know you have their best interests at heart because you have demonstrated time and again that you do care.

Conclusion

Getting decision-makers to trust data analytics is no different than getting anyone to trust anything. You need to start with alignment and set the right expectations; you need to build end-to-end processes that are robust; and you need to deliver the tools that facilitate the job users do.

Any experienced data engineer will tell you that efficiency and resource optimization are always top priorities. One powerful feature that can significantly optimize your dbt CI/CD workflow is dbt Slim CI. However, despite its benefits, some limitations have persisted. Fortunately, the recent addition of the --empty flag in dbt 1.8 addresses these issues. In this article, we will share a GitHub Action Workflow and demonstrate how the new --empty flag can save you time and resources.

What is dbt Slim CI?

dbt Slim CI is designed to make your continuous integration (CI) process more efficient by running only the models that have been changed and their dependencies, rather than running all models during every CI build. In large projects, this feature can lead to significant savings in both compute resources and time.

Key Benefits of dbt Slim CI

• Speed Up Your Workflows: Slim CI accelerates your CI/CD pipelines by skipping the full execution of all dbt models. Instead, it focuses on only the modified models and their dependencies and uses the defer flag to pull the unmodified models from production. So, if we have model A, B and C yet only make changes to C, then only model C will be run during the CI/CD process.

• Save Time, Snowflake Credits, and Money: By running only the necessary models, Slim CI helps you save valuable build time and Snowflake credits. This selective approach means fewer computational resources are used, leading to cost savings.

dbt Slim CI flags explained

dbt Slim CI is implemented efficiently using these flags:

--select state:modified:  The state:modified selector allows you to choose the models whose "state" has changed (modified) to be included in the run/build. This is done using the state:modified+ selector which tells dbt to run only the models that have been modified and their downstream dependencies.

--state <path to production manifest>: The --state flag specifies the directory where the artifacts from a previous dbt run are stored ie) the production dbt manifest. By comparing the current branch's manifest with the production manifest, dbt can identify which models have been modified.

--defer: The --defer flag tells dbt to pull upstream models that have not changed from a different environment (database). Why rebuild something that exists somewhere else? For this to work, dbt will need access to the dbt production manifest.

You may have noticed that there is an additional flag in the command above.  

--fail-fast: The --fail-fast flag is an example of an optimization flag that is not essential to a barebones Slim CI but can provide powerful cost savings. This flag stops the build as soon as an error is encountered instead of allowing dbt to continue building downstream models, therefore reducing wasted builds. To learn more about these arguments you can use have a look at our dbt cheatsheet.

dbt Slim CI with Github Actions before dbt 1.8

The following sample Github Actions workflow below is executed when a Pull Request is opened. ie) You have a feature branch that you want to merge into main.

Workflow Steps

Checkout Branch: The workflow begins by checking out the branch associated with the pull request to ensure that the latest code is being used.

Set Secure Directory: This step ensures the repository directory is marked as safe, preventing potential issues with Git operations.  

List of Files Changed: This command lists the files changed between the PR branch and the base branch, providing context for the changes and helpful for debugging.

Install dbt Packages: This step installs all required dbt packages, ensuring the environment is set up correctly for the dbt commands that follow. ‍

Create PR Database: This step creates a dedicated database for the PR, isolating the changes and tests from the production environment.

Get Production Manifest: Retrieves the production manifest file, which will be used for deferred runs and governance checks in the following steps.

Run dbt Build in Slim Mode or Run dbt Build Full Run: If a manifest is present in production, dbt will be run in slim mode with deferred models. This build includes only the modified models and their dependencies. If no manifest is present in production we will do a full refresh.

Grant Access to PR Database: Grants the necessary access to the new PR database for end user review.

Generate Docs Combining Production and Branch Catalog: If a dbt test is added to a YAML file, the model will not be run, meaning it will not be present in the PR database. However, governance checks (dbt-checkpoint) will need the model in the database for some checks and if not present this will cause a failure. To solve this, the generate docs step is added to merge the catalog.json from the current branch with the production catalog.json.

Run Governance Checks: Executes governance checks such as SQLFluff and dbt-checkpoint.

Problems with the dbt CI/CD Workflow

As mentioned in the beginning of the article, there is a limitation to this setup. In the existing workflow, governance checks need to run after the dbt build step. This is because dbt-checkpoint relies on the manifest.json and catalog.json. However, if these governance checks fail, it means that the dbt build step will need to run again once the governance issues are fixed. As shown in the diagram below, after running our dbt build, we proceed with governance checks. If these checks fail, we need to resolve the issue and re-trigger the pipeline, leading to another dbt build. This cycle can lead to unnecessary model builds even when leveraging dbt Slim CI.

‍

Leveraging the --empty Flag for Efficient dbt CI/CD Workflows

The solution to this problem is the --empty flag in dbt 1.8. This flag allows dbt to perform schema-only dry runs without processing large datasets. It's like building the wooden frame of a house—it sets up the structure, including the metadata needed for governance checks, without filling it with data. The framework is there, but the data itself is left out, enabling you to perform governance checks without completing an actual build.

Let’s see how we can rework our Github Action:

Workflow Steps

Checkout Branch: The workflow begins by checking out the branch associated with the pull request to ensure that the latest code is being used.

Set Secure Directory: This step ensures the repository directory is marked as safe, preventing potential issues with Git operations.  

List of Files Changed: This step lists the files changed between the PR branch and the base branch, providing context for the changes and helpful for debugging.

Install dbt Packages: This step installs all required dbt packages, ensuring the environment is set up correctly for the dbt commands that follow. ‍

Create PR Database: This command creates a dedicated database for the PR, isolating the changes and tests from the production environment.

Get Production Manifest: Retrieves the production manifest file, which will be used for deferred runs and governance checks in the following steps.

*NEW* Governance Run of dbt (Slim or Full) with EMPTY Models: If there is a manifest in production, this step runs dbt with empty models using slim mode and using the empty flag. The models will be built in the PR database with no data inside and we can now use the catalog.json to run our governance checks since the models. Since the models are empty and we have everything we need to run our checks, we have saved on compute costs as well as run time.  

Generate Docs Combining Production and Branch Catalog: If a dbt test is added to a YAML file, the model will not be run, meaning it will not be present in the PR database. However, governance checks (dbt-checkpoint) will need the model in the database for some checks and if not present this will cause a failure. To solve this, the generate docs step is added to merge the catalog.json from the current branch with the production catalog.json.

Run Governance Checks: Executes governance checks such as SQLFluff and dbt-checkpoint.

‍Run dbt Build: Runs dbt build using either slim mode or full run after passing governance checks.

Grant Access to PR Database: Grants the necessary access to the new PR database for end user review.

By leveraging the dbt --empty flag, we can materialize models in the PR database without the computational overhead, as the actual data is left out. We can then use the metadata that was generated during the empty build. If any checks fail, we can repeat the process again but without the worry of wasting any computational resources doing an actual build. The cycle still exists but we have moved our real build outside of this cycle and replaced it with an empty or fake build. Once all governance checks have passed, we can proceed with the real dbt build of the dbt models as seen in the diagram below.

‍

Conclusion

dbt Slim CI is a powerful addition to the dbt toolkit, offering significant benefits in terms of speed, resource savings, and early error detection. However, we still faced an issue of wasted models when it came to failing governance checks. By incorporating dbt 1.8’s  --empty flag into your CI/CD workflows we can reduce wasted model builds to zero, improving the efficiency and reliability of your data engineering processes.

🔗 Watch the vide where Noel explains the  --empty flag implementation in Github Actions:

Implementing dbt (data build tool) can revolutionize your organization's data maturity, however, if your organization is not ready to take advantage of the benefits of dbt it might not be the right time to start. Why? Because the success of data initiatives often hinges on aspects beyond the tooling itself.  

Many companies rush into implementing dbt without assessing their organization’s maturity and this leads to poor implementation. The consequences that come from a poorly implemented dbt initiative can leave the organization frustrated, overwhelmed with technical debt, and wasted resources. To avoid these pitfalls and ensure your organization is truly ready for dbt, you should complete an assessment of your organization's readiness by answering the questions presented later in this article.

What is Data Maturity?

Before diving into the maturity assessment questions, it’s important to understand what data maturity means. Data maturity is the extent to which an organization can effectively leverage its data to drive business value. It encompasses multiple areas, including:

Data-Driven Culture: Fostering an environment where data is integral to decision-making processes.

Data Quality: Ensuring data is accurate, consistent, and reliable.

Data Governance: Implementing policies and procedures to manage data assets.

Data Integration: Seamlessly combining data from various sources for a unified view.

A mature data organization not only ensures data accuracy and consistency but also embeds data-driven decision-making into its core operations.  

How dbt Improves Data Maturity

By leveraging dbt's features, organizations can significantly enhance their data maturity, leading to better decision-making, improved data quality, robust governance, and seamless integration. For example:

Data-Driven Culture: By using dbt, you can improve many aspects that contribute to creating a data-driven culture within an organization. One way is by encouraging business users to be involved in providing or reviewing accurate model and column descriptions which are embedded in dbt. You can also involve them in defining what data to test with dbt. Better Data Quality will improve trust in the data. More trust in the data will always lead to more frequent use and reliance on it.

Data Quality and Observability: dbt enables automated testing and validation of data transformations. This ensures data quality by catching issues like schema changes or data anomalies early in the pipeline. As your data quality and data observability needs grow you can assess where you are on the data maturity curve.  For example, in a sales data model, we can write tests to ensure there are no negative order quantities and that each order has a valid customer ID. With dbt you can also understand data lineage and this can improve impact and root cause analysis when insights don’t seem quite right.

Data Governance: dbt facilitates version control and documentation for all transformations, enhancing transparency and accountability. Organizations can track changes to data models ensuring compliance with data governance policies.

Data Integration: dbt supports the integration of data from multiple sources by providing a framework for consistent and reusable transformations. This allows for the creation of unified data models that provide a holistic view of business operations.

dbt Readiness Assessment

Now that we understand what data maturity is and how dbt can help improve it, you might be ready to jump on the dbt bandwagon. But first, we encourage you to assess your organization’s readiness for dbt. The journey to data maturity involves not only choosing the right tools but also ensuring that your organization is philosophically and operationally prepared to take full advantage of these tools. It is important to recognize that dbt’s approach requires a shift in mindset towards modern data practices, emphasizing transparency, collaboration, and automation.

To determine if your organization is mature enough for dbt or if dbt is the right fit, consider the following assessment questions:

Are you philosophically aligned?

dbt requires a philosophical alignment with its principles, such as ELT (Extract, Load, Transform) instead of the traditional ETL (Extract, Transform, Load) approach. dbt is also based on idempotency meaning that given the same input, you will always get the same output. This is different than traditional ETL that may use incompatible constructs like Auto-Incrementing Primary Keys. If your organization prefers processes that are incompatible with dbt’s methodology, you will face challenges fighting the dbt framework to make it do something it was not intended to do.

Are you just going to do a lift and shift?

Simply migrating existing processes and code to dbt without rethinking them won’t leverage dbt’s full potential. Assess whether you’re ready to redesign your workflows to take advantage of dbt’s capabilities such as incremental tables, snapshots, seeds, etc.

Are you going to take advantage of features like data quality and documentation?

dbt offers excellent features for data quality and documentation. Evaluate if your team is prepared to prioritize the utilization of these features to enhance transparency and trust in your data. Tests and model descriptions will not write themselves. When it comes to good descriptions, they shouldn't come from a data engineering team that does not know how the data is used or the best data quality rules to implement. Good descriptions must involve business user review at a minimum.

Are you going to open up dbt to teams outside of IT?

The goal of dbt is to empower various teams including IT and business users by using the same tooling. Consider if your organization is ready to foster this cross-functional collaboration. When you implement dbt correctly, you will empower anyone who knows SQL to contribute. You can have multiple teams contribute to the insight delivery process and still ensure proper governance and testing before updating production.

Are you going to automate processes like deployment and testing?

Automation is key to achieving efficiency with dbt. Implementing automated deployment, testing, and CI/CD pipelines can significantly improve your workflows. If you aren’t ready to automate, the benefits of dbt may not be fully realized. If you simply put in dbt without thinking about the end-to-end process and the failure points, you will miss opportunities for errors. The spaghetti code you have today didn't happen just because you were not using dbt.

Are you changing more than just tooling?

dbt is a framework, not a silver bullet. Merely changing tools without altering your underlying processes will not solve existing issues. This is a huge issue with organizations that have not done the work to create a data-driven culture. Assess if your team is ready to adopt better naming conventions and more structured processes to make data more understandable.  

Conclusion

Data immaturity might manifest as a reliance on manual processes, lack of data quality controls, or poor documentation practices. These factors can derail the effective implementation of dbt since dbt thrives in environments where data practices are robust and standardized. In other words, dbt alone will not solve these problems.

Ensuring your organization is ready for the changes that come with implementing dbt is not just best practice, it is essential for success. By thoroughly assessing your readiness, you can avoid technical debt, optimize your workflows, and fully harness the power of dbt. Remember, dbt is a powerful tool, but its effectiveness depends on the readiness of your organization to improve data practices and its alignment with dbt’s philosophy.

‍There is no doubt about the transformative potential of big data and analytics. This is particularly true for the Life Science sector as implementing technologies and ideologies can revolutionize drug development, tailor medicines to individual needs, dramatically improve patient care and more. The data supports this, with the longest running report of Fortune 1000 CIOs by Wavestone, showing 87.9% of companies believe “investments in Data & Analytics are a Top Organizational Priority.”  

Such great promise and high buy in should mean easy cultural adoption, right? Well not exactly. The 2024 report shows there has been a notable improvement; the percentage of top executives facing significant challenges in culture, people, and process/organization decreased from a staggering 90% in 2020 to 77.6% in 2024. While this reduction signifies a positive trend towards addressing these issues, the fact that more than three-quarters of leaders still encounter these problems underscores the widespread nature of these challenges. The persistently high percentage highlights the need for continued and focused efforts to overcome these barriers.  

In an effort to help further lower the 77.6%, this article aims to cover the benefits of data in the Life Science sector, highlight common culture issues, and provide some solutions to the problem. If your organization is among those facing these struggles, know that you are not alone.

Benefit of data in Life Science sector

The life science sector stands on the brink of a digital transformation, powered by the strategic use of data. It's clear that its impact is far-reaching, transforming every facet from research and development to patient care and beyond. Below are some examples of what this data-driven culture can improve.

• Accelerating Drug Development: One of the most significant benefits of data lies in its ability to streamline and accelerate the drug development process. By harnessing big data and advanced analytics, life science organizations can identify potential drug candidates faster than ever before. This not only reduces the time and cost associated with bringing new treatments to market but also allows for more targeted and effective therapies.
• Enhancing Personalized Medicine: Personalized medicine, tailored to the individual characteristics of each patient, is becoming a reality thanks to data. By analyzing vast datasets, including genetic information, researchers can understand how different people are likely to respond to treatments. This leads to more effective and personalized care plans, improving patient outcomes and reducing the likelihood of adverse reactions.
• Improving Clinical Trials: Data analytics revolutionize the way clinical trials are conducted. By leveraging data, scientists can design better trials, select suitable candidates more accurately, and monitor results in real time. This not only increases the efficiency and efficacy of trials but also enhances patient safety by identifying potential issues earlier in the process.
• Driving Operational Efficiency: Beyond research and patient care, data also plays a crucial role in enhancing the operational efficiency of life science organizations. From streamlining supply chains to optimizing resource allocation, data-driven insights help companies operate more efficiently, reduce costs, and respond more quickly to market changes.
• Fostering Innovation: At its core, the utilization of data fosters an environment of innovation within the life science sector. By providing a wealth of information and insights, data encourages researchers to explore new hypotheses, uncover hidden patterns, and push the boundaries of what's possible in medical science.
• Informing Policy and Decision-Making: Lastly, data aids in the development of more informed policies and decisions. By analyzing trends, outcomes, and costs, policymakers and healthcare providers can make evidence-based decisions that lead to better health outcomes and more efficient use of resources.

Digital transformation in healthcare - Key challenges

As we identified earlier in this article, culture, people, and process/organization challenges are the biggest obstacles for companies to achieve digital transformation and become data-driven. Identifying specific challenges is key to developing a solution.  

• Resistance to Change: Cultural norms and resistance to change within organizations can impede the adoption of new data strategies, especially when there is a lack of buy-in from key stakeholders. This can be especially present in the Life Science sector which tends to lean risk averse.
• Overemphasis on Buzzwords and End Products: Initiatives often start with buzzwords like 'Gen AI', focusing primarily on the final product without addressing underlying flawed infrastructures and processes. This inclination towards technology is because technology is tangible, you can talk to vendors and see their websites, however, consistent findings from the annual report reveal that technical challenges constitute only 23.4% of the issues; the bulk stems from ingrained cultural problems within organizations. Meaning, adopting new technologies without addressing these foundational processes is likely to lead to suboptimal outcomes.
• Lack of Alignment: When different departments or teams pursue the same objective without proper coordination, it results in alignment issues. These misalignments act as barriers to the seamless execution of unified data strategies, causing data management inefficiencies. Often, this situation stems from a lack of centralized guidance and strategic direction. It is important for organizational leaders to bridge these gaps through a top-down approach, ensuring all components work in harmony towards the collective aim.
• Cultural Complexity: Large enterprises especially face this issue. Cultural complexities, such as diverse social and cultural preferences, values, and disparities in knowledge, can pose challenges in implementing data strategies, particularly in institutions spanning developed and developing countries. This is why taking the time to align fundamentally is especially important for the enterprise.
• Misplaced Prioritization of Data: Often, data is viewed as a technology concern for IT to handle, while other teams focus on deliverables like patient service or clinical trials. This perspective is problematic as data is crucial for analyzing and ensuring the success of these initiatives. Typically, data management is considered too late in the process, when the necessary data may not have been effectively captured for analysis.  

Solutions to digital transformation challenges

Questioning the status quo

Pharma companies tend to be risk averse due to the nature of the data they are responsible for. This leads to limitations and constraints to innovative solutions. These constraints are often accepted without understanding the rationale behind these constraints or challenging them. New technology offers new ways to manage data that were not available in the past. If a company remains the most risk averse and the most conservative this will stifle innovation. The key takeaway is that it is not only about technology but about the new process potential that this new tech provides.

Focus on fundamentals

This goes back to the foundation. Technology changes, initiatives change, but data truths do not. These truths involve thinking about end-to-end processes, data quality, documentation, good guidelines conventions, data governance, reducing points of failure ect. It is easy to get swept up by the latest trend and want to jump in, but you cannot build a house on quicksand; new trends like Gen AI still require these things underneath.

Fundamental alignment

This starts with aligning the team. Alignment is one of the 3 core pillars to a data-driven culture. People will start out thinking they are on the same page because they are using the same terminology. But we all have different ideas influenced by our experiences. So, it is important to gather all stakeholders and go through a true alignment process which includes figuring out the current state, pain points, and aligning on the solution.  

Fundamental alignment includes adopting a top-down approach. The efforts one team is making are sure to affect the efforts of another. Leadership must align and connect the dots between all moving parts and understand that things are not living in isolation. This will solve headaches downstream and ensure a smoother process.  

Quick wins

Alignment alone won’t be enough to combat culture challenges. True alignment can lead to overly ambitious projects that may prove difficult to execute. This is the classic trap of trying to do too much at once. It is important to start with manageable, small-scale projects that can provide immediate benefits. These quick wins are vital for positively influencing the organizational culture. Additionally, they help maintain momentum: if a larger initiative is slow to yield results, these smaller successes ensure continuous progress. By allowing for ongoing adjustments and reprioritizations, these projects help prevent initiatives from being abandoned.

Conclusion

While the potential of big data and analytics in transforming the Life Science sector is undeniable, and investment in data is on the rise, the journey towards a data-driven culture remains one of the biggest challenges for data integration. The path forward requires a balanced approach that combines technology with fundamental changes in culture and processes. By embracing these changes, the life sciences sector can overcome existing barriers and fully harness the power of data to advance medical science and improve patient outcomes. At Datacoves we are passionate about helping companies achieve a data-driven culture. See how Datacoves helped Johnson&Johnson innovate their tech stack.

Prefer a Video?

This article was inspired by RAN BioLinks’ podcast episode “Why Life Science Organizations Fail to Implement Effective Data Strategies”.  The detailed conversation with Noel Gomez, a seasoned expert in data management within the life science industry, explores the critical challenges and innovative solutions for effective data strategies in healthcare.

Companies are investing heavily to become data-driven and to democratize data access. However, many are not achieving the transformative outcomes they expected.

The core issue? A lack of trust.

This mistrust stems from a lack of focus on core aspects that ensure a robust data-driven culture and critical mistakes in these areas.

Fortunately, these mistakes are self-inflicted which means they can be fixed, and this article aims to help highlight and address these pitfalls. By understanding and adhering to the core pillars of a data-driven culture and avoiding the common mistakes, organizations can develop and maintain a data-driven culture that people can trust.

What is data-driven culture?

It is no secret that there is power and opportunity in data, and data-driven culture is the approach which aims to take advantage of that.

A data-driven culture is not about hastily adopting the latest tools or technologies in the hope of resolving data challenges. This common mistake often leads to a focus on immediate results or 'shiny objects', such as acquiring cutting-edge technology or hiring new talent. Unfortunately, this approach tends to overlook essential priorities and gradually erodes the foundation of a data-driven culture: Trust in the data.

Many companies struggle with effectively using analytics because they overemphasize these immediate goals – the 'destination' – rather than appreciating the foundational journey necessary for impactful analytics. This journey involves more than just technology; it requires a shift in mindset and approach.

Data-driven culture represents an organizational approach where data is the cornerstone of decision-making processes. In such a culture, decisions are primarily informed by data analysis, rather than relying exclusively on intuition or past experiences. This approach involves strategically employing data at every level of the organization. It fosters an environment where data is not just an asset but the main driver of strategy, innovation, and operational choices. By harnessing the power and opportunities offered by data, a data-driven culture ensures that decisions across the organization are grounded in solid evidence and analytical insight, enhancing the overall decision-making quality and efficacy.

Key features of data-driven culture include:

Empowered Decision Making: Decisions are based on data analysis, leading to objective and impactful outcomes.

Accessibility of Data: Data is accessible across the organization, breaking down silos and empowering all employees.

Investment in Technology: Adequate tools and technologies are provided for effective data collection and analysis.

Data Literacy: Continuous training is provided to enhance the workforce's understanding and use of data.

Quality and Governance: High standards of data accuracy and security are maintained.

Agility: The organization adapts quickly to insights derived from data.

Collaborative Integration: Data insights are shared and integrated across various functions.

Outcome-Focused: Emphasis on measurable results driven by data insights.

Building a data-driven culture: Core pillars

All of that sounds great, but how do we achieve a data-driven culture?

Like mentioned earlier in the article, true success in analytics comes not from merely chasing new tools or methodologies but from establishing three core pillars as part of a Data-Driven Culture:

• Fundamental Alignment: It's essential to align analytics strategies with core business objectives, ensuring everyone involved shares a common vision and understanding.
• User-Focused Solutions: The end goal of analytics should be to serve the user's needs. This involves designing solutions that are practical, add real value, and enhance the decision-making process.
• Efficient Data Management: Implementing robust data processes is key. This involves ensuring data accuracy, accessibility, and understandability, which are crucial for informed decision-making.

By refocusing on these foundational elements, businesses can drive more meaningful and sustainable results from their analytic endeavors, leading to overall project success and satisfaction.  

Let's dive deeper into the core pillars and examine the common pitfalls within each pillar that I have observed lead to challenges.

Lack of alignment reduces faith in the solution: Fundamental alignment

Fundamental alignment is about synchronizing analytics strategies with the organization's core business objectives. This ensures everyone involved, from executives to frontline employees, share a common vision and understanding of what analytics aims to achieve. This alignment is crucial for creating a unified direction in data-driven initiatives and ensuring that every analytics effort contributes meaningfully to the overall business strategy.

This sounds great right? So much so that every project I've participated in began with high hopes and enthusiasm. Initially, there was a sense of unity – funding secured, partnerships with vendors established, and the latest technology acquired. This honeymoon phase of the data driven transformation, filled with optimism, had everyone working diligently, with management receiving regular updates and a general belief that we were on the right track.

Pitfall

The real test emerged when critical decisions were required. This was the point where the honeymoon phase often faded, revealing a lack of true alignment. Meetings became prolonged discussions where the team struggled to reach consensus. This challenge stemmed from either not spending enough time initially to ensure everyone was on the same page or not conducting a discovery phase at the start of the project.

Although we agreed on high-level objectives like digital transformation and self-service analytics, there was a misalignment in our deeper understanding and perspectives. We were each influenced by our varied backgrounds and expertise in different aspects of the project.

‍

Solution

This led me to a crucial fact: the importance of alignment before action. In projects where we dedicated time upfront for structured alignment and thorough product discovery, we not only achieved better estimations but also greater overall satisfaction. It became evident that successful analytics projects require a deep understanding of business objectives, the current state, potential risks, and a clear prioritization of features. This was because we developed a clear understanding and set up expectations that people could rely on throughout the course of implementation.

Crucially, alignment also involves clarity on what the project will not address, alongside the criteria for prioritization such as quality, completeness of features, and usability. Embracing agility does not mean forgoing thorough planning.

Ultimately, building trust in any project begins with listening, creating a shared vision, and setting the right expectations from the start. A well-defined and achievable plan, understood and agreed upon by all, is the foundation of success.

Bad user experiences erode confidence: User-focused solutions  

The end goal of analytics should be to serve the user's needs and involve designing practical solutions that add real value and enhance decision-making processes. This means creating analytics tools and processes that are intuitively aligned with how users work and make decisions, ensuring that these tools are not just technically proficient but also practically useful.

Pitfall

There are two pitfalls to avoid.

1. Trying to please everyone often leads to pleasing no one.

This is a common scenario in many companies where technical teams strive to meet all demands. Despite their efforts to deliver on projects, dissatisfaction with the end results is frequent.

2. Not addressing the actual user pain points.

This happens when the user does not actually get a good working solution out of the process.

Solution

During discovery it is important to discuss what is in scope, out of scope, essential, and nice to have. By categorizing this way you can better understand the needs of the group and use it to guide the process. With this process done, you can move forward with confidence that you are addressing the most important pain points.

Now that we have defined the pain points, the next step is to fully understand. The key is to not only understand their needs but the reasons behind them. What are the goals they're trying to achieve? What are the shortcomings of their current methods? Is the new process or tool genuinely an improvement over what they currently have?  For example, if users need to navigate a tool quickly, finding ways to reduce unnecessary clicks and simplifying access becomes important. Sometimes, it's more practical to keep an existing process unchanged until other parts are enhanced. By bringing the most critical information to the forefront, the solution becomes more user centric.

It is important to have these needs in mind at the beginning of the project and strive to truly understand. If not, you risk investing time, money, and resources in a tool that users don't need, and this can have a detrimental effect on the overall culture.

More importantly, this approach allows you to justify your decisions and explain why certain aspects are prioritized over others. When users see that their needs and challenges are understood and addressed, they are more likely to trust and accept the solutions provided. This trust is built through consistently demonstrating that their best interests are at heart.  

Data driven transformation: Efficient data management

Efficient data management involves implementing robust processes to ensure data accuracy, accessibility, and understandability. This pillar is key to informed decision-making as it underpins the reliability of data-driven insights. Effective data management includes organizing, storing, and safeguarding data to make it readily available and useful for users across the organization.

Pitfall

Let's face it, your data processes get no love. This is usually because they are "too technical." Users often do not concern themselves with databases, schemas, tables, or columns, let alone the process that turns raw facts into business-ready insights. It is easy for management to get excited about a fancy dashboard and the potential of Machine Learning and Gen AI, but when it comes to the actual data, interest tends to wane.

It makes sense; most people don't understand how the power grid works. We take it for granted that we flip a switch and expect the lights to turn on. We move on without a second thought. No one really cares about electricity until something goes wrong. Similarly, in many organizations, data issues often go unnoticed until a failure occurs. Sometimes these issues are immediately apparent, but other times they are silent. When a failure does happen, there is a scramble to fix it. Meetings are held, issues are identified, and patches are implemented to "prevent" future failures. However, the best time to think about potential problems isn't after they happen, but before — building systems that anticipate and are designed for resilience.

The real issue is that the process from raw data to insights isn't often viewed as a single system. It is all interconnected and should be treated as such. In the world of analytics, it sometimes feels like companies are trying to build a mansion on a foundation of quicksand. Initially, everything seems fine, and everyone is busy with their tasks, but when the foundation starts to give way, the focus shifts to propping up the weak points. You can't effectively build on quicksand; you need solid, repeatable processes from the start.

Solution

The focus should be on building systems that anticipate challenges and are designed for resilience. This involves integrating data management practices into the company's culture from the start, ensuring users trust the data and the processes that generate insights. If you want effective collaboration and impact analysis, these are difficult to retrofit later — they need to be part of the initial plan. Documented analytics isn't a magical solution; it needs to be ingrained in the culture and process from the beginning. The good news is that there are many examples and best practices from those who have navigated these challenges successfully.

For users to truly trust in analytics, they need to have faith in the data and the processes that generate it. They need to see and believe in a robust system built on a solid foundation.

Conclusion  

To achieve a data-driven culture, companies must refocus on three core pillars: fundamental alignment, user-focused solutions, efficient data management, and avoid common mistakes in these areas. Success in analytics isn't about chasing new tools or methodologies but about building a robust system from the ground up, aligning everyone's vision, and creating practical, value-added solutions. Prioritizing foundational elements over immediate shiny objects will lead to more meaningful, sustainable results and will build trust in the analytics process.

Digital transformation is often seen through the lens of technological advancement and process optimization. Most blog posts and guides out there revolve around implementing new software, automating tasks, and digitizing operations. Yet, there's a pivotal element that's frequently overlooked in these discussions, especially when it comes to an enterprise: the mindset and culture within an organization. This article aims to shed light on why this is crucial in achieving true digital transformation. But first, let's investigate what digital transformation is and why it is important.

Digital transformation defined

Digital transformation is the integration of digital technology into all areas of a business, fundamentally changing how it operates and delivers value to customers. It is more than just a technological upgrade; it is a cultural shift that requires organizations to continually challenge the status quo, experiment, and get comfortable with failure. This often means walking away from long-standing business processes that companies were built upon to embrace new ways of working. Most organizations find this part the most challenging.

Why is digital transformation important

• Keeping Up with the Digital Economy: In a world where technology evolves rapidly, businesses must adapt to stay relevant. Digital transformation allows companies to remain competitive in an increasingly digital economy.
• Enhanced Data Collection and Analysis: Digital transformation creates a system for gathering the right data and fully utilizing it for better business decisions, efficiencies, and customer insights.
• Customer Expectations: Today's customers expect a seamless digital experience. Businesses need to engage with customers on their terms, using digital tools and platforms that are convenient and user-friendly.
• Increased Agility and Innovation: Adopting digital solutions empowers organizations to be more agile and responsive to changes in the marketplace or industry. It fosters a culture of innovation, encouraging new ideas and approaches.
• Operational Efficiency: Automation and streamlining of processes reduce operational costs and improve efficiency. This allows employees to focus on more strategic tasks that add value to the business.
• Risk Management and Compliance: With the increasing importance of data security and privacy, digital transformation helps businesses keep up with changing regulations and protect sensitive information.
• Sustainability: Digital processes can reduce waste and improve energy efficiency, contributing to more sustainable business practices.

Enterprise digital transformation

To achieve digital transformation in an enterprise 9 times out of 10 there must be a change in company culture. However, changing a company's culture is a formidable task. It is rare to hear statements like, “We need to fundamentally change our problem-solving approach.” This realization became clear to me through my past experiences as I noticed that managers often lacked the influence to drive change at the highest organizational levels. Additionally, the pressure to deliver quick results within budget cycles frequently hindered genuine cultural transformation.

During my tenure at various companies, under numerous managers, the consistent message was the need for improvement. However, I have come to understand that organizations, much like fireflies, develop their own rhythms. It is this unique rhythm that sets apart innovative and transformative companies from those that merely follow without achieving similar success. What do I mean by this? Let’s turn to nature for an explanation.

Firefly phenomenon - Does it mean conformity or innovation in your organization?

Nature is fascinating, especially when observing how hundreds or thousands of fireflies can synchronize their flashes.

In organizations, a similar phenomenon occurs. People will sync up and follow the status quo, even if it is not what is best for the organization. This dramatically hinders digital transformation because the loudest are not always right and yet they cause others to sync up with them. This will cause innovation to be stopped in its tracks.

In addition to this firefly phenomenon, often action differs from ambition. I recall a staff meeting with a former CIO discussing a future less dependent on Microsoft and more open to non-Windows devices. It was clear that iPhones were going to change the corporate landscape. Despite this, every new tool implemented was still optimized for Internet Explorer. This discrepancy between ambition and action often drives analytical people like me to frustration. To effect change, persistence is key. I have had ideas initially dismissed as “not my job,” only to see one later turn into a patented invention.

This manifests itself in other ways as well; have you ever seen a company advocate for fewer meetings while simultaneously criticizing those who do not include “everyone” in decision-making? I have been in such situations and can attest that decision-making by committee is not inherently superior. In fact, the more people involved in an initiative, the less effective it tends to be. This, I believe, is due to the Dunning-Kruger effect.

The more people you involve in a transformation initiative, the more likely the discussions will deteriorate to bike shedding discussions. When there is a disconnect between what is said and what is done, people take notice, and it breeds discontent.

One firefly can only affect their neighbors

Even in my most successful transformation initiatives, the radius of transformation has been limited to my sphere of influence. Sure, some of my tools and processes got global and cross-functional acceptance, but the underlying principles never took hold because they were too radical for the organization at the time. I was not part of the IT organization so the things I did were typically seen as shadow IT. Instead of focusing on what I should not be doing, it would have been more progressive for them to see how I was practicing Agile principles. They could have inquired about how my project was doing DevOps before that was in style, or how it was that this non-sanctioned product was extremely well received and people sought me out to help them improve their processes.

This means if you want the organization to be more innovative, you need to find the obstacles that hold people back from being innovative. Often politics and bureaucracy impact an initiative more than the solution itself. If you force everyone to comply with existing tools and processes, then you are imposing a constraint on the team that will limit innovation.

A typical way this manifests itself is leadership pushing the idea that one platform or process can solve every need. This can come in the form of imposing that a particular group do data transformation, or a visualization tool be the way that everyone can do analytics. I have never seen one tool that is good at everything, and you end up balancing the single solution with an unmanageable array of tools and processes. A healthy organization is a learning organization that is always open to improvement.  When management encourages pushing boundaries and not taking anything as fact then the company can innovate.  

A great example of driving innovation is seen in the approach of Steve Jobs, co-founder of Apple Inc. Jobs was known for his ability to challenge conventional wisdom and existing standards in the technology industry. He emphasized the importance of understanding the fundamental principles underlying a problem to innovate and create groundbreaking solutions. One notable instance was the development of the iPhone, which revolutionized the smartphone industry. Jobs and his team did not just improve on existing phones; they rethought what a phone could be, focusing on user experience and simplicity. This approach led to a product that dramatically altered how people interact with technology.  

As a leader, you need to look for the fireflies who are using first principles like Steve Jobs to deliver innovative solutions and nurture, or create, a corporate culture that truly challenges what has been done without artificial constraints.

Reasoning by first principles removes the impurity of assumptions and conventions. What remains is the essentials. It’s one of the best mental models you can use to improve your thinking because the essentials allow you to see where reasoning by analogy might lead you astray.

Most fireflies eventually comply, or fly away – Loss of innovators

The transformative and innovative thinkers will either comply or leave, both of which are undesirable. In my case I tended to leave. In every organization where I have worked, I have managed to make a significant impact, often through sheer determination. During my time at one such company, our goal was to introduce a data catalog. By analyzing the problem I was able to discern what was essential for our organization vs an elaborate and idealistic vision which was capable of doing everything. While the IT organization felt it would be better to create a home-grown catalog I understood that our biggest obstacle was getting people to use a catalog in the first place, so time to market was critical. I found that Alation met the needs we had and IT kept to their vision to build an all encompassing catalog, In 3 months I had deployed Alation and 1.5 years later, the home grown solution was a tenth as good. This approach of breaking down the problem to its basic elements and building up from there was critical. It is often underestimated how challenging it is to develop and maintain custom software.  This experience highlights the effectiveness of first principles thinking in deploying practical and efficient solutions.

The reality is that not everyone possesses the tenacity to advocate for change, especially in the face of substantial resistance. Not only that, but I have also witnessed people being ostracized for thinking differently, while others were promoted for fitting in. It is crucial to seek out divergent thinkers and consider the validity of their perspectives, instead of forcing them to conform. This is why true digital transformation necessitates a shift in culture.  

When an individual, much like a firefly that does not flash in unison with the rest, finds themselves out of sync with the collective rhythm, they face a decision: conform and synchronize with the group or venture out to find a new collective that resonates with their unique spark.

How do we change the flash for all? Aligning mindsets for transformation

True transformational change must come from the top. Achieving enterprise digital transformation requires a deep and bold questioning of the status quo. We must critically assess our processes: Is a particular task truly necessary for a certain group? Can we identify and eliminate inefficiencies? Will adding another layer of approval or inspection genuinely enhance outcomes? It is essential to remember that human behavior often has a more profound impact than any technology or process we implement. When decision-making is centralized within one group, solutions are inevitably skewed to reflect their viewpoint. Too often, I have witnessed decisions justified by cost considerations that, upon closer inspection, proved detrimental in the broader context. An effective strategy involves analyzing the entire system, recognizing that optimizing the whole may require accepting lower efficiency in some areas.

The key is to align with the needs of users and the organization and engage leadership in this journey. With a united front, tackling the 'corporate dragons' becomes a more manageable endeavor. One practical approach is employing methodologies like the 'Job to be Done' framework.

Conclusion

Company culture and change management are frequently overlooked in the pursuit of process improvement. Employees operate within their limitations, while management ponders the lack of innovation and agility compared to other companies. The simpler path might seem to be increasing staff or updating technology, but the heart of transformation lies in the mindset of the organization. Leaders aiming for a lasting impact must embrace first principles thinking, ready to scrutinize and challenge established norms. Transformational change rarely stems from incremental improvements; truly innovative companies are those that dare to think and act differently. The organization thus faces a pivotal choice: will it adapt to a new rhythm, or compel its 'new fireflies' to fall in line with the existing order?

In the age of data-driven decision-making, companies grapple with the mammoth task of setting up a robust Modern Data Stack. The on premise legacy systems struggle to keep up, and standing up a Modern Data Stack (MDS) isn't just a tech upgrade; it's an essential pivot, ensuring businesses extract actionable insights from the raw data they encounter. However, the road to achieving this is complex and slower than the line at the DMV.

If the responsibility of establishing a Modern Data Stack falls on your shoulders and you're feeling the weight of its time/resource/knowledge-intensive nature, this post offers insights and solutions.We explore the hurdles businesses encounter while shaping their data infrastructure and how you can streamline and expedite the process.

What is a Modern Data Stack?

A Modern Data Stack refers to a suite of tools and digital technologies specifically designed for data management. Within this stack, some tools specialize in collecting data, while others focus on storing or processing it. As data moves through this system, it's transformed from raw input into actionable insights.

Many of these tools come from various providers and must be seamlessly integrated to ensure optimal performance. Leveraging the latest technologies, the modern data stack efficiently manages the entire data lifecycle, from collection to analysis. This stack is both scalable and flexible, ensuring it can adapt and grow with the ever-evolving demands of a business, and provide consistent performance regardless of data volume or complexity.

Below you can see an example Modern Data Architecture Diagram.

‍

Standing up a Modern Data Stack takes time 

The path to a comprehensive end-to-end enterprise data platform is not without challenges. Embarking on such a journey requires diligent research, because the process of migrating to a Modern Data Stack or establishing it from the ground up is intricate and piecemeal. Since there are many individual tools in the Modern Data Stack, you may have to tackle each tool individually so you can focus on setting it up correctly. Given the complexity of the endeavor, even with a skilled team on board, it can take between 6 to 9 months to build a complete end-to-end data solution. This may be frustrating, but understanding the pain points in setting up a Modern Data Stack can help to make educated decisions that accelerate the process.

High level pain points:

• Hundreds of tools to choose from - While having many options can be beneficial, it can also be overwhelming. The vast selection can lead to what's known as "tool overload," making it hard to pick the best fit.
• Difficult to Integrate various tools - Once you've picked your tools, the next step is integration. But with so many different systems and platforms available, getting them to work together can be like solving a complex puzzle.
• Architecting a secure platform - Everyone agrees that data security is critical. Creating a platform that's not just secure but also easy to replicate and audit is challenging and requires careful planning.
• Implementing best practices - The lack of standard processes in the data world can lead to inconsistencies. Finding and applying the best practices isn't just about knowledge; it's also about experience.

• Hidden Costs - Even though a lot of modern data stack tools are freely available, the hidden cost emerges in the form of time - spent on learning, configuring, testing, and refining. It’s like getting a "free puppy"; while there might be zero upfront expenses, the continuous care, attention, and commitment required are far from zero.

‍

Modern Data Stack: Guiding principles for success

A strong data platform is the backbone of good decision-making. It helps us see clear insights fast and strengthens our data teams. When creating or choosing such a system, keep these principles in mind:

• Trustworthiness - There needs to be trust that the data is always right and true.
• Usability - The system should be easy to use and understand.
• Collaboration - Teams should be able to work together in a secure way.
• Reusability - If one part of the system is good, we should be able to use it in other places too.
• Maintainability - There should be automated process and DataOps in place.
• Reliability - It should always work well, detect errors, not break down often, and keep data safe.

‍

Following these rules can help us get the most from our data and make the best decisions

Simplify the Modern Data Stack

Understanding the challenges and intricacies of setting up a Modern Data Stack makes it clear why we need efficient solutions. In the data world things move fast and speed is imperative. While there are numerous tools available that cater to specific components, Datacoves offers a more comprehensive approach, addressing the end-to-end data stack. Datacoves could reduce the setup of your Enterprise Data Platform from the usual 6-9 months to just 2-3 weeks. But how does it achieve this feat?

Datacoves is:

• A Turnkey Solution - Datacoves doesn't just offer a solution; it provides an all-encompassing package designed meticulously to streamline the entire data-to-insight trajectory. This isn't about starting from scratch; it's about leveraging a fully-equipped platform to jumpstart your journey.
• Guidelines and Expertise - No more searching in the dark. Datacoves ensures its users have a clear path ahead. The challenge of standardizing processes, which once seemed like climbing Everest, is simplified, thanks to the expert guidance provided.
• Scalability At Its Finest - Whether you’re a budding team of 3 just starting out or a robust squad of 300, Datacoves has been engineered to scale with your needs, ensuring consistency and efficiency at every stage.
• State-of-the-Art Tools - With tools like a finely-tuned VS Code in the browser with dbt Core, Datacoves ensures users aren't just walking but sprinting from the get-go. It's about giving you the best gear to make your climb smoother.
• Best Practices at Your Fingertips - Datacoves realizes that in the fast-paced world of data, time is of the essence. That's why, through integrated accelerators, it ensures that adhering to industry best practices isn’t a drawn-out quest but just a matter of configuring to your needs.

Highlighting Datacoves' features

Datacoves is not just another platform; it's a game-changer. Its project-based structure integrates seamlessly with any git repository, and it can be swiftly deployed in a private cloud to connect with existing tools. Each project provides multiple environments, facilitating role-based access and ensuring user-specific needs are met.

Here are just a few ways that Datacoves empowers the Data Engineer and Analytics Engineer to deliver quickly:

• Everything in one place - The objective is to streamline the Data/Analytics Engineer's workflow. By consolidating essential tools and functionalities into a single interface, users can load data, review entries in their data warehouse, configure DAGs, write code in VSCode, and more, all without switching tabs.

• ‍Airflow YML Configuration - With this feature, users can bypass the complexities of Python when working with Airflow. Instead, the YML configuration allows for a more direct way to set up your DAGs.

• ‍dbt-coves Extension - Within your vscode workspace, the dbt-coves extension is integrated, making tasks more efficient. Specifically, the "dbt-coves generate sources" command examines your database, updates files, and integrates them into your yml with ease.

• ‍ChatGPT Integration - Embedded directly in your vscode workspace, ChatGPT offers a hassle-free way to seek answers without changing tabs. This feature is especially handy for tasks like creating model descriptions—simply generate, adjust as needed, and move forward.

Datacoves’ Northstar

Datacoves aims to simplify, reduce friction, enhance collaboration, and inject software engineering practices into data operations. It seeks to empower teams, enabling swift productivity and ensuring teams function cohesively.

Intrigued by Datacoves? Dive deeper by watching the full video below or book a demo to experience its magic first-hand.

‍