DataDrive saves 200+ hours annually by streamlining pipelines with Datacoves
Alex Sweet
Head of Data Platform
.png)
The stability, Kubernetes-backed scalability, and overall reliability of the Airflow environment have been the cornerstone of achieving our data system goals.
Alex Sweet
Head of Data Platform
Customer Introduction
DataDrive provides managed modern analytics services, enabling businesses to make informed decisions without relying on expensive data teams or infrastructure. Using secure cloud technology and a Human-Cantered Data approach, they deliver actionable insights helping organizations achieve measurable, impactful, and data-driven business success.
Executive summary
Datadrive faced significant challenges with dbt Cloud, including limitations with its IDE and CI/CD. The team also struggled to streamline their data workflows, manage complex pipelines, and maintain operational efficiency. They needed a full-fledged orchestrator to manage their workflows without handling the complexity of the orchestration infrastructure. By leveraging Datacoves' managed Airflow, Datadrive benefited from a Kubernetes-backed, scalable Airflow environment, enabling seamless orchestration of diverse data pipelines. Datacoves' customizable IDE, managed dbt Core, and GitHub Actions integration enabled Datadrive to streamline dbt development and simplify workflows. This enhanced flexibility led to greater efficiency, time savings, and a scalable system, allowing Datadrive to focus on delivering client value.
The Challenge
Limitations with dbt Cloud IDE and Slim CI:
DataDrive used dbt Cloud for scheduling dbt jobs and felt its Integrated Development Environment (IDE) was slow, could not be customized like VS Code, and couldn't be used when developing other non-dbt code that lived in a repository. This led them to use dbt Core in VS Code locally for development instead. They also found that dbt Cloud’s Slim CI was too limited and impossible to fit their need for a more complex workflow.
Disjointed processes:
DataDrive's pipelines relied on time-based scheduling, which was neither efficient nor scalable and required tedious manual fixes when something went wrong. Their setup made it challenging to orchestrate various data flows, such as APIs, S3 pipelines, and Fivetran. They needed a more flexible orchestrator to integrate seamlessly with their existing tools while enabling custom workflows and managing secrets.
Infrastructure Maintenance:
DataDrive needed a solution that could meet their unique requirements, simplify integrations, and scale efficiently, all without requiring significant internal technical resources. They wanted to focus on serving their customers rather than dealing with the burden of maintaining infrastructure.
The Solution
Enhanced Flexibility and Interoperability:
DataDrive’s team is highly technical and capable of developing on VS Code locally; however, Datacoves’ in-browser VS Code allowed them to develop with no local installation and configuration. Equipped with managed dbt, support for any Python library, full terminal access, and extensions, the IDE allowed them to customize their development experience to fit their needs. It provided all the benefits of VS Code without the burden of managing versions, dependencies, or upgrades, making their development process more seamless and efficient.
Additionally, Datacoves integrated seamlessly with DataDrive’s GitHub account, offering customizable CI/CD templates for GitHub Actions and flexible Slim CI functionality. By delivering both the "what" and the "how," DataDrive was able to tailor CI/CD workflows to their needs without dealing with the complexity of setting up CI/CD pipelines or managing Docker images.
Kubernetes-Backed, Scalable Airflow:
After evaluating other orchestration solutions, DataDrive chose Datacoves’ managed, stable, and Kubernetes-backed Airflow environment for both development and production. These environments were preconfigured to integrate seamlessly with dbt and included custom decorators and operators, simplifying dependency management and upgrades. The team was able to streamline their pipelines and various ingestion patterns.
Personalized and Managed Solutions:
Datacoves provided DataDrive with a managed yet highly flexible system that streamlined setup and maintenance. With built-in tools for managing secrets, connections, and prebuilt Docker images, along with workflow accelerators, the platform saved the team valuable time—allowing them to focus on delivering insights.
Beyond technology, Datacoves’ fast and responsive support ensured a seamless transition. The team proactively resolved issues and became a trusted partner, offering insights, ready-to-use solutions, and evolving alongside DataDrive to meet emerging needs efficiently.
The Result
Simplified Configuration:
Implementing Datacoves Managed Airflow significantly simplified the configuration and connection of data extraction tools, transformation pipelines, and visualization extracts. The ease of use provided by Datacoves was a major value add, making it far simpler than managing configurations independently. The user-friendly framework eliminated the complexity of sequential orchestration, streamlining processes, and centralizing failure management. Now, instead of troubleshooting multiple points of failure in their orchestration, a single centralized view allows quick identification and resolution of issues.
Unmatched Flexibility:
Datacoves provides a level of flexibility unparalleled by other solutions. The platform enables seamless integration and scaling of data pipelines with additional monitoring capabilities. The ability to write custom Python code tailored to specific needs ensures that users can achieve their goals without being constrained by the platform's limitations. This adaptability has allowed teams to shift focus from infrastructure challenges to delivering meaningful insights and driving their data strategy forward.
Improved Efficiency and Time Savings:
Measurable improvements in data delivery efficiency have been observed since implementing Datacoves. Previously, teams spent approximately four hours a week addressing timing discrepancies and debugging pipeline issues. This effort, along with the associated fatigue, has been virtually eliminated. With Datacoves, on-time data delivery has become the standard, freeing up valuable time for teams to focus on impactful tasks rather than resolving technical faults. The platform’s robust design also supports unique use cases, ensuring seamless operation across multiple local or GitHub environments. "We used to spend about four hours a week on timing discrepancies in pipelines. With Datacoves and Airflow, that’s no longer an issue, as we now have a single point to monitor failures," said Alex.
Conclusion
Datacoves has been a transformative solution for DataDrive, enabling them to overcome significant operational challenges and scale their data systems efficiently. The integration of a customizable, Kubernetes-backed Airflow environment, coupled with a user-friendly interface for managing connections and secrets, provided the flexibility and control needed for their evolving needs. With streamlined workflows, reduced downtime, and enhanced data delivery efficiency, Datacoves proved to be an invaluable technology partner, empowering DataDrive to shift focus from infrastructure concerns to achieving the data strategy goals for their customers.
Related Case Studies
Accelerating dbt Core for Enterprise Data Platform
Migrating from Talend to a dbt-core managed platform
