In the modern data-driven world, corporations are investing significantly in data-related initiatives to generate insights that drive business decisions and stimulate growth. Surprisingly, these initiatives have an extremely low chance of succeeding. For instance, according to Gartner’s research conducted in 2016, approximately 60% of big data projects failed to progress beyond the pilot phase. In 2017, the upper limit was suggested to be as high as 85%. An article from VentureBeat, which analyzed data science projects in 2019, reports an 87% failure rate of such projects. Most recently, RAND Corporation undertook a study of experienced data scientists and engineers in 2024 and concluded that over 80% of AI initiatives fail to reach a useful stage of development, which is almost double the typical IT project failure rate. Finally, according to the MIT’s 2025 State of AI in Business report, 95% of generative AI pilots are projected to fail to deliver any ROI.
There is no universal agreement regarding the precise failure rate of data initiatives since different sources define “failure” differently. Nevertheless, it is evident that a significant fraction of data projects fail to reach maturity and provide any tangible business benefits. Having reviewed numerous such initiatives, I can confidently say that tools are rarely the primary reason for failure. Below, I have highlighted four critical factors that cut data projects short.
1. Lack of Clear Business Objectives
I have seen this derail too many projects for it to be low priority. Someone gets excited about a particular data set or methodology, goes off building something, and then realizes with horror long after the fact that they have no idea what business decision this project was meant to inform. Without defining your objective, you can't determine your priorities, and you can't effectively communicate to stakeholders what value you have provided when the inevitably ask, in frustration, "so what did we get for our time and money?" A project completed without a specific, actionable objective is just research, however well meaning.
2. Poor Collaboration Between Teams
Data projects are at the crossroads of data scientists, engineers, business analysts, and those who will ultimately make a decision based on the results. This is where my experience tells me that good intentions are often buried. Each party comes to the table with their own understanding of the problem to be solved, and collaboration between these groups is often hampered by poor communication and a one-size-fits-all approach. At the end of the day, the model or dashboard that emerges rarely reflects the needs of the business. According to Informatica’s 2025 CDO Insights study, data quality/preparation (43%) and technical maturity (43%) were cited as key impediments to realizing value from AI – and I would argue that these issues can hardly be separated from the technical ones, as they often arise due to the lack of communication between different stakeholders.
3. Insufficient Data Quality and Accessibility
No algorithm, however sophisticated, can make up for bad inputs. I've seen too many cases where projects had to be scrapped after months of waiting for data to become available, clean, and reliable enough to base decisions on. After all, by the time it's ready, the business need often changes. The CDO Insights 2023 report by Informatica highlights data quality and preparation as one of the two biggest impediments to capitalizing on AI, alongside technical maturity. I think it's safe to say that many underestimate the amount of time that needs to be invested in preparing data for discovery and analysis. Most projects have timelines that severely lack when it comes to budgeting for the less glamorous aspects of analytics like visualization and modeling.
6. Inadequate Change Management and Stakeholder Buy-In
Building something is only half the battle convincing an organization to embrace and implement it is the other half, and one that too many neglect to their detriment. Most analytic initiatives require some level of cultural or process change, and without clear leadership and a thoughtful adoption strategy, momentum will be lost in the weeds. According to Gartner, 70-80% of enterprise data initiatives are failing to deliver their intended ROI, while industry research on change management practices suggests that between 60-70% of all changes initiatives will fail entirely if poorly managed. In my own experience, I've seen strong analytic models and methodologies get parked on a shelf because the target audience wasn't engaged or equipped to absorb the information.
Conclusion
In any project I’ve been on, or heard about, the critical success factors were not related to tools. It was mostly clear mission definition, efficient communication, availability of relevant data, and the presence of an actionable strategy towards utilization and adoption by the rest of the organization. As I see it, it is always easier to fix these issues rather than to rely on some universal solution. The main point is to prioritize the problems based on the realistic threat to the project, not based on the size of the investment required for resolution. Having said that, I think it is possible to bring any project to production and get ROI if the four pillars are dealt with honestly and thoroughly.
