Study Reveals the Impact of Data Management on AI Progress

Today’s enterprises are eager to leverage artificial intelligence (AI) for a range of applications. However, AI progress is intricately linked to effective data management practices. Ensuring high standards in data quality, access, security, privacy, distribution, and literacy is crucial to avoid the pitfalls that can hinder AI development.

A recent report released by Starburst looks into the current state of data management practices and their impact on AI implementation. The survey gathered insights from 300 IT professionals across the United States and Western Europe. It highlights the challenges, strategies, and future trends that business leaders should consider to harness the potential of AI fully.

See also: Unifying the Data Warehouse and Data Lake Creates a New Analytical Rhythm

Key Findings from the Survey

The survey’s scope was revealing. As it turns out, a lot is going on with AI implementation behind the scenes.

Strong Intent and Progress in AI Adoption

The study reveals a robust intent among enterprises to implement AI. An overwhelming majority of respondents (87%) express a “strong” or “very strong” desire to incorporate AI into their operations within the next 12 months. And this enthusiasm is reflected in the tangible progress being made. Another strong majority of organizations report significant strides in their AI initiatives.

Alignment of Data Management with AI Success

A clear correlation emerges between data management practices and successful AI implementation. Approximately 90% of respondents report that their data management strategies are either “somewhat aligned” or “very aligned” with their AI innovation goals. This alignment is critical for organizations aiming to maximize AI’s potential. It enables them to create new products, enhance operational efficiency, and generate deeper business insights.

Primary Challenges in AI Evolution

The survey identifies several key challenges that organizations face in their AI journey:

• Organizing Data for AI Use: Over half of the respondents struggle with organizing structured data for machine learning and unstructured data for retrieval-augmented generation (RAG).
• Barriers to High-Quality Data: The most significant barriers to accessing high-quality data for AI projects are data privacy and security concerns and the sheer volume of data.

Strategies for Improving Data Management

To address these challenges, organizations are employing a range of strategies, but three specific methodologies showed a strong following:

1. Adopting Agile Methodologies: A solid 61% of respondents use agile methods to manage data projects, enhancing flexibility and responsiveness to changing AI needs.
2. Leveraging Cloud-Based Platforms: Close behind, 59% are utilizing cloud platforms to scale AI initiatives, allowing for efficient management of large data volumes and ensuring data accessibility.
3. Implementing Data Governance and Federated Data Access: 52% have adopted data governance and federated data access strategies to enhance data security, privacy, and accessibility across multiple sources.

The Importance of Real-Time Data

Access to real-time data is highlighted as a critical factor for AI success. 62% of respondents identify it as an area requiring the most attention. Ensuring data accuracy, consistency, and availability in real time is vital for the reliability and performance of AI models.

Emerging Trends and Recommendations

Looking ahead, the study suggests several trends and strategies for aligning data management with AI initiatives:

• Addressing Ethical Concerns and Leveraging Federated Learning: As AI adoption grows, ethical considerations in data use become increasingly important. Federated learning, which allows AI to train on decentralized data while maintaining privacy, is emerging as a critical trend.
• Building a Data-Driven Culture: Enhanced data literacy is essential for maximizing AI’s impact. 90% of respondents believe that greater data literacy would have at least a moderate impact on the success of AI projects. 40% expect a significant impact.

What This Means for Businesses Moving Forward

The findings from this study indicate that making AI progress will require businesses to rethink their approach to data management. Moving forward, organizations should:

• As real-time data becomes more critical, companies must invest in technologies and processes that ensure high-quality and readily available data.
• To stay competitive, businesses should adopt data management frameworks that can adapt to evolving AI technologies and methodologies.
• Building a culture that values data literacy and cross-functional collaboration will be key to maximizing the impact of AI. This involves increasing awareness of data’s strategic importance and encouraging collaboration across different parts of the organization.
• As AI becomes more central to business strategy, addressing ethical concerns and ensuring responsible data use will be crucial. Strategies like federated learning, which supports privacy while enabling AI training on decentralized data, will likely gain prominence.

“If there’s one takeaway from this research,” Adrian Estala, VP, Field Chief Data Officer for Starburst notes, “it’s that 63% of respondents lack the confidence in their data strategy needed to fully execute on their AI initiatives.”

“For years, organizations have been aware of the importance of solid data management, but many have delayed addressing it. Now, with AI adoption surging, the urgency is real. However, Open Hybrid Lakehouse data architectures make it easier than ever to manage,” he says. 

“The next generation of leaders will stand out by how effectively, ethically, and securely they harness enterprise data to power AI. Success won’t come from the AI model itself—plenty of powerful models exist—but from unlocking enterprise data with the right business context.”

Prioritizing Real-Time Data Access and Quality

The report underscores the critical role of robust data management in achieving AI success. Organizations must prioritize real-time data access, ensure data security and privacy, and foster a data-driven culture to leverage AI’s potential fully. By adopting scalable architectures and enhancing data quality, businesses can position themselves for long-term success in an increasingly data-centric world.

For enterprises looking to thrive in the AI era, the key will be developing a comprehensive data management strategy that supports real-time data access, addresses privacy concerns, and integrates agile and federated data approaches. Locking down an effective data management strategy could be the key to true AI progress.