As more organizations are surprised by their cloud expenses, they seek to reign in utilization, but they want to do this wisely without limiting essential access to data. Data observability software provides a broad and deep view of data from its starting point in a data pipeline through its deployment and utilization as a data product.
At the 2023 Gartner Data & Analytics Summit, Cloud Data Insights (CDI) had the chance to ask Rohit Choudhary, the CEO of Acceldata.io, to share his market insights gathered from customers’ priorities. He painted a clear picture of the drivers behind data observability and managing the cloud data footprint with an eye to cost optimization.
(The interview has been revised for clarity and readability. See Rohit Choudharys bio at the end of this article.)
CDI: How have your customers’ needs or attitudes changed given current economic conditions and post-pandemic digital transformation?
Choudhary: One of the silver linings in the current economic environment is the continued enterprise investment in data and analytics products. We expect this to continue for the rest of 2023. Early in January 2023, Gartner forecasted that worldwide IT spending will grow 2.4% in 2023, with a clear indication that enterprise IT spending remains strong. The software segment will be the fastest growing segment in 2023, with 9.3% YOY growth and spend hitting over $850 billion.
Our customers, who are data leaders and represent data teams at Global 2000 enterprises, continue to invest more in new innovative solutions such as data observability. We’re also seeing a shift towards a more balanced and more pragmatic approach to these new investments. A few common trends include:
- Tool consolidation and more interest in an integrated approach
- Focus on better performance
- Looking for value and strong ROI
- Hybrid deployment architecture
Within our prospect base, we continue to see data quality-related challenges, an inability to address hybrid cloud observability use cases, the lack of technical talent, and the inability to use legacy tools.
Given our customers’ goals, we are helping them fully understand the value of our platform with an ROI analysis for their use cases. And we have adopted a consultative partnership approach that starts with understanding customer pain points, product direction, and their desired business value before discussing solutions.
We certainly monitor market trends closely. Our key to success has been focusing on delivering the right business outcomes associated with these trends. They also help us time the execution of a multi-year strategy for our unified data observability platform that eliminates the need for siloed tools.
CDI: Is there one recent trend that stands out in terms of its impact on customers?
Choudhary: Cloud cost spend and optimization is a major challenge with cloud platforms today. The continued growth in cloud-based data platforms is resulting in modernization initiatives and is invariably leading to overspending and data-quality challenges.
We believe Acceldata is well-positioned to address these challenges with our unified data observability platform, which eliminates the scale challenge that many of our customers face.
CDI: The direct connection between data observability and cost optimization might not be as obvious as types of observability. Could you explain how they relate to each other?
Choudhary: The backdrop is that enterprises are collecting more data than ever before. And system complexity has increased way beyond anything that people can actually manage to deal with. Along with that, there is not enough talent. If you look at the fastest-growing job in North America, it’s either data scientist or data engineering. That means enterprises have less ability to administer systems manually. It also means that if there is a mistake made, that mistake is going to be very expensive.
Consumption-based pricing gives you flexibility but not the ability to control your speed. So if there is user behavior, which is detrimental to your budget, then you will discover it the harsh way. A customer came and told us that one SQL query costs them $275,000. The numbers are staggering because every expensive mistake can literally cause a bank run on your enterprise budgets. Another issue behind costs is that as you’re trying to harmonize all the data that you brought into the company and then activate it for operations, marketing, or advertising, that’s actually causing more and more processing–more stress on your systems. Every time that you process and the results are not reliable, you have to reprocess, which at least doubles the costs for the same amount of work. So you have to monitor and put guardrails around user behavior to control costs.
See also: Using Data Observability to Control Costs
CDI: Some of what you describe might fall under infrastructure or application observability. How do you disambiguate these from data observability?
Choudhary: Application observability is when you’re trying to look for trends in user experience–did the screen load up in 30 seconds? In the case of data observability, you’re monitoring the supply chain of data, which contains data that has originated in multiple different systems, including transaction data from applications, third-party data sources, which the enterprise goes and buys, and engagement systems where people are providing data. Now that has to be processed and transformed. It has to be made human-readable, and then it gets ready for consumption. Data observability has a complete focus on the state of the system, as opposed to the click of a button on the desktop or a touch on a mobile device. In addition to the data, you’re observing data products and outcomes.
Observing is essential. It’s a foundation, but customers need a lot more such as ways to quickly identify problems and address them. Over a period of time, data observability will become crucial for all kinds of data management.
CDI: Mitigating the rising cost of cloud has grown to be a main theme at the 2023 Gartner Data & Analytics Summit. How are costs affecting your customers?
Choudhary: In the last year or so, people are just getting surprised by the amount of money they pay for low latency, high-volume cloud, or high-volume data. Customers believe that they should be able to run an open source stack on an environment of their choosing, whether it is on-premises, private cloud, or public cloud, but it all depends on the business model.
See also: The Need for Data Observability in Today’s Cloud-Oriented Architectures
All the consumption use cases are headed toward the cloud, where the cost of a high volume of transactions can affect business fundamentals. An ad network, for example, might make only 20 cents on the dollar, so the infrastructure cost has to be 10% of that because it really affects your gross margin. And when your gross margins get affected, your share price gets affected, and your CEO or CFO level conversation is about how much money can be spent.
AI brings up another example. Every company is going to try to become an AI company, but if you introduce AI into your product, you potentially may not remain a SaaS company because the gross margins will be completely eroded by the cost of cloud infrastructure.
Now the total data supply chain is directed toward data and analytics. But in the next five years, 10% will be for machine learning and AI. But things have started progressing more quickly, so this might happen sooner.
CDI: What’s Acceldata’s area of focus for the coming year?
Choudhary: Well, we are a very, very IP- and engineering-centric company. One of the unique differentiators of our company is that we keep looking at these problems as though they were our own. It’s the bane of an observability company to integrate [with many platforms and tools], but it’s also basically engineers in a candy store.
The biggest initiatives that we ran in 2022 were essentially in scaling the data engine and reliability. The big change on the data reliability side, where reliability is giving quality a run for its money, is the way that structured data has completely disappeared, and analytics has to work with unstructured data. And if you look at the data supply chain, you know more data is coming.
See also: Data Observability’s Role in Ensuring Data Reliability
One of our customers presented at our sales kickoff. They said that if you model data as hostile and go with that assumption after you run the entire process, it almost seems like the whole industry is ready for disruption.
Bio: Rohit Choudhary is the CEO and Co-Founder of Acceldata, a San Jose-based startup that has developed a multidimensional Data Observability Cloud to help enterprises observe and optimize modern data systems and maximize return on data investment. Prior to Acceldata, Choudhary served as Director of Engineering at Hortonworks, where he led the development of Dataplane Services, Ambari, and Zeppelin, among other products. While at Hortonworks, Rohit was inspired to start Acceldata after repeatedly witnessing his customers’ multi-million dollar data initiatives fail despite employing the latest data technologies and experienced teams of data experts.
Rohit previously founded Appsterix, which was acquired by 24(7) Labs. He served as an engineering leader at 24(7) after Appsterix’s acquisition and also spent time managing engineering teams at Inmobi. Choudhary specializes in developing and scaling products and building and managing high-performance teams. He is based in Silicon Valley and earned his Bachelor of Engineering from SJCE in Mysore, India.
Elisabeth Strenger is a Senior Technology Writer at CDInsights.ai.
