Center for Data Pipeline Automation

Why the need for so many data pipelines?

There is so much interest in data pipelines, in general, and automated data pipelines, in particular, today because businesses are going through a fundamental transformation. More data is always being generated upon which actions can be taken. And businesses want to take advantage of new sources of data all the time.

For example, financial services companies routinely use their own data to make informed decisions. But now, with the move to cloud-native apps, the use of APIs, and the advent of initiatives like Open Banking, developers can theoretically integrate financial data from multiple institutions within the same application or share financial data between applications.

There are similar examples in other industries. In healthcare, organizations routinely make a patient’s data available to multiple apps used throughout the organization. Typically, they must incorporate data from the patient’s history, treatments, records from the primary care physician and specialists, insurance providers, and more into any decision-making application or process. In most organizations, every department and every specialist will be using different applications that require the various datasets to be present and available in specific formats at specific times.

As every business today is data-driven, data pipelines are the backbones of modern operations. Data pipelines move data from point A to B. Along the way, they use techniques like ETL (Extract, Transform, Load), ELT (Extract, Load, Transform), and more to get the data in the right format for use by each application. A pipeline might also perform other duties like data quality checks.

Data pipeline complexities grow

Building end-to-end data pipelines manually was the norm for decades. It is no longer an option. It takes too long to build the pipelines, and the process does not scale. In modern businesses, both issues (too much time and inability to scale) are unacceptable.

IT staff, developers, and data engineers are overwhelmed with requests. And each project is customized. This comes at a time when it is difficult to attract and retain skilled workers. The skills problem is becoming more acute. Many people with the knowledge and skills are retiring. Others are opting to leave their jobs or enter new fields leading to what many call the great resignation. And younger tech staff is harder to retain due to the great demand for their talents.

That alone is a problem, but the situation is much worse due to the complexity as the number of data pipelines grows. Quite often, businesses incur great technical debt rushing pipelines into production. That diverts what staff they have away from new projects as they cater to issues that arise with existing pipelines.

In a talk with RTInsights, Sean Knapp, founder and CEO of Ascend.io, put the issue into perspective. He noted why the structure behind traditionally built pipelines doesn’t scale. The scaling here is not about the data’s volume, velocity, variety, or veracity. It’s the scale in complexity.

Things often break down because traditional pipelines are brittle. They generally are loosely connected systems that are generally wired up by humans based on some assumption of that data and the system that uses the data at the point in time when everything was wired together. It is like the way telephone operators literally plugged in different phone systems and tried to keep everybody connected. That worked in the early eras, but ultimately businesses needed to move to a far more scalable and automatable solution.

The issue is very similar to what happened with software development. There was a huge surge around the need for more software engineers to build more products that were more interdependent on each other. That drove new eras of innovation and evolution because the number of things that were being built and the number of things that those things depended upon grew exponentially. However, there was a polynomial expansion in complexity. The monolithic development methods of old had to be replaced with modern approaches based on cloud-native and DevOps principles.

The industry is now at the same type of cusp with respect to data pipelines. Businesses have access to more powerful technology, allowing data engineers to build pipelines faster than ever. But what happens is everybody’s building pipelines that are dependent on each other is the introduction of a network effect. The network effect from data pipelines without higher levels of automation is crippling. And so, the teams’ productivity asymptotically approaches zero with the addition of each incremental data pipeline.

Enter data pipeline automation

Businesses today face key challenges, and data pipeline automation can assist in addressing each one.

The first challenge is driving new engagement models and digital transformation. This is about new business opportunities, innovation, and driving new business offerings to address these opportunities. Frequently it involves new thinking around digital transformation and ecosystems. Unfortunately, many digital transformation projects fail due to poor data integration.

A second challenge is accelerating data availability while reducing costs. With the ever-increasing number of applications, microservices, cloud, and on-premises data sources, the number and need for data pipelines is increasing. Most businesses have trouble handling this increased need at speed while trying to keep costs under control.

Automated data pipelines can help in each of these areas. Automated data pipelines replace the bottleneck of manually coded data pipelines. Modern approaches empower teams to build and deploy their pipelines. Supporting various data pipeline automation methods that work seamlessly together helps businesses remove the inefficiency and cost of manually building data pipelines that often do not work together.

So, what capabilities should such a massively scalable data pipeline automation effort include? The best way to answer that is to look at the challenges that must be overcome. Data pipeline automation must deal with several issues, including:

• Data is “heavy”, meaning it is costly to move and even more costly to process
• Data in an enterprise has thousands of sources, each of which is well-defined
• Data-driven business outcomes are well understood but hard to achieve
• The space between sources and outcomes is chaotic and poorly understood

The automation capabilities needed to close these gaps can propel businesses forward with greater productivity and business confidence.

The key to successful data pipeline automation

Similar to the consolidation of tools in previous waves of automation, data pipeline automation replaces data stacks that have been assembled from multiple tools and platforms.

Previous approaches have hit one important barrier to data pipeline automation: the need to scale. It turns out that the key for a business to break through the scaling barrier is to utilize an immutable metadata model of every aspect of the pipelines and automate every operation with it. This can be done with unique digital fingerprints that map not just every snippet of data but the data engineers’ code as well. That is the approach Ascend.io takes.

Such an approach lets businesses program specifically for end-to-end data pipeline operations at a near-infinite scale. With the ability to track pipeline state in networks at a vast scale, businesses can always know the exact state of every node in every pipeline with certainty. It can constantly detect changes in data and code across the most complex data pipelines and respond to those changes in real time.

The fingerprint linkages ensure that all dependent pipelines maintain data integrity and availability for all data users. For data teams, scalable technology becomes a vehicle for managing organizational change.

Digging deeper into data pipeline automation

At the heart of data pipeline automation is the ability to propagate change through the entire network of code and data that make up a pipeline. A data pipeline automation solution should be instantly aware of any change in the code or the arriving data. It should then automatically propagate the change downstream on behalf of the developer so it is reflected everywhere.

As the network of pipelines increases, this capability alone will save highly skilled technologists days of mundane work assessing and managing even the simplest of changes.

When data pipelines are chained together, changes propagate automatically throughout the network. This technique eliminates redundant business logic and reduces processing costs for the whole system. When resources are limited, pipeline automation provides controls to prioritize the pipelines that matter most.

The approach also provides continuity through different types of failures. Automated retry heuristics ride through cloud, data cloud, and application failures to reduce human intervention and minimize downtime.

Benefits of using automated data pipelines

Today, no intelligent systems deliver data at the pace and with the impact leaders need to power the business. The processes to consume and transform data are ad-hoc and manual, and costly to support. As a result, stakeholders limit their reliance on data, making decisions based on gut instinct rather than facts.

To move away from this less-than-optimal approach, companies need to make fundamental changes to their data engineering efforts and start running at speed and with agility. Data engineering efforts are almost exclusively concerned with data pipelines, spanning ingestion, transformation, orchestration, and observation — all the way to data product delivery to the business tools and downstream applications.

Automated data pipelines ensure unified data ingestion, transformation, and orchestration. They can substantially simplify data engineering efforts. They allow data teams to focus on business value rather than fixing code, holding together a patchwork of point solutions. As a result, data pipeline automation has the power to meet business demands and make improvements to an organization’s productivity and capabilities.

Like automation efforts of the past, such as those based on RPA, data pipeline automation can deliver significant benefits to a business. They include:

• Accelerate engineering velocity: When the team is no longer worrying about debugging vast libraries of code or tracing data lineage through obscure system logs, the speed of delivery increases exponentially. Engineers also gain the capacity to shift into higher-order thinking to solve data problems in conjunction with business stakeholders.
• Ease the hiring crunch: Enabled by a comprehensive set of data automation capabilities, companies no longer need to hire hard-to-find esoteric skill sets. Anyone familiar with SQL or Python can design, build, and troubleshoot data pipelines, making data far more approachable and making data engineering teams more affordable and nimble.
• Cost reduction in data tools: When data automation is purchased as an end-to-end platform, data engineering teams can reduce software costs from dozens of point solutions. They also realize dramatic savings in engineering time as engineers focus on creating data pipelines rather than maintaining an in-house platform.

Bottom line: Businesses simply need many data pipelines. Doing things manually and relying solely on a centralized set of highly skilled experts is not an option. While data engineers will always be needed to build complex pipelines, successful businesses rely on automated data pipelines to eliminate this bottleneck. That enables businesses to take full advantage of their data resources to make more informed decisions, improve customer engagements, and increase operational efficiencies.