Dr. Werner Vogels, Amazon.com VP and CTO, used his AWS re:Invent keynote speech to talk about the benefits of building asynchronous, loosely coupled systems and how event-driven architecture enables them.
“What I’ve observed is that the real world is asynchronous,” said Vogels. “There are so many events happening around you, trillions of events all the time.”
Unfortunately, the way applications and systems have developed for years does not lend itself to handling the asynchronous nature of the world. “If you look at your distributed systems and the way that we build our systems, synchrony leads to tightly coupled systems, said Vogels.
With such systems, businesses cannot make any changes. He cited an example of an online shopping cart. “If anything fails, probably the overall system will fail,” he noted. One way to avoid this is to move to an asynchronous system that is driven by an event broker. Such a system decouples everything.
“There is no tight coupling between any of the components. And so, the shopping cart will post an event, the order system will pick it up, and then the payment system will pick up the work after that,” he said.
The most important thing for such a system is that there is an architecture that can evolve very easily without having to change any of the other components. If a business wants to add something like a reporting or an invoicing service, it can do that without changing the overall system.
See also: AWS Eyes a Zero-ETL Future with Newly Announced Capabilities
Advantages of loosely coupled systems
Dr. Vogels noted that businesses want loosely coupled systems because they are natural. And they have a number of advantages. Because such systems have fewer dependencies, a business can change different components without having to change the others.
One advantage of this approach is that it is a natural way of isolating failures. If any components fail, the whole system continues to work. And if the new version of, say a reporting service comes back up, it just replaces the events that it has missed and executes. “This is what’s called an evolvable architecture,” said Vogels.
Evolvable is the key. A business does not have to build an entire system in one shot. It can start off with a smaller system that can do the work and then evolve it into the eventual complex system the business had in mind.
“So, the great thing about the loosely coupled systems is it’s easy to evolve your architecture,” he said. “Look at S3, in 2006 when we launched it, it consisted of eight separate microservices. Now, these are well over 255 different microservices that we could add to the system without taking the system down, without any impact on the other pieces.”
“We have new storage methodologies, we added new policy mechanisms, we added different storage tiers, all of that while the system was running,” he said. So, evolvability is extremely important when you think about designing your systems.”
Decomposing into small, well-understood building blocks is the fundamental part of building these asynchronous, loosely coupled, event-driven systems.
He noted that Amazon realized the necessity for adopting this approach from experience. Amazon, the retailer, started off as a monolith, and one that had reached the end of life because it couldn’t scale anymore. It couldn’t evolve the system.
“When we wanted to actually innovate more, we couldn’t do that because we couldn’t evolve the system,” he said. “So, we moved over to a service-oriented architecture with core services. And we went through a microservices architecture.” Now Amazon (the retailer) is basically a very large collection of microservices running over a shared infrastructure.
At this point in his talk, Dr. Vogels suggested everyone read the Distributed Computing Manifesto that he recently added to his blog All Things Distributed. The manifesto is a canonical document from the early days of Amazon that transformed the architecture of Amazon’s eCommerce platform. It highlights the challenges it was facing at the end of the 20th century and hinted at where it was headed.
“I urge you to read this document. It’s dated. I know. But many of you will recognize things in that document because it’s a reality for many of you still now that have to work in monoliths,” he said. In addition to a discussion about a move to service orientation, there is much discussion on how some of the work is workflow oriented.
Workflows are important because they enable Amazon to build applications from loosely coupled components. “In AWS, we have two services that basically provide these basic concepts for you,” he said. There’s Step Functions on one end that handles all the different types of execution and EventBridge, which is a serverless event bus that ingests data from apps, SaaS apps, and AWS services and routes that data to targets.
See also: Cloud Services for Event-Driven Architectures (EDAs)
Customer feedback taken to heart
He notes that one theme that keeps emerging is that businesses should be using serverless components to combine and stitch together components into a larger system.
“We’ve been looking at our customers that are building these workflows. And as always, customers start doing things that you never expected them to do,” he said. “It’s always important to listen and learn from what our customers are doing and then try to improve our systems.”
For example, he noted that there was a particular pattern that Amazon hadn’t anticipated when building Step Functions, and it’s that customers actually would like to do Map Reduce with their Step Functions. “I go, like, what? After all, we have Kafka and EMR,” he said. “But the customer is saying, no, no, no, no, no, no, no. We want something way simpler. I just want to write two Lambda functions, and that’s it.”
One example he cited was from National Oceanic and Atmospheric Administration (NOAA). The organization takes measurements of weather from data stations around the world and then, on a daily basis, publishes it in a public dataset. And the dataset is about 37 gigabytes and can be spread out over 500,000 files.
“Imagine you just want to get the average high temperature for a particular month from all those files,” he said. “It’s a very simple operation. They just wanted to have two simple functions, two simple Lambda functions to do this.”
“So, I’m happy to announce today that we’re now giving fundamental support for this pattern with AWS Step Functions Distributed Map. What we can do with these Step Functions is basically do a map and reduce step very easily. It starts off with using the Step Functions Distributed Map, which then will fire up maybe a thousand instances of your Lambda functions to process the data and then summarize that data in the final Lambda step. This allows you very easily to process very large amounts of data using very simple Lambda functions.”
This was just one of the many issues covered in this keynote address. You can view the entire speech here. And here is a summary of some of the major new offerings Dr. Vogels announced in his talk:
Developer Tools
- Amazon CodeCatalyst (preview), a Unified Software Development Service: Amazon CodeCatalyst is a unified software development and delivery service that enables software development teams to plan, develop, collaborate on, build, and deliver applications on AWS, reducing friction throughout the development lifecycle quickly and easily.
Serverless
- Step Functions Distributed Map – A serverless solution for large-scale parallel data processing: AWS announced the availability of a distributed map for AWS Step Functions. This flow extends support for orchestrating large-scale parallel workloads such as the on-demand processing of semi-structured data.
- The ability to create point-to-point integrations between event producers and consumers with Amazon EventBridge Pipes: Amazon EventBridge Pipes is a new feature of Amazon EventBridge that makes it easier to build event-driven applications by providing a simple, consistent, and cost-effective way to create point-to-point integrations between event producers and consumers, removing the need to write undifferentiated glue code.
- AWS Application Composer (Preview): AWS Application Composer helps developers simplify and accelerate architecting, configuring, and building serverless applications. Developers can drag, drop, and connect AWS services into an application architecture by using AWS Application Composer’s browser-based visual canvas. AWS Application Composer helps developers focus on building apps by maintaining deployment-ready infrastructure as code definitions, complete with integration configuration for each service.
Salvatore Salamone is a physicist by training who has been writing about science and information technology for more than 30 years. During that time, he has been a senior or executive editor at many industry-leading publications including High Technology, Network World, Byte Magazine, Data Communications, LAN Times, InternetWeek, Bio-IT World, and Lightwave, The Journal of Fiber Optics. He also is the author of three business technology books.
