How Understanding Our Kubernetes Cloud Costs Enabled Us to Scale

At Ogury, we offer a mobile-optimized advertising engine that enables our clients like VISA, Ford, McDonald’s, and others to connect with their target audiences at scale while preserving individuals’ security and privacy. Our AWS Cloud resources run APIs that harness the contextual, semantic, and unique mobile audience data (cookie-less and ID-less for privacy) we require to deliver our clients the capabilities they need to deliver relevant ad content. Our engine has utilized AWS since our launch in 2014, and we now run Kubernetes on EC2 instances after replacing an in-house container orchestration system. This is how we implemented cost visibility after migrating to Kubernetes and enabled successful scale.

See also: Managing Database Access for Kubernetes Workloads

Migrating to Kubernetes and recognizing the cost visibility challenge

While enabling some of the biggest global brands to reach target audiences at scale led to our own rapid growth, we were also quickly confronted with scalability issues. Necessitating a more robust approach to container orchestration, we began a two-year process of migrating to Kubernetes. 

In planning out and shaping these Kubernetes deployments, we found that open source projects offered ideal tools for our needs. For example, we liked open source Prometheus for its monitoring and metrics collection capabilities. We implemented open source Thanos to add high availability to Prometheus, as well as long-term data storage and metrics queries. When it came to tracking, attributing, and optimizing Kubernetes costs, we also experimented with an in-house implementation of an open source Grafana dashboard. Our goal here was to capture Prometheus metrics to achieve visibility into API call resource usage, costs, and opportunities for improvement. In practice, however, maintaining this dashboard brought in too much complexity. It left our team searching for a more efficient, user-friendly, and accurate approach.

Kubernetes costs became a particularly high-profile internal issue following our migration because the container orchestration platform had become the single-largest cost center under our technical team’s control. We were also lacking visibility into the precise sources of those container-based cloud costs, so controlling them wasn’t inherently easy. Recognizing the need to act sooner than later as our growth continued, our finance and technical team leaders worked together to make Kubernetes cost visibility a top priority—and to seek out the right strategy to get it under control. 

Implementing granular Kubernetes cost visibility and allocations

Our search began by vetting a number of large, broad cloud cost visibility options. However, we realized that the broadness (and the big price tags, with some even asking for a percentage of our total cloud spending) weren’t going to give us what we were looking for. This vetting process helped us to hone in on the exact needs of our use case: making clear sense of our Kubernetes costs so we could confidently optimize our environments. We then found the right Kubernetes-specific tool in open source Kubecost, and used the tool’s enterprise support to rapidly implement and integrate cost visibility into our Kubernetes deployments.

Having that Kubernetes cost visibility quickly transformed our technical team’s ability to understand where budget was going. The team can now drill down into spending data to view the granular cloud costs associated with each Kubernetes workload, each category, and even each service. Our cloud usage data is also integrated with our specific AWS cost structure, meaning that expenses listed in our cost management reports aren’t just general estimates, but highly accurate to our true costs. Our team frequently makes use of AWS spot instances with variable costs, increasing the importance of this direct pricing accuracy. 

With this strategy now in place, our team has gained a considerably more complete understanding of how individual Kubernetes services behave from a resource usage and cost standpoint, and our team can rapidly address Kubernetes workloads exhibiting anomalous and wasteful activity. Our team can now produce specific cost calculations on demand, and easily share Kubernetes cost visualization data and reports with the finance team.

Achieving decisive strategic advantages via accurate Kubernetes cost data

Granular Kubernetes cost visibility down to the namespace, service, and any other desired level of scrutiny has allowed our team to enact newfound opportunities for cost optimization. We now command a fully integrated and optimized open source stack, and plan to add anomaly detection and alerts to flag and address any cost overruns or surprise increases in expenses.

Most importantly, full cost visibility and robust spending controls have equipped our business to rapidly expand and scale our Kubernetes deployments without breaking the bank. In the last three years we’ve achieved an 8x increase in the number of API calls our Kubernetes environments handle (now totaling 10 billion+ each day). Our scale—and aggressive plans to continue our Kubernetes-backed rapid growth trajectory—would not be possible without first making our costs predictable, optimized, and under control.