Last month a member of the A Cloud Guru developer community posted a despondent message on the service’s Discord channel. They had been using a tutorial to build a machine learning model on the ‘free’ tier of Amazon Web Services (AWS), which allows limited use of apps on the cloud platform without incurring charges. The student thought they had deleted the project and stayed within the limit, and was shocked to discover they had run up a bill of $200.
“I have been charged $200 and I can’t even find what [I need] to stop,” they wrote. “I’m 20 and really don’t have $200 for [AWS]… I’m just gonna cry here for the rest of the day.”
This shouldn’t be possible, period. There needs to be an easy, safe way for students to explore AWS without incurring unexpected megacharges. pic.twitter.com/QHcsK4SH49
— Forrest Brazeal (@forrestbrazeal) May 4, 2021
The incident caused an outcry in the AWS user community and Amazon swiftly intervened and refunded the student. But such mishaps can happen to even the most experienced of engineers; in March AWS chief evangelist Jeff Barr took to Twitter to highlight a surprise bill he’d received after setting up a database on the wrong AWS account.
Building out cloud capabilities is one of the top priorities for tech leaders according to the recent Tech Monitor Technology Leaders Agenda, and AWS, as the market leader, will be hoping to collect a lot of this planned investment.
But these incidents highlight that, for organisations large and small, knowing what you're deploying on AWS – and how to switch it off – is crucial to ensure you don't get any unwanted surprises, and staying abreast of the situation is not always as easy as it seems. What's more, problems of usability and opaque pricing of AWS could become a barrier to the development of the next generation of cloud engineers, experts told Tech Monitor.
Why do people encounter AWS pricing problems?
Forrest Brazeal is a cloud architect and AWS expert who is part of A Cloud Guru. "I personally believe that AWS has the most confusing billing of any cloud provider, and the lowest amount of flexibility to control your spend," Brazeal told Tech Monitor.
The issue derives in part from AWS's legacy system, Brazeal argues. "AWS simply has the longest history and the most technical debt," he explains. "Remember, they were inventing a lot of the cloud paradigm as they went along. I know some are quick to accuse AWS of creating an intentionally predatory 'free tier', but I don’t get the sense from having been in the community a long time that that was anyone’s intention."
Nonetheless, AWS's various services often operate slightly differently, which makes it difficult to gain visibility of all workloads that are active on a single account. Bills, therefore, have to be calculated by amalgamating information from a lot of different sources and take a while to process, which isn't good news if a company has something running of which they not aware.
"Billing is a slow background process, it's not in real time," says Jon Bryant, lead financial solutions architect at Strategic Blue, a company that advises businesses on cloud deployments. "So if you're talking about accidentally turning on a massive instance and letting it run, or having your account hacked and someone misusing it, in six or 12 hours, before the next billing run decides to come through, you can run up an awful lot of cost."
For users of the free tier, the situation is further complicated because spending limits are not turned on by default, meaning once the limit of the 'free' service is reached costs start to pile up. It is thought the student who ended up with the $200 bill did not correctly switch off an instance they had set up during their tutorial.
This lack of controls marks AWS out from its two main competitions, Microsoft Azure and Google Cloud Platform (GCP). "Google has a hard stop, you can have an account without a credit card attached," says Bryant. "This means you can only spend within the free tier, and Azure has something similar. With AWS it's remarkably easy to think you've turned everything off, and the first time you discover something is still running is when you get a credit card bill because you've reached the end of the free tier. It is quite hard to spot every single thing that's running and work out how to turn it off."
The complexity of the AWS pricing system could deter newbies
Brazeal says technical choices made in the early days of AWS mean it favours corporate clients. "AWS optimised for the needs of its business customers – bias for uptime, worry about the bill later – and those choices simply aren’t conducive to the experience of individual cloud learners," he says.
But with cloud skills now in high demand, this poor accessibility for novices could hinder the development of the tech workforce. Cloud engineers are the second most sought-after discipline according to those polled for the Technology Leaders Agenda, and also the most difficult to find.
Amazon itself has recognised this, and says it plans to offer free cloud training to 29m people worldwide over the next four years, but Brazeal says until these people "have a safe way to learn", this promise "isn’t merely unachievable; it’s irresponsible." He adds: "This is a growing cohort that AWS must now serve in order to maintain its market-leading position in cloud."
What is AWS doing about billing?
Though problems with billing crop up fairly regularly, AWS apparently has no plans to change the set-up of its free tier. A spokesman contacted by Tech Monitor declined to comment on whether the company is considering updates to how the free tier works, instead directing users to a set of blogs explaining the system.
Hybrid cloud expert Chris Short was stung by a surprise $2,700 AWS bill last year owing to an error in his website set-up, and was eventually refunded the bulk of the money. He says the size of AWS means that on a technical level it's not feasible to introduce real-time billing, which would make it easier to identify problems as they're happening. "There’s absolutely nothing AWS can do to prevent this from happening, they’ve told as such," he says. "To get real-time price reporting requires engineering beyond their capabilities. That’s right, beyond AWS’s capabilities. It’s easier to refund customers than to get real-time price reporting going."
Brazeal says putting in place default usage limits, akin to those found on GCP and Azure, would be a welcome step. "GCP and Azure aren't perfect, [but] AWS doesn’t really have a concept of 'spending limits' for anyone," he says. "And for their business customers, who can negotiate multi-year discounts upfront, that’s fine. It’s just not a model designed with people’s personal credit cards in mind."
How to protect your business from unexpected AWS bills
There are functions within AWS that can be enabled to control costs. "AWS provides a budget alerting process that does both forward forecasting and actuals, and you can apply it on an account-by-account basis," says Strategic Blue's Bryant. "We would recommend switching this on as it's not on by default. AWS does have a script you can run so that, rather than just getting email alerts, processes are automatically shut down."
Alternatives to the free tier are also available for beginners, such as free credits for start-ups and those affiliated with educational institutions through AWS educate. "The best advice is simply not to attach personal risk to the use of AWS, Brazeal adds. "If you are a student at an affiliated institution, you can get no-credit-card access to some AWS services through AWS Educate. You need to do what you can to protect yourself."
