by James Huston
Right now, many companies are in the process of moving their applications and data to the cloud. In fact, the vast majority of startups have only ever existed in the cloud. If you’re only supporting a single application/data set, this can be a pretty simple infrastructure that consists of 1 or 2 AWS accounts supporting your development and production environments. At Red Ventures, we’ve chosen to go a different route.
We use multiple accounts, separating out our different internal applications and our partner-specific data into their own account structures. This gives us maximum flexibility for our engineers, allows us to move quickly, and helps simplify our security management.
On the security side, IAM works really well across one AWS account, but it doesn’t really let you dig down into specific permissions. You can’t really limit access to the VPC level. For example, allowing the creation of EC2 in just a particular VPC for a particular role is very complicated. As a result, I’ve seen a lot of teams end up with one giant account that’s shared all the way down the line. But that can become really dangerous, really fast.
If everything is bundled together, and everyone has access to everything – it’s really easy to create big problems up and down the line. Let’s say an engineer’s workflow is to start in development, but he or she can also go in and affect staging and production. You now have a high potential for these changes to affect other applications or create an extremely insecure environment for everyone in the blink of an eye.
It’s equally problematic if you go the other direction – no one has access. That would be hugely problematic for a company like ours where we pride ourselves on moving very very quickly.
The crux of the problem is this: your customers’ data really guides what we have to do. Big customers – the enterprise accounts we all want to work with – will always want you to store their data separately. If you’re sharing a single account for lots of sensitive data, that’s going to be a problem for you.
Here’s what you can do:
- Use a shared product that can avoid storing customer data. If you don’t store sensitive data, then you don’t have a problem. Most applications might not be able to do this, but make sure you do your due diligence and only store the minimum amount of data needed.
- Use a single system, like RDS, and create a separate database in that system for each customer. This is a pretty common way to do it, and it’s how we started out 15 years ago. However, you’ll almost definitely run into growing pains. Plus, it can become a security issue when you’re trying to control access across the server with a bunch of different users. (I know – it shouldn’t be hard. But it is.)
- Create a new instance for each customer. This works a little better because you’re not going to have multiple clients/apps connecting to a single instance. Additionally, at the VPC level, you can put each customer’s data into separate VPCs. Now you’re thinking, “This is great! Now you can’t even route between them (except in very controlled situations).”
- But here’s the approach I like: If you get to a point where you can actually take your customers’ data (and their part of the application) and move it to its own account, now you can manage all of the permissions – specific to the customer AND specific to the engineers working on it.
But it’s important to remember: A lot of apps aren’t built cloud-first or in a way where you can break them up and run app/data specific to a customer. Additionally, once you get up to about 20 or more accounts it becomes a lot to manage, so definitely give some thought as to whether or not this is something your team will able to work on. (Think automation and CloudFormation or Terraform.)
Watch James speak about cloud security and more at DevOps Days Charlotte.
Once you get past multiple instances, you don’t really gain a whole lot on stability whether you go to separate VPCs or separate accounts. So the big question – in terms of scaling stability – is this: should you run it all on one instance, or should you use multiple?
Take a look at Terraform or CloudFormation to program your infrastructure. If you’re doing the right stuff here, this is much less an issue because you can push it out across any environment you need to very easily.
How many people does it take to manage an infrastructure that looks like this? IAM can get very complicated if you’re trying to do it all on a single account. But what I recommend is taking these and making them small modules that you can then bounce out to all your accounts and manage in one central place. Now you’ve got lots of little things that are easy to reproduce and manage vs. one giant, ugly spider web that will likely get you into trouble at some point.
Here’s my take: Simple is better. Yes, I have a lot more things to manage, but they’re simple, easy things that are easy to reproduce. Plus, I feel comfortable handing them off to others who can then run with it – and that makes us faster and better as a team.
James Huston is a Director of Platform Engineering. Before joining Red Ventures, he did many amazing things to bridge the gap between engineering and ops. Outside of work, he is starting a bad habit of collecting cars, tinkering at home improvement, and, as our CEO says, collecting bad wines.