How to configure a production-grade AWS account structure using Gruntwork AWS Landing Zone

You use separate AWS accounts to isolate different environments from each other and to limit the "blast radius" when things go wrong. For example, putting your staging and production environments in separate AWS accounts ensures that if an attacker manages to break into staging, they still have no access whatsoever to production. Likewise, this isolation ensures a developer making changes in staging is less likely to accidentally break something in production.

If you configure your AWS account structure correctly, you’ll be able to manage all user accounts in one place, making it easier to enforce password policies, multi-factor authentication, key rotation, and other security requirements. Using multiple AWS accounts also makes it easier to have fine-grained control over what permissions each developer gets in each environment.

A properly configured AWS account structure will allow you to maintain an audit trail of all the changes happening in all your environments, check if you’re adhering to compliance requirements, and detect anomalies. Moreover, you’ll be able to have consolidated billing, with all the charges for all of your AWS accounts in one place, including cost breakdowns by account, service, tag, etc.

An overview of the core concepts you need to understand to set up an AWS account structure, including AWS Organizations, IAM Users, IAM Roles, IAM Groups, CloudTrail, and more.

An overview of how to configure a secure, scalable, highly available AWS account structure that you can rely on in production. To get a sense of what production-grade means, check out The production-grade infrastructure checklist.

A step-by-step guide to configuring a production-grade AWS account structure using the Gruntwork AWS Landing Zone solution, including how to manage it all with customizable security baselines defined in Terraform.

What to do once you’ve got your AWS account structure configured.

The email address you provide when creating a new AWS account becomes the user name of your root user. This email address must be unique across ALL AWS accounts globally, so you can’t use the same email address to create multiple AWS accounts.

When creating a new AWS account, you will create a console password that, along with the root user’s user name, you can use to login to the AWS console.

After the initial sign up, if you wish to login as the root user, you have to go to https://console.aws.amazon.com and login using the root user’s email address and password.

The root user can optionally have a set of access keys, which are the credentials you use to login to your AWS account programmatically (e.g., on the command line or when making API calls). Access keys consist of two parts: an access key ID (for example, AKIAIOSFODNN7EXAMPLE) and a secret access key (for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY).

You can enable Multi-Factor Authentication (MFA) for the root user (strongly recommended), which will require you to provide not only the user name and password when logging in, but also a temporary, one-time token generated by either a virtual or physical MFA device (e.g., the Google Authenticator app, RSA key fob, or a YubiKey). This adds a strong second layer of security for your root user, as logging in now requires both something you know (the user name and password) and something you have (the virtual or physical MFA device). Note that, by default, if you enable MFA for a root user, the MFA token will only be required when logging in with the user name and console password in your web browser; you will NOT be required to provide an MFA token when logging in programmatically with access keys. If you want to require MFA tokens for programmatic access too (strongly recommended), you will need to use IAM policies, which are described later.

The root user has access permissions to everything in your AWS account. By design, there’s almost no way to limit those permissions. This is similar in concept to the root or administrator user of an operating system. If your root user account gets compromised, the attacker will likely be able to take over everything in your account. Therefore, you typically only use the root user during initial setup to create IAM users (the topic of the next section) with more limited permissions, and then you’ll likely never touch the root user account again.

Every IAM user in your AWS account must have a unique user name.

Each IAM user can optionally have a console password. The user name and console password allows you to login as an IAM user to your AWS account in a web browser by using the IAM user sign-in URL.

Every AWS account has a unique IAM user sign-in URL. Note that to login as an IAM user, you do NOT go to https://console.aws.amazon.com, as that’s solely the sign-in URL for root users. Instead, IAM users will need to use a sign-in URL of the form https://<ID_OR_ALIAS>.signin.aws.amazon.com/console, where ID_OR_ALIAS is either your AWS account ID (e.g., https://111122223333.signin.aws.amazon.com/console) or a custom account alias that you pick for your AWS account (e.g., https://my-custom-alias.signin.aws.amazon.com/console). Whenever you create a new IAM user, make sure to send that IAM user their user name, console password, and the IAM user sign-in URL.

Each IAM user can optionally have a set of access keys, which are the credentials you use to login to your AWS account programmatically (e.g., on the command line or when making API calls). Access keys consist of two parts: an access key ID (for example, AKIAIOSFODNN7EXAMPLE) and a secret access key (for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY).

Each IAM user can enable Multi-Factor Authentication (MFA) (strongly recommended), which will require you to provide not only the user name and console password when logging in, but also a temporary, one-time token generated by either a virtual or physical MFA device (e.g., the Google Authenticator app, RSA key fob, or a YubiKey). This adds a strong second layer of security for your IAM user, as logging in now requires both something you know (the user name and password) and something you have (the virtual or physical MFA device).

You can configure a password policy in your AWS account to enforce requirements on console passwords, such as minimum length, use of special characters, and password expiration.

By default, a new IAM user does not have permissions to do anything in the AWS account (principle of least privilege). In order to grant this user permissions, you will need to use IAM policies, which are the topic of the next section.

Each IAM policy is a JSON document that consists of one or more statements, where each statement can allow or deny specific principals (e.g., IAM users) to perform specific actions (e.g., ec2:StartInstances, s3:GetObject) on specific resources (e.g., EC2 instances, S3 buckets). Here’s an example IAM policy that allows an IAM user named Bob to perform s3:GetObject on an S3 bucket called examplebucket:

{
  "Version":"2012-10-17",
  "Statement": [
    {
      "Effect":"Allow",
      "Principal": {"AWS": ["arn:aws:iam::111122223333:user/Bob"]},
      "Action":["s3:GetObject"],
      "Resource":"arn:aws:s3:::examplebucket/*"
    }
  ]
}

Each AWS account comes with a number of managed policies, which are pre-defined IAM policies created and maintained by AWS. These included policies such as AdministratorAccess (full access to everything in an AWS account), ReadOnlyAccess (read-only access to everything in an AWS account), AmazonEC2ReadOnlyAccess (read-only access to solely EC2 resources in an AWS account), and many others. AWS managed policies are owned by AWS and cannot be modified or removed.

While managed policies give you coarse-grained, generic permissions, to get more fine-grained, custom permissions, you can create custom IAM policies (known as customer-managed policies).

A standalone policy is an IAM policy that exists by itself and can be attached to other IAM entities. For example, you could create a single policy that gives access to a specific S3 bucket and attach that policy to several IAM users so they all get the same permissions.

An inline policy is a policy that’s embedded within an IAM entity, and only affects that single entity. For example, you could create a policy embedded within an IAM user that gives solely that one user access to a specific S3 bucket.

Whereas an IAM user allows a human being to access AWS resources, one of the most common use cases for an IAM role is to allow a service—e.g., one of your applications, a CI server, or an AWS service—to access specific resources in your AWS account. For example, you could create an IAM role that gives access to a specific S3 bucket and allow that role to be assumed by one of your EC2 instances. The code running on that EC2 instance will then be able to access that S3 bucket without you having to manually copy AWS credentials (i.e., access keys) onto that instance.

Another common use case for IAM roles is to grant an IAM entity in one AWS account access to specific resources in another AWS account. For example, if you have an IAM user in account A, then by default, that IAM user cannot access anything in account B. However, you could create an IAM role in account B that gives access to a specific S3 bucket in account B and allow that role to be assumed by an IAM user in account A. That IAM user will then be able to access the contents of the S3 bucket by assuming the IAM role in account B. This ability to assume IAM roles across different AWS accounts is the critical glue that truly makes a multi AWS account structure possible.

Just as you can attach IAM policies to an IAM user and IAM group, you can attach IAM policies to an IAM role.

You must define a trust policy for each IAM role, which is a JSON document (very similar to an IAM policy) that specifies who can assume this IAM role. For example, here is a trust policy that allows this IAM role to be assumed by an IAM user named Bob in AWS account 111122223333:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Principal": {"AWS": "arn:aws:iam::111122223333:user/Bob"}
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::444455556666:role/foo"
    }
  ]
}

IAM roles do not have a user name, password, or permanent access keys. To use an IAM role, you must assume it by making an AssumeRole API call (see the AssumeRole API and assume-role CLI command), which will return temporary access keys you can use in follow-up API calls to authenticate as the IAM role. The temporary access keys will be valid for 1-12 hours, depending on IAM role settings, after which you must call AssumeRole again to fetch new keys. Note that to make the AssumeRole API call, you must first authenticate to AWS using some other mechanism. For example, for an IAM user to assume an IAM role, the workflow looks like this:

Figure 1. The process for assuming an IAM role

Most AWS services have native support built-in for assuming IAM roles. For example, you can associate an IAM role directly with an EC2 instance, and that instance will automatically assume the IAM role every few hours, making the temporary credentials available in EC2 instance metadata. Just about every AWS CLI and SDK tool knows how to read and periodically update temporary credentials from EC2 instance metadata, so in practice, as soon as you attach an IAM role to an EC2 instance, any code running on that EC2 instance can automatically make API calls on behalf of that IAM role, with whatever permissions are attached to that role. This allows you to give code on your EC2 instances IAM permissions without having to manually figure out how to copy credentials (access keys) onto that instance. The same strategy works with many other AWS services: e.g., you use IAM roles as a secure way to give your Lambda functions, ECS services, Step Functions, and many other AWS services permissions to access specific resources in your AWS account.

AWS Single Sign-On is a managed service that allows you to configure SSO for IdPs that support SAML, such as Active Directory and Google. It provides a simple SSO experience for the AWS web console, although signing in on the command line requires multiple steps, including manually copy/pasting credentials.

Gruntwork Houston allows you to configure SSO for IdPs that support SAML or OAuth, including Active Directory, Google, Okta, GitHub, and others. It provides a simple SSO experience for the AWS web console, command-line access, VPN access, and SSH access. Houston is currently in private beta, so if you’re interested, please email us to find out how to get access.

The first AWS account you create is the root account (sometimes also called the master account). This will be the parent account for your organization. This account has powerful permissions over all child accounts, so you should strictly limit access to this account to a small number of trusted admins.

You can use AWS Organizations to create one or more child accounts beneath the root account.

You can group child accounts into one or more organization units. This gives you a logical way to group accounts: for example, if your company has multiple business units, then each business unit could be represented by one organization unit, and each organization unit can contain multiple child accounts that can be accessed solely by members of that business unit.

All of the billing from the child accounts rolls up to the root account. This allows you to manage all payment details in a single account and to get a breakdown of cost by organization unit, child account, service type, etc.

When creating a child account, you can configure AWS Organizations to create an IAM role within that account that allow users from the root account to access the child account. This allows you to manage the child accounts from the parent account without having to create an IAM user in every single child account.

You can use Service control policies (SCPs) to define the maximum available permissions for a child account, overriding any permissions defined in the child account itself. For example, you could use SCPs to completely block a child account from using specific AWS regions (e.g., block all regions outside of Europe) or AWS services (e.g., Redshift or Amazon Elasticsearch), perhaps because those regions or services do not meet your company’s compliance requirements (e.g., PCI, HIPAA, GDPR, etc).

You will want a single security account for managing authentication and authorization. This account is not used to run any infrastructure. Instead, this is where you define all of the IAM users and IAM groups for your team (unless you’re using Federated auth, as described later). None of the other child accounts will have IAM users; instead, those accounts will have IAM roles that can be assumed from the security account. That way, each person on your team will have a single IAM user and a single set of credentials in the security account (with the exception of the small number of admins who will also have a separate IAM user in the root account) and they will be able to access the other accounts by assuming IAM roles.

You can have one or more application accounts for running your software. At a bare minimum, most companies will have a production account ("prod"), for running user-facing software, and a staging account ("stage") which is a replica of production (albeit with smaller or fewer servers to save money) used for internal testing. Some teams will have more pre-prod environments (e.g., dev, qa, uat) and some may find the need for more than one prod account (e.g., a separate account for backup and/or disaster recovery, or separate accounts to separate workloads with and without compliance requirements).

The shared-services account is used for infrastructure and data that is shared amongst all the application accounts, such as CI servers and artifact repositories. For example, in your shared-services account, you might use ECR to store Docker images and Jenkins to deploy those Docker images to dev, stage, and prod. Since the shared-services account may provide resources to (e.g., application packages) and has access to most of your other accounts (e.g., for deployments), including production, from a security perspective, you should treat it as a production account, and use at least the same level of precaution when locking everything down.

You may want to have one or more sandbox accounts that developers can use for manual testing. The application accounts (e.g., dev and stage) are usually shared by the whole company, so these sandbox accounts are intentionally kept separate so that developers can feel comfortable deploying and undeploying anything they want without fear of affecting someone else (in fact, the gold standard is one sandbox account per developer to keep things 100% isolated).

One other type of account that often comes in handy is a testing account that is used specifically for automated tests that spin up and tear down lots of AWS infrastructure. For example, at Gruntwork, we use Terratest to test all of our infrastructure code, and when testing something like our Vault modules, we end up spinning up and tearing down a dozen Vault and Consul clusters after every single commit. You don’t want all this infrastructure churn in your application or sandbox accounts, so we recommend having a separate AWS account dedicated for automated tests.

You will want a single logs account for aggregating log data. All the other accounts—root, security, application accounts, shared-services, etc.—will send their AWS Config and CloudTrail data to this account so that you have a single, central place where all logs are stored and can be viewed. This account will also contain a KMS customer master key (CMK) that is used to encrypt CloudTrail logs.

When creating a new child account using AWS Organizations, this is a role you create automatically that allows the admin users in the root account to have admin access to the new child account. This role is useful for initial setup of the new child account (e.g., to create other roles in the account) and as a backup in case you somehow lose access to the child account (e.g., someone accidentally deletes the other IAM roles in the account). Note that the name of this role is configurable, though we generally recommend sticking to a known default such as OrganizationAccountAccessRole.

This IAM role grants full access to everything in the child account. These are essentially admin permissions, so be very thoughtful about who has access to this IAM role.

This IAM role grants read-only access to everything in the child account.

This IAM role grants "developer" access in the child account. The exact permissions your developers need depends completely on the use case and the account: e.g., in pre-prod environments, you might give developers full access to EC2, ELB, and RDS resources, whereas in prod, you might limit that solely to EC2 resources. For larger teams, you will likely have multiple such roles, designing them for specific teams or tasks: e.g., allow-search-team-access-from-other-accounts, allow-frontend-team-access-from-other-accounts, allow-dba-access-from-other-accounts, etc.

This IAM group gives users full access to everything in the security account. It should only be used for a small number of trusted admins who need to manage the users and groups within this account.

These IAM groups are how you grant IAM users in the security account access to other child accounts. For each AWS account <ACCOUNT>, and each IAM role <ROLE> in that account, you have a group that grants sts:AssumeRole permissions for that role: e.g., users you add to the _account-dev-full-access group will get sts:AssumeRole permissions to the allow-full-access-from-other-accounts IAM role in the dev account (so they will have full access to that account) and users you add to the _account-prod-read-only group will get sts:AssumeRole permissions to the allow-read-only-access-from-other-accounts IAM role in the prod account (so they will have read-only access to that account).

These IAM groups don’t grant any IAM permissions, but instead are used by ssh-grunt to determine who is allowed to SSH to your EC2 instances. Each EC2 instance you launch can configure ssh-grunt with the names of the IAM group(s) that will be allowed to SSH to the instance, with or without sudo permissions. The group names are completely up to you, so you could have many such groups, with whatever names you pick. Once you add an IAM user to that group, that user will be able to SSH to the corresponding EC2 instances using their own IAM user name and the SSH key associated with their IAM user account.

All the IAM roles in your non-security child accounts that are meant to be assumed by users should require an MFA token in the trust policy. Since these IAM roles are the only way to access those child accounts (i.e., there are no IAM users in those child accounts), this ensures that it’s only possible to access those accounts with MFA enabled. Note: the OrganizationAccountAccessRole IAM role is created automatically by AWS Organizations, so you’ll need to manually update it in each child account to require MFA.

The only place you have IAM users and groups are in the root and security account. None of the user accounts should have any IAM policies directly attached, so the only thing to think through is the policies attached to the IAM groups. To enforce MFA, make sure that all of these policies require an MFA token. Note that all of these policies also should attach "self-management" permissions that allow IAM users just enough permissions to access their own user account without an MFA token so they can configure an MFA token in the first place.

This is an IAM role that grants permissions for automatically deploying (e.g., as part of a CI / CD pipeline) some specific service. For example, this role may have a trust policy that allows it to be assumed by a Jenkins server in the shared-services account, and gives that server permissions to deploy EC2 Instances and Auto Scaling Groups. Note that anyone who has to your CI server (e.g., anyone who can create/modify/execute Jenkins jobs) can effectively make use of all the permissions in this IAM role, so be very thoughtful about what this role can do.

This is an IAM role that grants permission to look up IAM group membership and the public SSH keys of IAM user accounts. Typically, you’d have this role in your security account to allow the EC2 instances in other accounts to authenticate SSH attempts using ssh-grunt.

Most EC2 instances, Lambda functions, and other AWS services you launch will have an IAM role that gives that service the permissions it needs to function. For example, the IAM role for the Consul cluster gives the EC2 instances in that cluster ec2:DescribeInstances, ec2:DescribeTags, and autoscaling:DescribeAutoScalingGroups permissions so that the instances can look up instance, tag, and auto scaling group information to automatically discover and connect to the other instances in the cluster.

The trust policy in service IAM roles cannot require MFA, as automated services can’t use MFA devices. That means you need to take extra care in terms of who can assume this IAM role, what permissions the role has, and locking down the services. For example, if you have Jenkins running on an EC2 instance, and you give that EC2 instance access to an IAM role so it can deploy your apps, you should do your best to minimize the permissions that IAM role has (e.g., to just ecs permissions if deploying to ECS) and you should ensure that your Jenkins instance runs in private subnets so that it is NOT accessible from the public Internet (see How to deploy a production-grade VPC on AWS).

The trust policy in service IAM roles will need to specify the appropriate Principal to allow an AWS service to assume it. For example, if you’re running Jenkins on an EC2 instance, and you want that EC2 instance to be able to assume an IAM role to get specific permissions (e.g., to get permissions to deploy some code in one of your child accounts), you’ll need a trust policy that looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Principal": {"Service": "ec2.amazonaws.com"}
    }
  ]
}

While IAM roles offer a convenient way to give an EC2 instance permissions to make API calls without having to manually copy credentials to the EC2 instance, the default security configuration for them is not particularly secure. That’s because the IAM role is exposed to the code on the EC2 instance through EC2 instance metadata, which is an http endpoint (http://169.254.169.254) that anyone on the EC2 instance can access. That means that any compromise of that EC2 instance instantly gives an attacker access to all the permissions in that IAM role. We strongly recommend mitigating this by limiting access to the endpoint solely to specific OS users (e.g., solely to the root user), e.g., by using iptables. You can do this automatically using ip-lockdown

# Make EC2 instance metadata only accessible to the root user
ip-lockdown "169.254.169.254" "root"

If you need to give something outside of your AWS account access to your AWS account—for example, if you’re using CircleCi as your CI server and need to give it a way to deploy code into your AWS accounts—then you will need to create a machine user. This is an IAM user designed for use solely by an automated service. You create the IAM user in the security account, add the user to specific IAM groups that grant the user the permissions it needs, generate access keys for the user, and provide those access keys to the external system (e.g., by storing the access keys as the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables in CircleCi). Note that you cannot require MFA for a machine user, so before giving credentials to an external system, think very carefully if that system is worth trusting with access to your AWS account, and limit the machine user’s permissions as much as possible.

Since all of your users will be managed in the IdP, you do not need to create any IAM users or IAM groups (other than the handful of IAM users in the root account).

With federated auth, you will be granting your IdP users access to specific IAM roles in specific accounts. Therefore, your child accounts will need more or less all the same basic IAM roles described earlier. However, the trust policy on those IAM roles will be quite different. For example, if you are using federated auth with SAML, the Action you allow will be sts:AssumeRoleWithSAML rather than sts:AssumeRole and the Principal will be your SAML provider:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRoleWithSAML",
      "Principal": {
        "Federated": "arn:aws:iam::111122223333:saml-provider/<YOUR_SAML_PROVIDER>"
      }
    }
  ]
}

One other big difference with IAM roles for federated auth is that these IAM roles should NOT require an MFA token. That’s because the MFA token check in AWS IAM policies only works with AWS MFA tokens, and not whatever MFA configuration you have with your IdP. With federated auth, AWS fully trusts the IdP to figure out all auth details, so if you want to require MFA, you need to do that in the IdP itself (i.e., in Google, Active Directory, or Okta).

This guide uses code from the Gruntwork Infrastructure as Code Library, as it implements most of the production-grade design for you out of the box. Make sure to read How to use the Gruntwork Infrastructure as Code Library.

This guide uses Terraform to define and manage all the infrastructure as code. If you’re not familiar with Terraform, check out A Comprehensive Guide to Terraform, A Crash Course on Terraform, and How to use the Gruntwork Infrastructure as Code Library.

This guide uses Terragrunt to configure the infrastructure as code. To get familiar with Terragrunt, explore the features, read the guides, or dive into the documentation.

You will need to initialize an infrastructure-live repository to contain all of the Terragrunt configuration code for your infrastructure. You may use the for-production example code to start with.

As part of this guide, you will create IAM users, including, optionally, credentials for those IAM users. If you choose to create credentials, those credentials will be encrypted with a PGP key. You could provide the PGP keys manually, but a more manageable option may be to have your team members to sign up for Keybase, create PGP keys for themselves, and then you can provide their Keybase usernames, and the PGP keys will be retrieved automatically.

Do NOT store the root user’s password, or secrets of any kind, in plain text. Instead, always use a secrets manager such as 1Password, LastPass, or pass to store the credentials in an encrypted format.

Do NOT re-use passwords from other websites, or any password that you can remember at all. Instead, generate a random, cryptographically secure, long password (20+ characters) for the root user. All the password managers mentioned above can generate and store passwords for you in one step, so use them!

Make sure to enable MFA for your root user. Feel free to use a virtual or hardware MFA device—whichever is easier or required by your company—as either one dramatically improves the security of your root user.

Make sure to delete the root user’s access keys, so that the only way to login as the root user is via the web console, where MFA is required.

In the next section, you will create an IAM user in the root account with admin permissions. Once you’ve created that IAM user, you should do everything as that IAM user, and more or less never touch the root user account again. The only time you’ll need it is for account recovery situations (e.g., you accidentally deleted the IAM user or lost your credentials) or for the small number of tasks that require root user credentials.

Do NOT store the credentials or any kind of secret in plain text. Instead, always use a secrets manager such as 1Password, LastPass, or pass to store the credentials in an encrypted format.

Do NOT re-use passwords from other websites, or any password that you can remember at all. Instead, generate a random, cryptographically secure, long password (20+ characters). All the password managers mentioned above can generate and store passwords for you in one step, so use them!

Always make sure to enable MFA for your IAM user. Feel free to use a virtual or hardware MFA device—whichever is easier or required by your company—as either one dramatically improves the security of your IAM user. Note that using SMS (text messages) for MFA is no longer recommended by NIST due to known vulnerabilities with the cellular system, so using a virtual or hardware MFA device is preferable; that said, MFA with SMS is still better than no MFA at all.