HashiConf Last chance to register for our cloud conference October 10-12. Register today
Vault
Vault Home

AWS secrets engine

The AWS secrets engine generates AWS access credentials dynamically based on IAM policies. This generally makes working with AWS IAM easier, since it does not involve clicking in the web UI. Additionally, the process is codified and mapped to internal auth methods (such as LDAP). The AWS IAM credentials are time-based and are automatically revoked when the Vault lease expires.

Vault supports three different types of credentials to retrieve from AWS:

  1. iam_user: Vault will create an IAM user for each lease, attach the managed and inline IAM policies as specified in the role to the user, and if a permissions boundary is specified on the role, the permissions boundary will also be attached. Vault will then generate an access key and secret key for the IAM user and return them to the caller. IAM users have no session tokens and so no session token will be returned. Vault will delete the IAM user upon reaching the TTL expiration.
  2. assumed_role: Vault will call sts:AssumeRole and return the access key, secret key, and session token to the caller.
  3. federation_token: Vault will call sts:GetFederationToken passing in the supplied AWS policy document and return the access key, secret key, and session token to the caller.

Static roles

The AWS secrets engine supports the concept of "static roles", which are a 1-to-1 mapping of Vault Roles to IAM users. The current password for the user is stored and automatically rotated by Vault on a configurable period of time. This is in contrast to dynamic secrets, where a unique username and password pair are generated with each credential request. When credentials are requested for the Role, Vault returns the current Access Key ID and Secret Access Key for the configured user, allowing anyone with the proper Vault policies to have access to the IAM credentials.

Please see the API documentation for details on this feature.

Setup

Most secrets engines must be configured in advance before they can perform their functions. These steps are usually completed by an operator or configuration management tool.

  1. Enable the AWS secrets engine:

    $ vault secrets enable aws
Success! Enabled the aws secrets engine at: aws/

    By default, the secrets engine will mount at the name of the engine. To enable the secrets engine at a different path, use the -path argument.

  2. Configure the credentials that Vault uses to communicate with AWS to generate the IAM credentials:

    $ vault write aws/config/root \
    access_key=AKIAJWVN5Z4FOFT7NLNA \
    secret_key=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 \
    region=us-east-1

    Internally, Vault will connect to AWS using these credentials. As such, these credentials must be a superset of any policies which might be granted on IAM credentials. Since Vault uses the official AWS SDK, it will use the specified credentials. You can also specify the credentials via the standard AWS environment credentials, shared file credentials, or IAM role/ECS task credentials. (Note that you can't authorize vault with IAM role credentials if you plan on using STS Federation Tokens, since the temporary security credentials associated with the role are not authorized to use GetFederationToken.)

    Notice: Even though the path above is aws/config/root, do not use your AWS root account credentials. Instead generate a dedicated user or role.

  3. Configure a Vault role that maps to a set of permissions in AWS as well as an AWS credential type. When users generate credentials, they are generated against this role. An example:

    $ vault write aws/roles/my-role \
    credential_type=iam_user \
    policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:*",
      "Resource": "*"
    }
  ]
}
EOF

    This creates a role named "my-role". When users generate credentials against this role, Vault will create an IAM user and attach the specified policy document to the IAM user. Vault will then create an access key and secret key for the IAM user and return these credentials. You supply a user inline policy and/or provide references to an existing AWS policy's full ARN and/or a list of IAM groups:

    $ vault write aws/roles/my-other-role \
    policy_arns=arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess,arn:aws:iam::aws:policy/IAMReadOnlyAccess \
    iam_groups=group1,group2 \
    credential_type=iam_user \
    policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:*",
      "Resource": "*"
    }
  ]
}
EOF

    For more information on IAM policies, please see the AWS IAM policy documentation.

Usage

After the secrets engine is configured and a user/machine has a Vault token with the proper permission, it can generate credentials.

  1. Generate a new credential by reading from the /creds endpoint with the name of the role:

    $ vault read aws/creds/my-role
Key                Value
---                -----
lease_id           aws/creds/my-role/f3e92392-7d9c-09c8-c921-575d62fe80d8
lease_duration     768h
lease_renewable    true
access_key         AKIAIOWQXTLW36DV7IEA
secret_key         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
security_token     <nil>

    Each invocation of the command will generate a new credential.

    Unfortunately, IAM credentials are eventually consistent with respect to other Amazon services. If you are planning on using these credential in a pipeline, you may need to add a delay of 5-10 seconds (or more) after fetching credentials before they can be used successfully.

    If you want to be able to use credentials without the wait, consider using the STS method of fetching keys. IAM credentials supported by an STS token are available for use as soon as they are generated.

  2. Rotate the credentials that Vault uses to communicate with AWS:

    $ vault write -f aws/config/rotate-root
Key           Value
---           -----
access_key    AKIA3ALIVABCDG5XC8H4

    Note: Due to AWS eventual consistency, after calling the aws/config/rotate-root endpoint, subsequent calls from Vault to AWS may fail for a few seconds until AWS becomes consistent again. See the AWS secrets engine API for further information on config/rotate-root functionality.

Example IAM policy for Vault

The aws/config/root credentials need permission to manage dynamic IAM users. Here is an example AWS IAM policy that grants the most commonly required permissions Vault needs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iam:AttachUserPolicy",
        "iam:CreateAccessKey",
        "iam:CreateUser",
        "iam:DeleteAccessKey",
        "iam:DeleteUser",
        "iam:DeleteUserPolicy",
        "iam:DetachUserPolicy",
        "iam:GetUser",
        "iam:ListAccessKeys",
        "iam:ListAttachedUserPolicies",
        "iam:ListGroupsForUser",
        "iam:ListUserPolicies",
        "iam:PutUserPolicy",
        "iam:AddUserToGroup",
        "iam:RemoveUserFromGroup"
      ],
      "Resource": ["arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:user/vault-*"]
    }
  ]
}

Vault also supports AWS Permissions Boundaries when creating IAM users. If you wish to enforce that Vault always attaches a permissions boundary to an IAM user, you can use a policy like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iam:CreateAccessKey",
        "iam:DeleteAccessKey",
        "iam:DeleteUser",
        "iam:ListAccessKeys",
        "iam:ListAttachedUserPolicies",
        "iam:ListGroupsForUser",
        "iam:ListUserPolicies",
        "iam:AddUserToGroup",
        "iam:RemoveUserFromGroup"
      ],
      "Resource": ["arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:user/vault-*"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:AttachUserPolicy",
        "iam:CreateUser",
        "iam:DeleteUserPolicy",
        "iam:DetachUserPolicy",
        "iam:PutUserPolicy"
      ],
      "Resource": ["arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:user/vault-*"],
      "Condition": {
        "StringEquals": {
          "iam:PermissionsBoundary": [
            "arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:policy/PolicyName"
          ]
        }
      }
    }
  ]
}

where the "iam:PermissionsBoundary" condition contains the list of permissions boundary policies that you wish to ensure that Vault uses. This policy will ensure that Vault uses one of the permissions boundaries specified (not all of them).

STS credentials

The above demonstrated usage with iam_user credential types. As mentioned, Vault also supports assumed_role and federation_token credential types.

STS federation tokens

Notice: Due to limitations in AWS, in order to use the federation_token credential type, Vault must be configured with IAM user credentials. AWS does not allow temporary credentials (such as those from an IAM instance profile) to be used.

An STS federation token inherits a set of permissions that are the combination (intersection) of four sets of permissions:

  1. The permissions granted to the aws/config/root credentials
  2. The user inline policy configured in the Vault role
  3. The managed policy ARNs configured in the Vault role
  4. An implicit deny policy on IAM or STS operations.

Roles with a credential_type of federation_token can specify one or more of the policy_document, policy_arns, and iam_groups parameters in the Vault role.

The aws/config/root credentials require IAM permissions for sts:GetFederationToken and the permissions to delegate to the STS federation token. For example, this policy on the aws/config/root credentials would allow creation of an STS federated token with delegated ec2:* permissions (or any subset of ec2:* permissions):

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": [
      "ec2:*",
      "sts:GetFederationToken"
    ],
    "Resource": "*"
  }
}

An ec2_admin role would then assign an inline policy with the same ec2:* permissions.

$ vault write aws/roles/ec2_admin \
    credential_type=federation_token \
    policy_document=@policy.json

The policy.json file would contain an inline policy with similar permissions, less the sts:GetFederationToken permission. (We could grant sts:GetFederationToken permissions, but STS attaches attach an implicit deny that overrides the allow.)

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": "ec2:*",
    "Resource": "*"
  }
}

To generate a new set of STS federation token credentials, we simply write to the role using the aws/sts endpoint:

$ vault write aws/sts/ec2_admin ttl=60m
Key             Value
lease_id        aws/sts/ec2_admin/31d771a6-fb39-f46b-fdc5-945109106422
lease_duration  60m0s
lease_renewable false
access_key      ASIAJYYYY2AA5K4WIXXX
secret_key      HSs0DYYYYYY9W81DXtI0K7X84H+OVZXK5BXXXX
security_token  AQoDYXdzEEwasAKwQyZUtZaCjVNDiXXXXXXXXgUgBBVUUbSyujLjsw6jYzboOQ89vUVIehUw/9MreAifXFmfdbjTr3g6zc0me9M+dB95DyhetFItX5QThw0lEsVQWSiIeIotGmg7mjT1//e7CJc4LpxbW707loFX1TYD1ilNnblEsIBKGlRNXZ+QJdguY4VkzXxv2urxIH0Sl14xtqsRPboV7eYruSEZlAuP3FLmqFbmA0AFPCT37cLf/vUHinSbvw49C4c9WQLH7CeFPhDub7/rub/QU/lCjjJ43IqIRo9jYgcEvvdRkQSt70zO8moGCc7pFvmL7XGhISegQpEzudErTE/PdhjlGpAKGR3d5qKrHpPYK/k480wk1Ai/t1dTa/8/3jUYTUeIkaJpNBnupQt7qoaXXXXXXXXXX

STS AssumeRole

The assumed_role credential type is typically used for cross-account authentication or single sign-on (SSO) scenarios. In order to use an assumed_role credential type, you must configure outside of Vault:

  1. An IAM role
  2. IAM inline policies and/or managed policies attached to the IAM role
  3. IAM trust policy attached to the IAM role to grant privileges for Vault to assume the role

assumed_role credentials offer a few benefits over federation_token:

  1. Assumed roles can invoke IAM and STS operations, if granted by the role's IAM policies.
  2. Assumed roles support cross-account authentication
  3. Temporary credentials (such as those granted by running Vault on an EC2 instance in an IAM instance profile) can retrieve assumed_role credentials (but cannot retrieve federation_token credentials).

The aws/config/root credentials must be allowed sts:AssumeRole through one of two methods:

  1. The credentials have an IAM policy attached to them against the target role:

    {
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "Resource": "arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:role/RoleNameToAssume"
  }
}

  2. A trust policy is attached to the target IAM role for the principal:

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:user/VAULT-AWS-ROOT-CONFIG-USER-NAME"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

When specifying a Vault role with a credential_type of assumed_role, you can specify more than one IAM role ARN. If you do so, Vault clients can select which role ARN they would like to assume when retrieving credentials from that role.

Further, you can specify both a policy_document and policy_arns parameters; if specified, each acts as a filter on the IAM permissions granted to the assumed role. If iam_groups is specified, the inline and attached policies for each IAM group will be added to the policy_document and policy_arns parameters, respectively, when calling sts:AssumeRole. For an action to be allowed, it must be permitted by both the IAM policy on the AWS role that is assumed, the policy_document specified on the Vault role (if specified), and the managed policies specified by the policy_arns parameter. (The policy_document parameter is passed in as the Policy parameter to the sts:AssumeRole API call, while the policy_arns parameter is passed in as the PolicyArns parameter to the same call.)

Note: When multiple role_arns are specified, clients requesting credentials can specify any of the role ARNs that are defined on the Vault role in order to retrieve credentials. However, when policy_document, policy_arns, or iam_groups are specified, that will apply to ALL role credentials retrieved from AWS.

Let's create a "deploy" policy using the arn of our role to assume:

$ vault write aws/roles/deploy \
    role_arns=arn:aws:iam::ACCOUNT-ID-WITHOUT-HYPHENS:role/RoleNameToAssume \
    credential_type=assumed_role

To generate a new set of STS assumed role credentials, we again write to the role using the aws/sts endpoint:

$ vault write aws/sts/deploy ttl=60m
Key             Value
lease_id        aws/sts/deploy/31d771a6-fb39-f46b-fdc5-945109106422
lease_duration  60m0s
lease_renewable false
access_key      ASIAJYYYY2AA5K4WIXXX
secret_key      HSs0DYYYYYY9W81DXtI0K7X84H+OVZXK5BXXXX
security_token  AQoDYXdzEEwasAKwQyZUtZaCjVNDiXXXXXXXXgUgBBVUUbSyujLjsw6jYzboOQ89vUVIehUw/9MreAifXFmfdbjTr3g6zc0me9M+dB95DyhetFItX5QThw0lEsVQWSiIeIotGmg7mjT1//e7CJc4LpxbW707loFX1TYD1ilNnblEsIBKGlRNXZ+QJdguY4VkzXxv2urxIH0Sl14xtqsRPboV7eYruSEZlAuP3FLmqFbmA0AFPCT37cLf/vUHinSbvw49C4c9WQLH7CeFPhDub7/rub/QU/lCjjJ43IqIRo9jYgcEvvdRkQSt70zO8moGCc7pFvmL7XGhISegQpEzudErTE/PdhjlGpAKGR3d5qKrHpPYK/k480wk1Ai/t1dTa/8/3jUYTUeIkaJpNBnupQt7qoaXXXXXXXXXX

Troubleshooting

Dynamic IAM user errors

If you get an error message similar to either of the following, the root credentials that you wrote to aws/config/root have insufficient privilege:

$ vault read aws/creds/deploy
* Error creating IAM user: User: arn:aws:iam::000000000000:user/hashicorp is not authorized to perform: iam:CreateUser on resource: arn:aws:iam::000000000000:user/vault-root-1432735386-4059

$ vault revoke aws/creds/deploy/774cfb27-c22d-6e78-0077-254879d1af3c
Revoke error: Error making API request.

URL: POST http://127.0.0.1:8200/v1/sys/revoke/aws/creds/deploy/774cfb27-c22d-6e78-0077-254879d1af3c
Code: 400. Errors:

* invalid request

If you get stuck at any time, simply run vault path-help aws or with a subpath for interactive help output.

STS federated token errors

Vault generates STS tokens using the IAM credentials passed to aws/config.

Those credentials must have two properties:

  • They must have permissions to call sts:GetFederationToken.
  • The capabilities of those credentials have to be at least as permissive as those requested by policies attached to the STS creds.

If either of those conditions are not met, a "403 not-authorized" error will be returned.

See http://docs.aws.amazon.com/STS/latest/APIReference/API_GetFederationToken.html for more details.

Vault 0.5.1 or later is recommended when using STS tokens to avoid validation errors for exceeding the AWS limit of 32 characters on STS token names.

AWS instance metadata timeouts

Affects Vault 1.4 and later

Anytime Vault uses the instance metadata service on an EC2 instance, such as for getting credentials from the instance profile, there may be a delay with the introduction of v2 of the instance metadata service (IMDSv2). The AWS SDK used by Vault first attempts to connect to IMDSv2, and if that times out, it falls back to v1. In Vault 1.4, this timeout can take up to 2 minutes. In Vault 1.5.5 and later, it can take up to 2 seconds with this fix: #10133.

The timeout occurs in situations where there is a proxy between Vault and IMDSv2, and the instance hop limit is set to less than the number of "hops" between Vault and IMDSv2. For example, if Vault is running in docker on an EC2 instance with the instance hop limit set to 1, the AWS SDK client will attempt to connect to IMDSv2, timeout, and fall back to IMDSv1 because of the extra network hop between docker and IMDS.

To avoid the timeout behavior, the hop limit may be adjusted on the underlying EC2 instances. With the docker example, setting the hop limit to 2 will allow the AWS SDK in Vault to connect to IMDSv2 without delay.

API

The AWS secrets engine has a full HTTP API. Please see the AWS secrets engine API for more details.

Edit this page on GitHub