After several years of working with AWS, IAM remains one of the most frequently used services in my daily routine. Yet, despite my familiarity with it, a recent production incident taught me that there’s always more to learn.

AWS bucket policies play a crucial role in restricting access to our buckets. They allow us to specify who can read from and write to them, particularly when dealing with cross-account permissions. Let’s delve into an example of such a bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::***:role/my-app-role"
            },
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
            ],
            "Resource": "arn:aws:s3:::stav-bucket/*"
        }
    ]
}

Recently, I needed to replace one of my roles — the same role that possessed read/write permissions for several of my buckets. Initially, I assumed that as long as I retained the role name, everything would remain unaffected. In other words, the role’s ARN within my bucket permissions would remain consistent, thus preserving the permissions even after the role replacement.

However, instead of business as usual, I found myself inundated with “Access Denied” errors from my application, unable to perform operations on the bucket. After inspecting the role (which seemed fine), I pinpointed the issue to the bucket policy. Indeed, something had changed:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "AROAQV5PXLVLG47PAPA3S"
            },
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
            ],
            "Resource": "arn:aws:s3:::stav-bucket/*"
        }
    ]
}

The role ARN had been replaced with a cryptic string, leaving me wondering about its origin and significance. Further investigation revealed that this string was, in fact, the unique ID of my role — a string generated by AWS upon role creation.

When saving a policy that references a role’s ARN, AWS internally transforms the ARN into its unique principal ID. While you typically encounter the ARN in the console, AWS internally maps it to its corresponding ID. However, when the mapping breaks — such as when the original role is deleted — the original ARN, represented by its unique ID, resurfaces. Consequently, the role loses its permissions over the bucket.

Even if I were to recreate the “my-app-role” role, its unique ID would differ, and the role would remain unauthorized until I adjusted the bucket policy accordingly.

Reflecting on this incident, I realized its security implications. This mechanism minimizes the risk of privilege escalation through role deletion and recreation. Even if someone were to delete or replace my role, they wouldn’t gain access to my other AWS resources governed by their policies. Each new role iteration comes with a new unique ID, ensuring that my resources do not recognize it as the same role.

While this discovery initially confounded me, I now appreciate the security measures it underscores.

Note: While this blog post focuses on roles, it’s worth noting that AWS users undergo similar processes, generating unique IDs when utilized. Additionally, this phenomenon extends beyond bucket policies to other AWS resource policies, such as trust relationship policies within roles.