CloudTrail evolution

At the end of 2013 (yes, 5 years ago already), AWS announced a new service, CloudTrail. It claimed to provide increased visibility into user activity for demonstrating compliance; it never claimed to be an audit log, but is as close as can be architected and not be in the critical path for API request execution.

From the start, CloudTrail supported immediate cross-account delivery of logs. These logs were thus untouched by the generating account — there was no user-deployed replication of data files between AWS Accounts, and thus no question of the user origin Account editing the CloudTrail logs before a separate security team got access.

You can see why this was a massive success. Oh, and the first trail per region was complementary, except for the traffic charges incurred to copy the logs from the originating Region to the destination, and for the actual storage of the logs after delivery (but the customer may chose a retention policy — see S3 Lifecycle Policies).

I wanted to demonstrate the impact of the design of CloudTrail has had in innovating over the last half a decade…

The Early Years

Initially CloudTrail would itself execute from an AWS Service team’s own AWS account — one per Region. When provisioning an S3 Bucket for receiving these logs, a customer would have to find the authoritative list of Account IDs to whitelist for S3:PutObject.

Each time a new AWS Region was being launched (eg, Stockholm), a new Account ID for CloudTrail’s service account would have to be discovered, and added to the destination S3 Bucket Policy.

Furthermore, CloudTrail was initially a per-Region service, so customers would have to scurry over every AWS account and define a CloudTrail trail in the newly launched Region; until this did this, the new region was effectively a blind spot for governance and compliance that read the logs.

So you can see in the above, with three accounts, and three Regions; we had to define CloudTrail 9 times! Lets look now at today’s 20 regions, and a customer with 20 AWS Accounts, and we’re having to defined CloudTrail 400 times. Luckily, there’s an API, and CloudFormation support…

The Middle Years: Per-Account simplification

These first two problems (Service Account IDs for the S3 Bucket Policy, and new-Region blind spot) were solved in time. I and many others contributed to the product feature request and Support Requests feedback to help shape this: the CloudTrail account identity was solved with IAM Service Principals, and as such, the service name “cloudtrail.amazon.com” now matches the CloudTrail service in every current and future (non-Chinese, non-US-GovCloud) commercial Region.

This bucket policy now looks like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AWSCloudTrailAclCheck20150319",
            "Effect": "Allow",
            "Principal": {"Service": "cloudtrail.amazonaws.com"},
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::myBucketName"
        },
        {   "Sid": "AWSCloudTrailWrite20150319",
            "Effect": "Allow",
            "Principal": {"Service": "cloudtrail.amazonaws.com"},
            "Action": "s3:PutObject",
            "Resource": [ "arn:aws:s3:::myBucketName/AWSLogs/myAccountID1/*", "arn:aws:s3:::myBucketName/AWSLogs/myAccountID2/*", "arn:aws:s3:::myBucketName/AWSLogs/myAccountID3/*" ],
            "Condition": {"StringEquals": {"s3:x-amz-acl": "bucket-owner-full-control"}}
        }
     ]
}

Note: we still whitelist per accounts, so that other AWS customers couldn’t send their CloudTrail logs to our bucket — which would be weird, and potentially just incur storage charges to us.

And for the new-Region problem: Global Cloud Trail Trails were introduced, that would have a Home Region the Global Trail was defined in, but collect activity from ever other Region.Further improvements came in the way of cryptographically signed Digest files that would form a chain of files each of which contained information about the files that came before, providing an unbreakable chain of history such that the modification of a CloudTrail data file, or modification or removal of a previous digest file could be detected.

Now its a little more manageable; a customer with 20 AWS Accounts need only defined CloudTrail 20 times. With APIs and CloudFormation, there is a chance of consistency.

More and more services became capable of generating CloudTrail logs over the years, and the JSON logging format had a few modifications, which was beautifully handled by the version log entry format (currently has log entry versions up to 1.06).

And now with an Organisation approach

AWS Organisations is starting to build out more of a corporate approach to the decade long multi-account approach. A somewhat clunky Landing Zone service tried to make this a little more turn-key, but AWS Organisations is now starting to deliver on simplification.

With a Verified master account (which historically was your Consolidated Billing Account), you can now push a master CloudTrail definition to all subsidiary accounts. This is done once, and all subscribed account configure this trail. Furthermore, those subsidiary accounts cannot remove the enforced Organisation Tail while a part of the organisation.

Thus consistency is ensured, and a security team no longer has to scan these workload accounts to ensure that they are still logging CloudTrail, and logging it to the correct enterprise destination(s).

Thus combining this with cross-account logging, we end up with something looking a little like this:

Warning: Log Prefix has changed

However, the Prefix that CloudTrail logs to in this configuration has changed ever so slightly. Where previously we whitelisted individual accounts in the Resources part of our S3 bucket, we now have to whitelist a Prefix that includs our Organisation ID — but we don’t need to be worried about other (non-organisation-members) sending logs to us:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AWSCloudTrailAclCheck20150319",
            "Effect": "Allow",
            "Principal": {"Service": "cloudtrail.amazonaws.com"},
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::myBucketName"
        },
        {
            "Sid": "AWSCloudTrailWrite20150319",
            "Effect": "Allow",
            "Principal": {"Service": "cloudtrail.amazonaws.com"},
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::myBucketName/AWSLogs/o-1234567/*", 
            "Condition": {"StringEquals": {"s3:x-amz-acl": "bucket-owner-full-control"}}
        }
     ] 
}

What remains to be tested by my (at least) is creating a new account and seeing it get an Organisation Trail pushed to it upon create; adding an already-created account and seeing the same; and lastly removing an account from an organisation and seeing it not be able to log any more.

In Summary

Hats off to the IAM, CloudTrail, Organisations teams for making this all come together (as well as other service teams who have managed to get CloudTrail support into their products.

Customers will want to move to this, but three may be adjustment required for any analytics solutions reading the CloudTrail files due to the new location after implementing this change. Customers may chose to leave their existing CloudTrail in place for a few days after deploying an Organisation Trail in order to update those analytics services.

CloudTrail continues to be at the heart of Governance and Compliance assurance when running on AWS.

There’s much more to talk about here for the configuration of the destination S3 Bucket. If you would like to spend a deep dive with me, check out our in-person Advanced Security & Operations on AWS course. If you’d like this delivered in your city, please get in touch with us at Nephology.