Customization and Automation in AWS Audit Manager

Introduction

AWS announced the Audit Manager service at re:Invent 2020 to provide customers with an easier way to gather the evidence required when assessing the risk and compliance of AWS environments across various standards and regulations, including PCI, HIPAA, SOC 2, and eleven others at initial release. Typically, to prove compliance with these frameworks, evidence must be created and provided to auditors showing how individual controls are satisfied. Oftentimes this means spending days or weeks taking screenshots, downloading logs, and parsing through mountains of information, or in more advanced cases, creating homegrown scripts to gather the required information. Audit Manager is AWS’s proposed solution to that problem. It removes some of the legwork previously necessary by identifying and continuously collecting control evidence from various data sources across AWS automatically.

Before continuing, it is important to note that Audit Manager only assists in the gathering and preparatory review of evidence in an audit. It does not assess or guarantee compliance in any way.

Audit Manager Basics

When discussing Audit Manager there are five primary components to be aware of:

1. Controls describe how to implement a specific requirement. For example, PCI DSS requirement 1.3.2 states that you must limit inbound internet traffic to IP addresses within a DMZ. The control for the requirement describes what this means exactly and what must be done to satisfy the requirement. Audit Manager allows you to use pre-packaged controls, referred to as Standard controls, that are associated with the supported Frameworks, or to create your own Custom controls.

2. Evidence consists of a snapshot in time record that can be used to show compliance with a control. This could include compliance check results, user activity logs, or resource configuration snapshots. Evidence is maintained within Audit Manager as immutable references of snapshots in time. Each piece can be reviewed, commented on, approved, and added to assessment reports for an auditor as part of a compliance workflow.

3. Frameworks are collections of controls that align with compliance and industry standards. Some example frameworks supported at release are PCI DSS, HIPAA, SOC 2, GDPR, and FedRAMP.

4. Assessments are a specific evaluation of controls in a specified framework. It is important to note that assessments are AWS region-specific. This allows you to run different assessments in regions that may have different regulatory requirements, for example, the US or the EU. Unfortunately, this also can cause some heartburn as well when you need to run the same assessment in multiple regions, which we will discuss later.

5. Assessment Reports are generated from individual assessments and contain assessment evaluation results and evidence that can be shared with auditors as evidence of compliance.

How does it work?

Behind the curtains, Audit Manager ties each defined control to one or more AWS Config Rules or Configuration Items, AWS CloudTrail events, or AWS Security Hub findings. These are referred to as data sources. Audit Manager queries these various sources on timelines defined within each control to generate and store evidence specific to that control’s requirements. For example, to collect evidence that CloudTrail is enabled, Audit Manager will query AWS Security Hub findings to generate the evidence. To gather evidence that security groups are not allowing ingress from 0.0.0.0/0 on port 22 it would make calls to the EC2 API. How often these calls are made depends on the data source and type of information being collected.

There are some controls for which Audit Manager cannot perform automated evidence collection for. In these instances, it requires manual uploads of evidence. An example of this may be a FedRAMP Access Management Control that requires evidence that personnel are approved by the Organization to have system access. AWS has no means to validate this information so it must be provided manually from an external source.

It’s also worth noting that Audit Manager supports AWS Organizations and is able to perform evidence gathering across multiple accounts in an organization from a single delegated account. This greatly simplifies auditing a multi-account environment. Though it does require Security Hub to be set up in the same delegated account as Audit Manager and have Security Hub and Config enabled across all accounts being audited.

Standard vs Custom Controls and Frameworks

In most situations, the standard, or pre-packaged, controls and frameworks should be sufficient. At ScaleSec we highly recommend aligning with existing standards and frameworks where possible. That being said, there may be situations where you need to tailor or enhance a framework or control, or maybe create an entirely unique framework to align with internal business requirements. When this is the case custom frameworks and controls can save the day. For example, customization may be required if you need to modify the standard FedRAMP Moderate baseline to support FedRAMP High controls or if there is a need to create a framework from scratch to reflect business-specific AWS baseline requirements for internal auditors.

Creating Custom Frameworks and Controls with Code

AWS has thorough documentation on creating and customizing frameworks and controls via the console UI.

• Creating Custom Frameworks (https://docs.aws.amazon.com/audit-manager/latest/userguide/custom-frameworks.html)
• Create Custom Controls (https://docs.aws.amazon.com/audit-manager/latest/userguide/create-controls.html)

One of the first limitations you’ll notice with Audit Manager is that it is a region-specific service and that frameworks and controls are not shared across regions. This is not an issue if you use the standard frameworks and controls since those are identical across all regions. However if you are creating custom frameworks and controls you will need to find a way to recreate these in every region and account in which you plan on running assessments. Clicking through the console to recreate the same controls in multiple regions is not ideal. Typically the solution is to just use IaC tools such as CloudFormation or Terraform to quickly deploy duplicate configurations. Unfortunately as of this writing, these tools do not support framework or control creation, only assessment creation. It shouldn’t be long before the IaC tools support automating these components, but until then we are forced to rely on the AWS CLI and API to create these.

Example Code Walkthrough

Let’s walk through some examples of creating custom controls and frameworks using the AWS CLI. In addition, check out our ScaleSec repo for the full working CLI examples as well as Python examples for the same tasks.

For our example, we want to create an AWS baseline that is unique to my company to provide reports back to internal auditors. We want to include some standard controls available in Audit Manager as well as add our own custom control. Specifically, our business has a requirement that, in production accounts, IAM user account sign-ins to the console should only be used for break-glass emergency actions. To support this requirement we want to collect evidence every time a console login occurs by an IAM user account. We will make this a custom control in our custom framework.

First, we create the custom control using the AWS CLI. To do this we create a JSON file (cloudtrail-example.json) with details on the control’s data source. Notice that the sourceSetUpOption is System_Controls_Mapping. While not necessarily intuitive, this instructs Audit Manager to automatically perform evidence gathering on a scheduled basis. We also set the sourceType to be Cloudtrail as we are specifically looking for the CloudTrail event called signin_ConsoleLogin. If desired, you can also add multiple data sources to the JSON file if the control requires gathering multiple pieces of evidence, but for this example, we’ll keep just the one.

Cloudtrail-example.json

[
  {
    "sourceName": "signin_ConsoleLogin",
    "sourceDescription": "Audit AWS console logins",
    "sourceSetUpOption": "System_Controls_Mapping",
    "sourceType": "AWS_Cloudtrail",
    "sourceKeyword": {
      "keywordInputType": "SELECT_FROM_LIST",
      "keywordValue": "signin_ConsoleLogin"
    },
    "troubleshootingText": "Validate that console login was authorized"
  }
]

With the input file completed, we can now run the following CLI command to create the custom control. The control-mapping-sources flag points to the JSON we created above. Save the output from the command for reference on a future step, specifically the unique ID assigned to the control.

 aws auditmanager create-control
 --name "Log all AWS console access"
 --description "This control will audit and collect evidence every time an IAM user account is used to sign into the AWS console."
 --control-mapping-sources file://source-definitions/cloudtrail-example.json

Now if we go into the Audit Manager console we will see the new Custom Control. We can repeat the above process to create any required custom controls.

Now that we have our custom control let’s create a framework. Again we will use the AWS CLI and a JSON Input file (control-sets.json). The JSON file contains the list of controls, custom or standard, that will be added to our framework. Within the JSON you need to include the Unique ID for each control you want in the control set. Unfortunately, this is unique per-account and not available via the console, so you’ll need to query the API. You can use the AWS CLI here and run aws auditmanager list-controls --control-type Custom or aws auditmanager list-controls --control-type Standard and parse out the IDs for the controls you require in the framework. The example below includes the ID for our custom control ID created earlier as well as two sample standard controls.

control-sets.json

[
  {
    "name": "Custom Control Set",
    "controls": [
      {
        "id": "4da046c8-4202-4e9e-aa71-32dac11c3f9e"
      },
      {
        "id": "692e1dd9-c2bf-4407-b2ea-56933e51f041"
      },
      {
        "id": "ec36488e-2ef3-4ad6-b7dd-e814d84aa1f4"
      }
    ]
  }
]

We now run the command to create the framework and control set. The control-sets flag points at our JSON file above.

aws auditmanager create-assessment-framework
 --name "Enterprise Custom Framework Demo"
 --description "This is a demo framework with both custom and standard controls included"
 --control-sets file://source-definitions/control-sets.json

Our custom framework is now created, and can be used to create a unique assessment.

To use the framework, take a look at AWS’s documentation on creating the assessment and generating reports.

Once the assessment is created, if you login to the console as an IAM user there will be evidence collected. You can add it to an assessment report and provide it to an auditor. The evidence in the report will look similar to below.

Conclusion

Audit Manager can significantly reduce the time and effort required from your team to collect evidence for audits. There are a large number of pre-packaged frameworks available at launch and it is sure to grow as the service evolves. For now, it may require a little customized code to automate deployments across regions and accounts, but it is doable until IaC tools support the capability. The CLI examples above or the Python samples in our repo should get you started. If you need help implementing Audit Manager across your organization or in creating custom frameworks or control sets reach out and we’ll gladly assist!