AWS Log Blog

AWS Log Blog

Log Blog

As companies migrate to cloud providers, protecting valuable assets is of utmost importance. The Shared Responsibility Model says that organizations are responsible for monitoring access to sensitive data in the cloud, as well as what actions are being performed in their environment.

Logging, one of the Three Pillars of Observability, is one of the most powerful tools to provide the visibility to monitor a cloud environment. Traditional forensics cannot be performed on a Lambda, because the data and resources used for those activities are not available to the end user. It is impossible to access and analyze the memory data on an instance that only existed for a few minutes and is now terminated. Given this ephemeral nature, it is best to have robust logging enabled allowing for detection, automated response, and isolation.

With that in mind, what are some common sources for logs in AWS and when should they be enabled? In this blog, we will dive into some of the logging sources that can be leveraged for Detection in the Well-Architected Security Pillar [PDF]. Included for each is the type of log each source provides as well as how to centrally manage and store the files to meet compliance requirements.

CloudTrail Logs

CloudTrail should always be enabled in your environment. It is an AWS Service which provides audit logs for most services on the platform. These logs show what actions an entity performed, who that entity was, when the action was invoked, and to what resources. These actions are known as Management Events. CloudTrail can also be configured to log Data Events, which are S3 bucket and object focused, and Lambda function executions.

By default, 90 days of CloudTrail logs are searchable on the console. In order to retain the logs and import them into another system, create and configure a Trail to send to a S3 bucket and/or CloudWatch. Data Events (API calls against S3 Objects and Lambda invocations) are not searchable in the CloudTrail Console, so in order to view them, they must be sent to one of these places.

It is a best practice to enable an Organizational Trail which applies to all AWS accounts in the Organization. An organizational trail will ensure that CloudTrail is always enabled on Organizational accounts, as well as centralize the logs and allow scaling AWS accounts horizontally.

Not every AWS service sends events to CloudTrail, and not every action is logged in CloudTrail. This means there are some actions which cannot be accounted for in CloudTrail logs. Check here for each service’s CloudTrail support.

CloudWatch Logs

CloudWatch is the catch all destination for AWS services’ logs. Lambda, RDS, API Gateway, Route53, and more are all stored in CloudWatch log groups, though not always by default. Sending logs to CloudWatch is usually a feature that must be enabled on the service.

By default, logs sent to CloudWatch are stored there indefinitely, but custom retention schedules can be configured on a per-log group basis. While it is cheaper to store the logs in S3, the CloudWatch-specific analysis tool CloudWatch Log Insights would not be able to index and search the full set of historical data. Each environment is different and there is likely a happy medium to be found between the storage tiers.

CloudWatch logs are portable. They can be subscribed to Kinesis streams, Kinesis Firehose, and streamed directly to ElasticSearch for analysis and action.

Config Logs

Ever wanted to track resources configuration changes that happen in an account? When enabled, Config will publish logs that list a resource’s current state and what has just changed. Use the logs to look into past configuration settings for resources.

Slightly more advanced usage of Config, called Config Rules, can be leveraged to create and enforce resource configuration baselines. For example, to ensure all EC2 instances have a required tag, Config Rules can monitor those resources. If the tag is not found, Config can remediate the instance. Config Rule capabilities are also available for Organizations so that rules can be deployed and managed to enforce centrally managed governance.

VPC Flow Logs

To watch for suspicious network traffic and find points of latency in a VPC, enable VPC Flow Logs.

VPC Flow logs are not enabled by default, but when configured can encompass an entire VPC, a specific subnet, or an individual network interface. When enabling Flow logs, the fields included as well as the order in which they should appear are customizable.

Flow logs can be published to CloudWatch or directly to an S3 bucket for Athena querying or ingestion into a SIEM tool.

When analyzing VPC Flow logs to track the traffic to and from resources with IP addresses, for resources that reside within the VPC, the IPv4 address in the flow log will always be presented as the internal RFC1918 address even if the Elastic Network Interface has a public IP.

GuardDuty Logs

Leveraging VPC Flow logs, CloudTrail, and DNS logs, GuardDuty is threat detection as a service. VPC Flow logs do not need to be explicitly enabled for GuardDuty to leverage them. GuardDuty retrieves these logs directly from the VPC Flow log service. GuardDuty also has a backend connection to the internal AWS DNS resolvers, the logs of which are not exposed to AWS account owners.

GuardDuty generates findings and ranks them on a scale from 0.1 to 8.9, with higher values ranking the characteristics of the finding based upon security risk. These findings can also be used with Cloudwatch Event triggers to initiate incident response with Lambdas or a Security Orchestration, Automation, and Response (SOAR) tool.

See our blog on GuardDuty for a deep dive on its features.

EC2 Instance and Container Logs

While not an AWS-specific log type, logs from EC2 instances and containers are an important data source to enable. An ephemeral instance that performs a single task and then is destroyed cannot be isolated for memory dumps or disk images, so logs are of utmost importance.

When configured on a default AMI with Imagebuilder or through Systems Manager, Cloudwatch agents can send OS and application logs/metrics directly to a CloudWatch log group. Sending to a CloudWatch log group is supported with the CloudWatch agent, as well as EKS with Fluentd.

Common configurations for sending directly to S3 typically use tools like Fluentd or Logstash. If the main SIEM tool is Splunk, the Splunk Universal forwarder can also be configured to send directly to Splunk.

S3 Access Logs

S3 Access logging provides insight on requests to access S3 objects. This is similar to CloudTrail Data Events, but S3 Access logs have a few failings in comparison.

There is slight overlap between the two types of logs on object and bucket actions, however S3 Access logs will provide additional fields.

If availability and timely notification is essential, it should be noted that S3 Access logs are best effort delivery and may experience delivery delays of a few hours. CloudTrail Data events are delivered every 5 minutes.

Choosing to enable and utilize one log type or both will be specific to the environment and the needs of the application. However for security events, CloudTrail will present the most timely and reliable source.

S3 Access Logs are sent directly to a specific logging bucket. Ensure Access logging is not enabled upon the Access logging bucket itself, or it will recursively log its own actions, which only ends once the AWS bill is noticed.

AWS Web Application Firewall (WAF) Logs

AWS WAF defends Application Load Balancers, API Gateway RESTful APIs, and Cloudfront distributions against Cross Site Scripting, SQL injection, and more. With WAF Classic, logging was managed on an individual account basis. The latest version of AWS WAF, also referred to as WAFV2, introduces a new logging method that leverages Kinesis Firehose to stream to AWS Firewall Manager.

With Firewall Manager, WAFV2 can be centrally logged and managed. If WAF Classic resources exist in an environment, they should be migrated to WAFV2 in order to leverage the new features.

What about … ?

While this guide is not an inclusive list of log types in AWS, it has touched upon some of the major logging types on the platform. Most services in AWS expose logs in some form or another, and deciding which to enable and what use cases to address is an organizational, risk-based decision. Some of these log types can generate an extremely large amount of data, which costs money to store. The cost/benefit ratio on storing logs that are not used to find security issues is relatively low. Turning on logging sources should be a methodical process of log enablement, auto-remediation when possible, and response playbooks.