Skip to content
Jason DykeApr 9, 2020 12:00:00 AM9 min read

Automate Security on GCP with Event Threat Detection

Automate Security on GCP with Event Threat Detection

Automate Security on GCP with Event Threat Detection

Introduction

In this article, we will cover how to automate an Event Threat Detection finding that is focused on IAM Anomalous grants. This specific event is defined as: “Detection of privileges granted to Cloud Identity and Access Management (Cloud IAM) users and service accounts that are not members of the organization”. Leveraging automation for security in the cloud has amazing benefits including sub-second resolutions, removal of human error (no console clicking!), robust logging for postmortems, and much more. This is a completely serverless, event driven security solution. All of the code is open sourced and available for your immediate deployment here*.

* Find a bug or want to make an improvement? Feel free to create an issue or a Pull Request.

Prerequisites

In order to deploy this solution, there are a couple of prerequisites needed.

The following APIs need to be turned on. For information about enabling services, visit the official documentation.

  • Cloud Resource Manager

  • Cloud Functions

  • Cloud Storage (enabled by default)

  • Event Threat Detection

  • Identity and Access Management (IAM)

A service account is needed to deploy the Terraform code with the following permissions. For information about creating a GCP Service Account, visit the official documentation.

At the organization level:

  • Organization Role Administrator

  • Logs Configuration Writer

  • Organization Administrator

At the project level:

  • Pub/Sub Admin

  • Cloud Functions Admin

  • Storage Admin

  • Service Account Admin

  • Service Account User

Solution Overview

In this section, we will breakdown how each of the below services interact to remediate an Event Threat Detection (ETD) security finding.

Event Driven Remediation Flow

Event Driven Remediation Flow (click to enlarge)

Event Threat Detection (Beta)

Event Threat Detection (ETD) is a security service in GCP that continuously monitors logs for suspicious activity and has a built in ruleset for different finding categories. This blog is focused on auto-remediation for the rule IAM: Anomalous grant. An Anomalous Grant finding is triggered when an IAM member is created outside of the organization’s domain. To test this solution, we will create a member with an @gmail.com email address and assign the role “Project Editor”. ETD uses Google’s own threat intelligence and can send its findings to Cloud Logging as well as the Security Command Center.

Once the Event Threat Detection API is enabled, there are a couple of configuration changes required. ETD is not currently supported in Terraform so these updates must be made manually.

  • Verify the rule for “IAM Anomalous Grant” is enabled.
Verify the rule for “IAM Anomalous Grant” is enabled
  • Include all current and future projects for the sources.
Include all current and future projects for the sources
  • Turn on “Log Findings to Stackdriver” and select a project to send findings.
Verify the rule for “Turn on “Log Findings to Stackdriver” and select a project to send findings

Cloud Logging

Cloud Logging (previously known as Stackdriver Logging) is GCP’s native logging and monitoring solution that ETD analyzes for suspicious activity. When ETD has a finding, it can send the finding to Cloud Logging which can then export the finding from Cloud Logging to Cloud Pub/Sub.

Aggregated Log Sink

The aggregated log sink is used to export the ETD findings from Cloud Logging to Cloud Pub/Sub. The sink is deployed on the Organization level and covers all projects in the organization. A specific filter is applied to the log sink to only capture and export the logs we intend to act on. The Terraform code is below.

  • destination is configured to send the logs to a Cloud Pub/Sub topic

  • filter is a variable which will only export the logs specified

  • include_children configures the aggregated sink to also apply to child GCP projects under the organization

resource "google_logging_organization_sink" "iam_anomalous_grant_log_sink" {
name = "${var.org_id}-iam-anomalous-grant-log-sink"
org_id = var.org_id
destination = "pubsub.googleapis.com/projects/${var.project}/topics/${google_pubsub_topic.iam_anomalous_grant_topic.name}"
filter = var.org_sink_filter
include_children = true
}
view raw main.tf hosted with ❤ by GitHub
  • org_sink_filter is the variable for the aggregated log sink

  • The variable will only capture the IAM: Anomalous grant findings:

variable "org_sink_filter" {
description = "The Log Filter to apply to the Org Level export."
default = "resource.type:threat_detector resource.labels.detector_name=iam_anomalous_grant"
}
view raw variables.tf hosted with ❤ by GitHub

Cloud Pub/Sub

Cloud Pub/Sub is key in our automation flow because it ties together all the services and facilitates the event driven remediation. Logs that are sent to Cloud Pub/Sub can be sent to numerous places, but in our flow, we are sending the logs to a Cloud Function via a trigger. When the Cloud Function is deployed, it will configure a trigger on the Pub/Sub topic to automatically kick off its code to remediate a finding when one occurs.

  • This Terraform code creates the Cloud Pub/Sub topic and the Cloud Function code will create the trigger tying them together.
resource "google_pubsub_topic" "iam_anomalous_grant_topic" {
name = "${lower(var.name)}-iam-anomalous-grant-topic"
project = var.project
}
view raw main.tf hosted with ❤ by GitHub

Cloud Functions

Cloud Functions is a serverless compute platform that will run code in many different runtimes. In this event driven flow, Python 3.7 is the runtime of choice and will automatically remediate the IAM: Anomalous grant finding from ETD. The remediation steps are broken down below, but the end result is that all suspicious IAM members that were added to the project or organization are removed. Additionally, the Cloud Function leverages a custom service account that is using only the minimum required permissions.

Let’s first take a look at the Terraform code used to deploy the Cloud Function.

  • source_archive_bucket and source_archive_object are where the Cloud Function’s code is uploaded

  • entry_point is the main function that processes the Cloud Pub/Sub message

  • event_trigger is where the Cloud Function subscribes to the Cloud Pub/Sub topic

resource "google_cloudfunctions_function" "iam_anomalous_grant_function" {
provider = google-beta
name = local.function_name
description = "Google Cloud Function to remediate Event Threat Detector IAM anomalous grant findings."
available_memory_mb = 128
source_archive_bucket = google_storage_bucket.bucket.name
source_archive_object = google_storage_bucket_object.archive.name
timeout = 60
entry_point = "process_log_entry"
service_account_email = google_service_account.iam_anomalous_grant_sa.email
runtime = "python37"
event_trigger {
event_type = "google.pubsub.topic.publish"
resource = google_pubsub_topic.iam_anomalous_grant_topic.name
}
}
view raw main.tf hosted with ❤ by GitHub

Now, let’s take a look at the python main.py that makes up the Cloud Function.

  • First, it’s important to decode the Cloud Pub/Sub message from base64 to JSON and establish our service connection information to Google’s API. Converting the message to JSON allows us to more easily interact with the incoming data.
def process_log_entry(data, context):
data_buffer = base64.b64decode(data['data'])
log_entry = json.loads(data_buffer)
service = create_service()
def create_service():
return googleapiclient.discovery.build('cloudresourcemanager', 'v1')
view raw main.py hosted with ❤ by GitHub
  • Before we progress further, we need to find all of the IAM Anomalous members that were created.
## Finds the bound anomalous member as GCP stores it. Member input capitalization may vary
## so this is to capture how it is stored in GCP.
def find_member(outside_member_id_list):
members = []
try:
for member in outside_member_id_list:
member = member['member']
members.append(member)
except:
logging.debug("Could not find anomalous member on resource.")
sys.exit(0)
return members
view raw main.py hosted with ❤ by GitHub
  • Once we have the IAM members, we check to verify that the IAM member(s) that triggered the Event Threat Detection finding still exists.
## Check if member is still bound to resource
def check_member_on_resource(outside_member_ids, resource_bindings):
for bindings in resource_bindings:
for values in bindings.values():
for member in outside_member_ids:
if member in values:
try:
print(f"Member found: {member}")
return True
except:
logging.info("Did not find the anomalous member.")
continue
else:
logging.debug("Did not find the anomalous member.")
view raw main.py hosted with ❤ by GitHub
  • Now that we know if the user exists, we proceed based on whether the IAM member was bound on the organization or the project level.
## Determine if project or organization and perform logic based on bound resource layer
properties = log_entry['jsonPayload']['properties']
for entry in properties:
if 'project_id' in entry:
resource = properties['project_id']
print(f"The Project ID is {resource}")
resource_bindings = retrieve_bindings(service, resource)
check_if_member_exists = check_member_on_resource(outside_member_ids, resource_bindings)
if check_if_member_exists is True:
bindings_removed = remove_anomalous_iam_resource(outside_member_ids, resource_bindings)
set_iam_binding_resource(resource, service, bindings_removed)
else:
logging.debug("Member does not exist.")
sys.exit(0)
elif 'organization_id' in entry:
resource = 'organizations/' + properties['organization_id']
print(f"The Organization is {resource}")
resource_bindings = retrieve_bindings(service, resource)
check_if_member_exists = check_member_on_resource(outside_member_ids, resource_bindings)
if check_if_member_exists is True:
bindings_removed = remove_anomalous_iam_resource(outside_member_ids, resource_bindings)
set_iam_binding_resource(resource, service, bindings_removed)
else:
logging.debug("Member does not exist.")
sys.exit(0)
## Resources have IAM bindings. We need to return those to parse through.
def retrieve_bindings(service, resource):
if 'organizations' in resource:
request = service.organizations().getIamPolicy(resource=f"{resource}")
response = request.execute()
resource_bindings = response.pop("bindings")
print(f"Current organization bindings: {resource_bindings}")
else:
request = service.projects().getIamPolicy(resource=resource)
response = request.execute()
resource_bindings = response.pop("bindings")
print(f"Current project bindings: {resource_bindings}")
return resource_bindings
## Looks for our anomalous IAM member and removes from resource bindings
def remove_anomalous_iam_resource(outside_member_ids, resource_bindings):
bindings_removed = resource_bindings
for dic in bindings_removed:
if 'members' in dic:
for values in dic.values():
for member in outside_member_ids:
if member in values:
try:
values.remove(member)
print(f"Member removed: {member}")
except:
logging.info(f"{member} not found.")
continue
else:
logging.debug(f"{member} not found.")
print(f"New bindings after anomalous member(s) removed: {bindings_removed}")
return bindings_removed
## Set our new resource IAM bindings
def set_iam_binding_resource(resource, service, bindings_removed):
set_iam_policy_request_body = {
"policy": {
"bindings": [
bindings_removed
]
}
}
if 'organizations' in resource:
request = service.organizations().setIamPolicy(resource=f"{resource}", body=set_iam_policy_request_body)
binding_response = request.execute()
print(f"New policy attached to the organization: {binding_response}")
else:
request = service.projects().setIamPolicy(resource=resource, body=set_iam_policy_request_body)
binding_response = request.execute()
print(f"New policy attached to the project: {binding_response}")
view raw main.py hosted with ❤ by GitHub

Deployment and Testing

Deploying and testing this solution is fairly straightforward if you have an organization and a project in GCP.

To deploy this solution:

  1. Clone the Repository locally.
git clone
git@github.com:ScaleSec/gcp_threat_detection_auto_remediation.git

2. Change directory into the newly cloned repository.

cd gcp_threat_detection_auto_remediation

3. Create a terraform.tfvars file — Replace the values before running the below command:

cat > terraform.tfvars <<EOF  

org_id = “<<replace with your org id>>”

project = “<<replace with your project id>>”

EOF

4. Authenticate your Google Cloud Service Account in one of ways defined in the Terraform documentation. This is required due to issue #5288.

gcloud auth activate-service-account

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/yourSAK
ey.json

5. Run the following Terraform commands:

terraform init
terraform plan
terraform apply

To test the solution:

  • Add a @gmail.com IAM member to your project with the role Project Editor.

  • Refresh the IAM page and the previously added IAM member should be removed.

  • Navigate to the Cloud Functions page in the Cloud Console and select the function deployed via Terraform.

  • Click the ‘View Logs’ Button to view the outputs of the function.

Conclusion

This event driven security remediation flow will remediate the ETD IAM: Anomalous grant finding in around one second depending on the number of IAM members in your organization. The entire architecture is controlled via code and can be stored in GitHub or your code repository of choice to take full advantage of all the benefits of a CI/CD pipeline. If you’d like to deploy this solution yourself, you can find all of the code here.

 

avatar

Jason Dyke

Read more articles by Jason Dyke

RELATED ARTICLES

The information presented in this article is accurate as of 7/19/23. Follow the ScaleSec blog for new articles and updates.