Automate Security on GCP with Event Threat Detection
In this article, we will cover how to automate an Event Threat Detection finding that is focused on IAM Anomalous grants. This specific event is defined as: “Detection of privileges granted to Cloud Identity and Access Management (Cloud IAM) users and service accounts that are not members of the organization”. Leveraging automation for security in the cloud has amazing benefits including sub-second resolutions, removal of human error (no console clicking!), robust logging for postmortems, and much more. This is a completely serverless, event driven security solution. All of the code is open sourced and available for your immediate deployment here*.
* Find a bug or want to make an improvement? Feel free to create an issue or a Pull Request.
In order to deploy this solution, there are a couple of prerequisites needed.
The following APIs need to be turned on. For information about enabling services, visit the official documentation.
Cloud Resource Manager
Cloud Functions
Cloud Storage (enabled by default)
Event Threat Detection
Identity and Access Management (IAM)
A service account is needed to deploy the Terraform code with the following permissions. For information about creating a GCP Service Account, visit the official documentation.
Organization Role Administrator
Logs Configuration Writer
Organization Administrator
Pub/Sub Admin
Cloud Functions Admin
Storage Admin
Service Account Admin
Service Account User
In this section, we will breakdown how each of the below services interact to remediate an Event Threat Detection (ETD) security finding.
Event Driven Remediation Flow (click to enlarge)
Event Threat Detection (ETD) is a security service in GCP that continuously monitors logs for suspicious activity and has a built in ruleset for different finding categories. This blog is focused on auto-remediation for the rule IAM: Anomalous grant. An Anomalous Grant finding is triggered when an IAM member is created outside of the organization’s domain. To test this solution, we will create a member with an @gmail.com email address and assign the role “Project Editor”. ETD uses Google’s own threat intelligence and can send its findings to Cloud Logging as well as the Security Command Center.
Once the Event Threat Detection API is enabled, there are a couple of configuration changes required. ETD is not currently supported in Terraform so these updates must be made manually.
Cloud Logging (previously known as Stackdriver Logging) is GCP’s native logging and monitoring solution that ETD analyzes for suspicious activity. When ETD has a finding, it can send the finding to Cloud Logging which can then export the finding from Cloud Logging to Cloud Pub/Sub.
The aggregated log sink is used to export the ETD findings from Cloud Logging to Cloud Pub/Sub. The sink is deployed on the Organization level and covers all projects in the organization. A specific filter is applied to the log sink to only capture and export the logs we intend to act on. The Terraform code is below.
destination is configured to send the logs to a Cloud Pub/Sub topic
filter is a variable which will only export the logs specified
include_children configures the aggregated sink to also apply to child GCP projects under the organization
| resource "google_logging_organization_sink" "iam_anomalous_grant_log_sink" { |
| name = "${var.org_id}-iam-anomalous-grant-log-sink" |
| org_id = var.org_id |
| destination = "pubsub.googleapis.com/projects/${var.project}/topics/${google_pubsub_topic.iam_anomalous_grant_topic.name}" |
| filter = var.org_sink_filter |
| include_children = true |
| } |
org_sink_filter is the variable for the aggregated log sink
The variable will only capture the IAM: Anomalous grant findings:
| variable "org_sink_filter" { |
| description = "The Log Filter to apply to the Org Level export." |
| default = "resource.type:threat_detector resource.labels.detector_name=iam_anomalous_grant" |
| } |
Cloud Pub/Sub is key in our automation flow because it ties together all the services and facilitates the event driven remediation. Logs that are sent to Cloud Pub/Sub can be sent to numerous places, but in our flow, we are sending the logs to a Cloud Function via a trigger. When the Cloud Function is deployed, it will configure a trigger on the Pub/Sub topic to automatically kick off its code to remediate a finding when one occurs.
| resource "google_pubsub_topic" "iam_anomalous_grant_topic" { |
| name = "${lower(var.name)}-iam-anomalous-grant-topic" |
| project = var.project |
| } |
Cloud Functions is a serverless compute platform that will run code in many different runtimes. In this event driven flow, Python 3.7 is the runtime of choice and will automatically remediate the IAM: Anomalous grant finding from ETD. The remediation steps are broken down below, but the end result is that all suspicious IAM members that were added to the project or organization are removed. Additionally, the Cloud Function leverages a custom service account that is using only the minimum required permissions.
Let’s first take a look at the Terraform code used to deploy the Cloud Function.
source_archive_bucket and source_archive_object are where the Cloud Function’s code is uploaded
entry_point is the main function that processes the Cloud Pub/Sub message
event_trigger is where the Cloud Function subscribes to the Cloud Pub/Sub topic
| resource "google_cloudfunctions_function" "iam_anomalous_grant_function" { |
| provider = google-beta |
| name = local.function_name |
| description = "Google Cloud Function to remediate Event Threat Detector IAM anomalous grant findings." |
| available_memory_mb = 128 |
| source_archive_bucket = google_storage_bucket.bucket.name |
| source_archive_object = google_storage_bucket_object.archive.name |
| timeout = 60 |
| entry_point = "process_log_entry" |
| service_account_email = google_service_account.iam_anomalous_grant_sa.email |
| runtime = "python37" |
| event_trigger { |
| event_type = "google.pubsub.topic.publish" |
| resource = google_pubsub_topic.iam_anomalous_grant_topic.name |
| } |
| } |
Now, let’s take a look at the python main.py that makes up the Cloud Function.
| def process_log_entry(data, context): |
| data_buffer = base64.b64decode(data['data']) |
| log_entry = json.loads(data_buffer) |
| service = create_service() |
| def create_service(): |
| return googleapiclient.discovery.build('cloudresourcemanager', 'v1') |
| ## Finds the bound anomalous member as GCP stores it. Member input capitalization may vary |
| ## so this is to capture how it is stored in GCP. |
| def find_member(outside_member_id_list): |
| members = [] |
| try: |
| for member in outside_member_id_list: |
| member = member['member'] |
| members.append(member) |
| except: |
| logging.debug("Could not find anomalous member on resource.") |
| sys.exit(0) |
| return members |
| ## Check if member is still bound to resource |
| def check_member_on_resource(outside_member_ids, resource_bindings): |
| for bindings in resource_bindings: |
| for values in bindings.values(): |
| for member in outside_member_ids: |
| if member in values: |
| try: |
| print(f"Member found: {member}") |
| return True |
| except: |
| logging.info("Did not find the anomalous member.") |
| continue |
| else: |
| logging.debug("Did not find the anomalous member.") |
| ## Determine if project or organization and perform logic based on bound resource layer |
| properties = log_entry['jsonPayload']['properties'] |
| for entry in properties: |
| if 'project_id' in entry: |
| resource = properties['project_id'] |
| print(f"The Project ID is {resource}") |
| resource_bindings = retrieve_bindings(service, resource) |
| check_if_member_exists = check_member_on_resource(outside_member_ids, resource_bindings) |
| if check_if_member_exists is True: |
| bindings_removed = remove_anomalous_iam_resource(outside_member_ids, resource_bindings) |
| set_iam_binding_resource(resource, service, bindings_removed) |
| else: |
| logging.debug("Member does not exist.") |
| sys.exit(0) |
| elif 'organization_id' in entry: |
| resource = 'organizations/' + properties['organization_id'] |
| print(f"The Organization is {resource}") |
| resource_bindings = retrieve_bindings(service, resource) |
| check_if_member_exists = check_member_on_resource(outside_member_ids, resource_bindings) |
| if check_if_member_exists is True: |
| bindings_removed = remove_anomalous_iam_resource(outside_member_ids, resource_bindings) |
| set_iam_binding_resource(resource, service, bindings_removed) |
| else: |
| logging.debug("Member does not exist.") |
| sys.exit(0) |
| ## Resources have IAM bindings. We need to return those to parse through. |
| def retrieve_bindings(service, resource): |
| if 'organizations' in resource: |
| request = service.organizations().getIamPolicy(resource=f"{resource}") |
| response = request.execute() |
| resource_bindings = response.pop("bindings") |
| print(f"Current organization bindings: {resource_bindings}") |
| else: |
| request = service.projects().getIamPolicy(resource=resource) |
| response = request.execute() |
| resource_bindings = response.pop("bindings") |
| print(f"Current project bindings: {resource_bindings}") |
| return resource_bindings |
| ## Looks for our anomalous IAM member and removes from resource bindings |
| def remove_anomalous_iam_resource(outside_member_ids, resource_bindings): |
| bindings_removed = resource_bindings |
| for dic in bindings_removed: |
| if 'members' in dic: |
| for values in dic.values(): |
| for member in outside_member_ids: |
| if member in values: |
| try: |
| values.remove(member) |
| print(f"Member removed: {member}") |
| except: |
| logging.info(f"{member} not found.") |
| continue |
| else: |
| logging.debug(f"{member} not found.") |
| print(f"New bindings after anomalous member(s) removed: {bindings_removed}") |
| return bindings_removed |
| ## Set our new resource IAM bindings |
| def set_iam_binding_resource(resource, service, bindings_removed): |
| set_iam_policy_request_body = { |
| "policy": { |
| "bindings": [ |
| bindings_removed |
| ] |
| } |
| } |
| if 'organizations' in resource: |
| request = service.organizations().setIamPolicy(resource=f"{resource}", body=set_iam_policy_request_body) |
| binding_response = request.execute() |
| print(f"New policy attached to the organization: {binding_response}") |
| else: |
| request = service.projects().setIamPolicy(resource=resource, body=set_iam_policy_request_body) |
| binding_response = request.execute() |
| print(f"New policy attached to the project: {binding_response}") |
Deploying and testing this solution is fairly straightforward if you have an organization and a project in GCP.
git clone
git@github.com:ScaleSec/gcp_threat_detection_auto_remediation.git2. Change directory into the newly cloned repository.
cd gcp_threat_detection_auto_remediation
3. Create a terraform.tfvars file — Replace the values before running the below command:
cat > terraform.tfvars <<EOF
org_id = “<<replace with your org id>>”
project = “<<replace with your project id>>”
EOF
4. Authenticate your Google Cloud Service Account in one of ways defined in the Terraform documentation. This is required due to issue #5288.
gcloud auth activate-service-account
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/yourSAK
ey.json
5. Run the following Terraform commands:
terraform init
terraform plan
terraform apply
Add a @gmail.com IAM member to your project with the role Project Editor.
Refresh the IAM page and the previously added IAM member should be removed.
Navigate to the Cloud Functions page in the Cloud Console and select the function deployed via Terraform.
Click the ‘View Logs’ Button to view the outputs of the function.
This event driven security remediation flow will remediate the ETD IAM: Anomalous grant finding in around one second depending on the number of IAM members in your organization. The entire architecture is controlled via code and can be stored in GitHub or your code repository of choice to take full advantage of all the benefits of a CI/CD pipeline. If you’d like to deploy this solution yourself, you can find all of the code here.