GCP Secret Manager - First Look

Google Cloud Platform (GCP) Secret Manager is currently in Beta and allows for a native way to manage secrets in the Google public cloud. I have recently written about secrets managers in a blog post and a whitepaper late last year. I’m pleased to take a first look at Secret Manager. If you are unsure what a secret is or why it needs to be managed, I highly recommend reading our previous blog post first.

Secrets Storage

A good secrets manager should be able to store a variety of secrets; usually at the minimum a secrets manager has the ability to store key/value pairs.

GCP Secret Manager allows for this by first creating a secret and then creating a secret version. Note that it is the secret version that contains the actual contents of a secret and to create a version of a secret you must have the roles/secretmanager.admin Cloud Identity and Access Management (IAM) role on the secret itself or all secrets in the project.

To illustrate this, here is the full process for creating a simple secret in Secret Manager:

#create a secret
gcloud beta secrets create SECRET_NAME --
replication-policy="automatic"

#create a secret version
gcloud beta secrets versions add "SECRET_NAME" 
--data-file="/path/to/file.txt"

In Secret Manager, secret names are must be unique within their project, but the data associated with it is regional. This is where the replication policy, used in the example above comes into play. With automatic replication, Google will choose the regions in which secrets are replicated. With a user-managed replication policy, secrets will only be replicated to regions that are specified. Replication policies can help organizations reach compliance-related goals for data that must be stored in different regions.

Secrets Rotation

Changing Secrets

Having the ability to rotate secrets, preferably in an automated manner, is essential in keeping them safe. Being able to programmatically query the secrets manager for the latest version of the secret during runtime or upon application boot helps to ease the pain that can be associated with changes in configuration.

Secret Manager allows for both adding new versions of secrets via gcloud beta secrets versions add and allows for updating of metadata (such as labels or replication information) as shown in an example below:

#update the labels (metadata) of a secret
gcloud beta secrets update SECRET_NAME --
update-labels=KEY=VALUE

Due to the immutable nature of secret versions, a new version must be created any time the content of a secret changes.

Secrets Generation

Dynamic Secrets

Leveraging dynamic secrets, such as keys that are generated when accessed and are associated with a Time To Live (TTL), is an easy way to reinforce security and automate the secrets lifecycle management associated with services that utilize them. In general, using dynamic secrets whenever possible helps the security posture of applications utilizing a secrets manager.

Unfortunately, as of this writing Secret Manager doesn’t have support out of the box for any type of dynamic secrets. However, it is possible to create your own dynamic secrets or auto-rotation in a GCP-native fashion by leveraging a combination of other services such as Cloud Scheduler, Pub/Sub, and Cloud Functions.

Automation Integration

Secrets Management works hand in hand with automation associated with Continuous Integration (CI) and/or Continuous Deployment (CD) as well as DevSecOps best practices. Secrets Management integration is usually performed via plugins or providers that can integrate with CI/CD systems or infrastructure as code (IaC). Secret Manager has Cloud Software Development Kit (SDK) support for the API, CLI, C#, Go, Java, Node.js, PHP, Python, and Ruby. At the time of this writing, there’s no HashiCorp Terraform provider for Secret Manager, but there is an open GitHub issue with updates on the work that’s being done on it — so it is safe to say we will see this soon.

In addition, for those of you that use Berglas, there is compatibility with Secret Manager and it offers convenience wrappers for managing secrets no matter where they reside (Cloud Storage or Secrets Manager). There’s even a command to begin a one-time migration from Berglas to Secret Manager via berglas migrate. Beginning with Berglas v.0.5.0 you can access secrets from Secret Manager using the following command (supplying project and API key):

berglas access sm://my-project/api-key

Observability

Being able to audit the use of secrets and knowing who/what is accessing them is an important factor in keeping secrets safe. It is through observability of secrets that indicators of compromise (IoC) can be found via audit and access logs. These logs can be ingested by intrusion detection/prevention systems (IDS/IPS) and other anomaly detection systems to alert on potential breaches in the security of secrets.

Secret Manager has first-class support for Cloud Audit Logs that show both administrative activity (such as updating or deleting secrets) as well as data access activity (such as accessing secret versions). An exhaustive list of audited operations can be found in the official documentation.

Access Management & Governance

For GCP, we at ScaleSec recommend that the secrets in your cloud lie in a dedicated Secrets Project no matter your secrets solution, and this advice is no different when using Secrets Manager. In addition to project-level controls, Secret Manager has governance abilities by enabling/disabling secrets, leveraging Cloud IAM for granular access control, and using VPC Service Controls for project-level API boundaries.

Secret Disabling & Revocation

In Secret Manager, the secret version’s state can be modified to control access. The version can be in either an enabled or disabled state. When a secret is disabled, the secret version becomes inaccessible. To disable a secret the following command can be used by supplying the version’s VERSION_ID and SECRET_ID:

gcloud beta secrets versions disable VERSION_ID 
--secret="SECRET_ID"

The disable command can be useful in situations in which a secret should not be used anymore, or during a situation when a secret could be potentially compromised. When a secret is in a disabled state, the contents still exist, but are not accessible. To enable a secret, the following command can be used, again by supplying the VERSION_ID and SECRET_ID:

gcloud beta secrets versions enable VERSION_ID 
--secret="SECRET_ID"

Note that enabling or disabling a secret requires the roles/secretmanager.admin IAM role on the particular version of the secret, the parent of the secret, or the project the secret resides in.

Access Control

Access Control in GCP typically leans on the mechanisms provided by Cloud IAM. Secret Management is no exception to this, and it uses Cloud IAM for both data access and administrative access of secrets. Secret Manager relies on three different IAM roles, described in the table below to control access to secrets. It should be known that the roles/secretmanager.admin role is granted on the project level which means that permissions such as list and get are granted on the project resource.

Project owners have permissions to access secrets and grant permissions through Cloud IAM. This leads to another important set of considerations to keep in mind, which are the effects of “inherited” permissions; that is to say users with primitive roles such as Owner and Editor as well as predefined or custom roles which contain Secrets manager permissions on a parent folder or project level will be able to perform actions on all secrets lower in the hierarchy.

Note that there are a number of permissions here that can be used to create custom roles that can allow for additional granularity to enforce least privilege access when it comes to access control of secrets. Also notice that these roles can be set as low as a per-secret level, which can help to restrict access to a subset of secrets for entities that can use one of these roles.

In addition to Cloud IAM, Secret Manager uses VPC Service Controls to establish service perimeters and enable context-based access to Secret Manager. This means you can limit access to Secret Manager using a number of attributes such as IP address and time of day.

Protection at Rest

When your secrets are stored, they should be protected. This is usually done via encryption via a cryptographically sound algorithm and a key management option that works for an organization’s use case. Secret Manager, like other Google Cloud services, always encrypts data before it is persisted to disk. Secret Manager supports both Google-managed encryption keys (GMEKs) and Customer-managed encryption keys (CMEKs). With no additional configuration or modifications, Secrets Manager encrypts data using AES-256 and encrypts/decrypts data transparently.

Secret Manager uses GMEKs by default, and CMEKs are not yet generally available, but interest can be expressed to the Google Cloud team. As of writing, Customer-supplied encryption keys (CSEKs) are not available as an option.

Protection in Transit

Arguably just as important as having secrets protected at rest is having them protected while they are in transit. This is especially important as applications take advantage of constantly performing operations such as reading and modifying secrets. In addition, the protection of information in transit is important to a solid defense in depth strategy for secrets management.

By default, Google encrypts and authenticates data in transit when it moves to and from Google Cloud. However, inside a Google-managed physical boundary data in transit is always authenticated, but not necessarily encrypted. Since API interactions with Secret Manager are always communicated over a secure HTTP(s) connection, this will suffice as data protected in transit for almost every use case.

Disaster Recovery

Secrets Versioning

Being able to version secrets is a great way to develop a simple rollback strategy in the case that something happens to them. Having versioning easily available via API or SDK calls can automate a rollback process in the event of a misconfigured secret or broken deployment of an application.

As noted earlier, Secret Manager relies on versioning for the creation of and rotation of secrets. As a matter of fact, a secret does not have any content until a version is created of it. So it is apparent that versioning is a key component of this product.

High Availability

As services and applications adopt a secrets manager, they tend to rely on them more, and rightfully so. As this tightly-coupled relationship develops, it becomes more important to have a secrets manager that is highly available, or applications are guaranteed to suffer downtime. This is usually achieved via using a managed, cloud-native solution or running a third-party solution in cluster mode.

Secret Manager is the first native, self-contained GCP solution to secrets management, and as such it is a managed solution that is backed by Google. However, this product is still in Beta which means there are no SLAs or technical support obligations associated with this product, which is important to keep in mind when thinking about using this solution in production.

Cost

Any secrets manager has a level of effort to get it up and running and usually has a monetary cost associated with its use. The cost of a secrets manager solution is an important factor to keep in mind when examining solutions for your products and services. The following are two categories of costs which should be taken into consideration with GCP’s Secret Manager

Operational

The cost operations such as setting up infrastructure, installing software, and enabling APIs may be “free” from a monetary standpoint, but does take time and effort.

As with most cloud service provider solutions to their own cloud, the operational cost of using Secret Manager is less than bringing in a third-party solution or developing a solution in GCP from other services. Again, note that this solution is in Beta and as such doesn’t have any guarantees for technical support.

Monetary

Services that are lower on operational overhead are typically higher on monetary cost. This is usually done via charging on the number of API calls or number/size of secrets that are stored.

Secret Manager has a cost associated with both API operations and active secret versions (active meaning secrets that are either in the ENABLED or DISABLED state). See the table below for cost details:

Conclusion

If you are using Google Cloud Platform, and don’t currently have a secrets manager, I would recommend giving Secret Manager a try. To help kickstart this process, you can use Cloud DLP to help find secrets using infoType detectors for various secrets. The following command can be used to search files in a given directory (such as a Cloud Storage bucket) and can produce a report of secrets to migrate:

find . -type f | xargs -n1 gcloud alpha dlp 
text inspect \ 
--info-types="AUTH_TOKEN,ENCRYPTION_KEY,GCP_CREDENTIAL
S,PASSWORD" \ --content-file

For those of you currently using a different secrets management solution, I would hold off on transitioning to this one until it’s out of Beta. GCP Secret Manager is a fully-fledged secrets management solution that most importantly offers a GCP-native option for secrets management, but doesn’t really have a “killer feature” that would convince me to switch management solutions at this time. Nonetheless, if you are curious about this new solution — it is easy to set up and use for non-production environments.