A Comparison of Secrets Managers for GCP
Earlier this year, Google made Secret Manager generally available (GA), and with this release came a new, native secrets manager solution available for workloads in Google Cloud. In an earlier blog post, we took a first look at Secret Manager.
As more secrets management solutions come to market, the decision of whether or not to use this solution may not be as clear for some organizations as it is for others. This article will examine three of the most popular solutions for Google Cloud Platform (GCP) and compare their features, security concerns, and cost associated with each one.
For those new to secrets or secrets managers, check out our previous article on secrets managers here. In short, secrets are values that typically authenticate applications and users to give them access to sensitive systems, services, and information. A secrets manager is used to secure and easily manage these secrets and is essential to creating applications that are suitable for deployment on modern cloud platforms, such as a Twelve-Factor Application.
Secrets manager solutions come in two different categories: native and third party. Cloud Native solutions are built into the cloud service provider platforms (i.e. Cloud Services) and third party solutions can be deployed onto the cloud native solution. This article will compare one native solution, GCP Secret Manager, and two third party solutions, HashiCorp Vault and Berglas.
Third party solutions are usually not tied to any specific cloud and can be very useful for those who want a unified secrets solution across on-premise or other clouds such as Amazon Web Services and Microsoft Azure. Public clouds have native solutions to do the undifferentiated heavy lifting — that is, taking care of managing servers and APIs under the hood so that focus can remain on developing applications and services.
Some secret managers focus on doing one thing (managing key/value storage of secrets), whereas other solutions are feature-rich and can include access brokering and inline encryption. Note that this is not an exhaustive list of features for each secrets manager, but it does provide an outline of the most common features.
A comparison of features for each secrets manager
Most secrets come in the form of key value pairs, such as username/password or environment variable name/value. Being able to store key-value pairs is a fundamental feature to being a secrets manager and is present in all three solutions.
Secret Manager
Secret Manager does not share any limits on the size of secrets that can be stored within it, however it is safe to say that it should be able to handle any type of typical key/value store secret.
Berglas
Berglas uses a GCM cipher mode to encrypt data, meaning the data must fit in memory and is limited to 64GiB. Seeing that Berglas is targeted at holding typical secrets such as API keys, passwords, and certificates, most users should not run into this limit.
HashiCorp Vault
HashiCorp Vault does not enforce a size limit for key-value pairs, but it can vary depending on the storage backend. It should be noted that Vault’s HTTP API has a default limit of 32MB to help mitigate denial of service (DoS) attacks due to arbitrarily large requests, but this can be adjusted by changing the configuration file.
Being able to rotate secrets is essential in maintaining proper security hygiene for applications and services. Rotating secrets ensures that any secrets that exist in code or infrastructure will not be valid if compromised, and in addition ensures that the application is doing proper fetching from the secrets manager, thus encouraging a tight integration with the secrets manager and those applications that adopt it. In other words, regularly rotating secrets not only keeps applications secure, but encourages the adoption of a secrets manager throughout the project and/or company. For those who are compliance conscious, NIST 800–57 prescribes the rotation of secrets and having a proper rotation mechanism can help significantly with achieving that or similar types of goals.
Secret Manager
At the time of writing, GCP Secret Manager does not have a single API call to rotate secrets. Luckily, creating a new version of a secret managed by Secret Manager is a single API call and can easily be scripted into applications, cloud functions, or cron jobs.
Berglas
Like Secret Manager, Berglas does not have a mechanism in place to simply rotate secrets, instead the secret value would have to be updated using either the Berglas CLI or library.
HashiCorp Vault
Vault has a feature called dynamic secrets, which allows for Just-in-time (JIT) provisioning of secrets . Basically this means that dynamic secrets are generated when they are accessed and don’t exist until they are read, lowering the risk of someone stealing them or another client using the same secrets. As of the time of this writing, Vault supports dynamic secrets with a variety of backends including databases, messaging queues, cloud providers, Secure Shell (SSH), and Public Key Infrastructure (PKI). Other types of secrets can be enabled with plugins.
Password Generation is not an essential feature of a secrets manager, but can be useful for automation purposes and serves to be a useful tool to have in the cloud security toolbox.
Both Berglas and GCP Secret Manager do not have this feature since their focus is on being a simple key/value storage for generally used, basic secrets.
HashiCorp Vault has the vault-secrets-gen plugin to generate high entropy passwords and passphrases with a variety of options to control the secret’s length and complexity.
Infrastructure as Code (IaC) is important for cloud environments, and we at ScaleSec are indeed fans of it. The two most prominent declarative IaC solutions for GCP are HashiCorp Terraform and Google Cloud Deployment Manager. Using a secrets manager in IaC ensures that secrets are not hardcoded into source code and the most recent version of a particular secret can be fetched on demand when code is run.
Deployment Manager
Deployment Manager has support for all three solutions and is able to both read and manage secrets through the use of adding an API as a Type Provider. By adding the Type Providers for your secret manager of choice, all resources of that secret manager are exposed to Deployment Manager as base types that can be used in configurations. Deployment manager does this by leveraging the RESTful API that supports Create, Read, Update, and Delete (CRUD) operations — a feature that each of these secret managers have.
HashiCorp Terraform
Terraform also supports all three solutions and is able to both create and read secrets. HashiCorp Vault has first class support as a Terraform provider. The terraform-provider-google-beta provider has support for performing CRUD operations on secrets stored within Secret Manager. There’s also a Berglas Terraform provider available that also allows for the creation and management of secrets using Berglas.
It is extremely important for any secrets manager to effectively keep secrets safe. Security for secrets managers must be examined in terms of confidentiality, integrity, and availability of the secrets that they are managing. The following topics will be examined: what access control mechanisms are in place to manage both the modification and the retrieval of secrets, how secrets are protected at rest, and how secrets are protected in transit. This is by no means an exhaustive list of security concerns that should be considered when selecting a secrets manager for use in the cloud, but highlights the most important points that should be taken into account before selecting a secrets manager.
A summary of security features for each secrets manager
Secret Manager
The Secret Manager API is governed by Google Cloud IAM. Cloud IAM governs the access of payload of secrets, administration, and viewing of metadata of secret manager resources individually through the use of Cloud IAM roles. A list of these roles can be found in the table below:
The three predefined roles associated with Secret Manager
Note that, as with other Google resources, the effects of the hierarchy created by Cloud IAM exist for resources, including secrets (which is the lowest level resource in this case) managed by Secret Manager. To help facilitate the principle of least privilege and granting the minimum level of access to an identity, it is recommended that if an identity only needs access to a single secret’s value — only grant that identity the ability to access the single secret.
Berglas
Berglas also leverages Cloud IAM, but in a different way due to the fact that it uses Cloud Storage buckets to store secrets. Authorizations to secrets (e.g. specifying who/what has access to a secret) is governed by using the berglas grant and berglas revoke commands or associated API methods. These API methods use Cloud IAM internally by adding the
roles/storage.objectCreator (to create a secret),
roles/storage.objectViewer (to view a secret), and
roles/storage.objectAdmin (to delete a secret).
Cloud KMS permissions are also necessary for encrypting/decrypting data as necessary for access.
HashiCorp Vault
HashiCorp Vault manages access control through path-based policies. Through the use of policies, Role-Based Access Control can be achieved by specifying access privileges via path-based matching. Vault policies can be written in JSON or HashiCorp Configuration Language (HCL) for easier human readability. Like IAM policies, Vault policies have an implicit deny as well, and paths in Vault must be explicitly allowed along with the corresponding capabilities.
An example of a Vault policy written in HCL is given below. Note the glob (*) syntax for allowing access to all paths under the specified prefix (in this case secret/), a path for specific access (such as secret/super-secret), and conditional access based on allowed parameters (shown in the secret/restricted block):
| path "secret/*" { |
| capabilities = ["create", "read", "update", "delete", "list"] |
| } |
| path "secret/super-secret" { |
| capabilities = ["deny"] |
| } |
| path "secret/restricted" { |
| capabilities = ["create"] |
| allowed_parameters = { |
| "foo" = [] |
| "bar" = ["zip", "zap"] |
| } |
| } |
Secret Manager
Secret Manager, like other Google Cloud services, always encrypts data before it is persisted to disk. This is known as default encryption in which Google both creates and manages the cryptographic material used to protect data at rest. In other words, data is already encrypted transparently using AES-256. Through the use of Cloud KMS, Secret Manager also supports Customer-managed encryption keys (CMEKs). As of writing, Customer-supplied encryption keys (CSEKs) are not available as an option.
Berglas
When encrypting a secret, Berglas generates a data encryption key (DEK) using 256-bit Advanced Encryption Standard (AES) cipher in the Galois Counter Mode (GCM) via Go’s crypto package for each secret. This means that each secret has its own DEK. Berglas then encrypts the plaintext data using the locally-generated DEK, producing encrypted ciphertext, prepended with the unique nonce created from AES-GCM. Berglas then uses a process called envelope encryption to then encrypt the DEK using a specified Cloud KMS key, known as a key encryption key (KEK). Finally, Berglas stores the Cloud KMS key name, encrypted DEK, and encrypted ciphertext as a single blob in Cloud Storage.
When decrypting a secret, Berglas downloads the blob from Cloud Storage and separates the Cloud KMS key name, encrypted DEK, and ciphertext from the blob. Then using Cloud KMS, the DEK is decrypted, and then the ciphertext data is locally decrypted using the decrypted DEK.
HashiCorp Vault
The storage backends used by HashiCorp Vault are designed with a zero trust mindset. Vault uses a security barrier for all requests made to the backend. The security barrier automatically encrypts all data leaving Vault using a 256-bit AES-GCM with a 96-bit nonce. The nonce is randomly generated for every encrypted object. When data is read from the security barrier the GCM authentication tag is verified during the decryption process to detect integrity breaches.
By design, Vault needs an encryption key in order to decrypt data. This encryption key is stored with the data, but is encrypted with another encryption key known as the master key. The master key is actually not stored anywhere. Instead of distributing this single master key to an operator, Vault uses an algorithm known as Shamir’s Secret Sharing to split the key into shards. A certain threshold of shards is required to reconstruct the master key. This protects against insider threat because in the case of a detected intrusion, the Vault data can be sealed in order to quickly minimize damages. The master key must then be reconstructed using the distributed shards to reach a quorum and unseal the vault so that applications and services can access secrets again. Google Cloud KMS keys, including the use of CMEKs and CSEKs, can also be used to auto-unseal the Vault.
Secret Manager
By default, Google encrypts and authenticates data in transit when it moves to and from Google Cloud. However, inside a Google-managed physical boundary data in transit is always authenticated, but not necessarily encrypted. Since API interactions with Secret Manager are always communicated over a secure HTTPS connection, this will suffice as data protected in transit for almost every use case.
Berglas
Since Berglas uses Google services like Cloud Storage and Cloud KMS under the hood, API interactions are also communicated over a secure HTTPS connection, and thus data is protected in transit for most use cases.
HashiCorp Vault
Since HashiCorp Vault is run on hardware provisioned by the consumer of it, the networking situation can differ between installations. Transport Layer Security (TLS) can be configured on Vault via a listener to enable encryption in transit, and thus protects data as it travels over a network. Vault can be configured to be highly available and have end-to-end TLS over a network with relative ease.
Cost must also be considered since the pricing models differ between native and third party offerings. The native option, Secret Manager has a per active secret monthly charge and access operations for those secrets, however has no operational cost. Both third party secret managers (HashiCorp Vault and Berglas) have operational cost due to the infrastructure necessary to support these solutions, however have no cost for per active secret monthly charge or access operations. Also note that HashiCorp Vault has a licensing cost for Enterprise features.
A cost comparison for each secrets manager
Secret Manager has a cost associated with both API operations and active secret versions (active meaning secrets that are either in the ENABLED or DISABLED state). Also note that this is not only a per secret charge, but is also _a per location _charge. This means that for each secret stored in Secret Manager, multiply by the number of locations that secrets are stored in.
Berglas is simply a command line tool and library for storing and retrieving secrets stored in Cloud Storage with encryption fulfilled by Cloud KMS. When you bootstrap Berglas, you specify a BUCKET_LOCATION (default US multi-regional location) where the bucket will be located, and Berglas will provision one for you; optionally you can have full control over the Cloud Storage bucket and Cloud KMS key via custom setup.
In addition to the Cloud Storage setup, Berglas can integrate with App Engine (Standard and Flex), Cloud Run, Cloud Functions, Cloud Build, Kubernetes, and berglas exec (anywhere). Depending on the deployment of Berglas, the cost of using one of these integrations also need to be kept in mind.
HashiCorp Vault comes in both open-source and two enterprise versions: Enterprise Platform and Enterprise Modules. Both enterprise solutions come at an additional cost, which can be given by contacting HashiCorp. Vault is also a self-hosted solution for both open source and enterprise, therefore both provisioned infrastructure and operational upkeep must be considered when calculating cost. There is no additional cost for the number of stored secrets or secrets retrieval.
For most typical setups, the cost of HashiCorp Vault to store secrets in GCP is comparable to that of Berglas due to the fact that HashiCorp Vault runs on VM instances or containers on other GCP services in addition to the storage backend.
For new and/or lightweight projects, GCP Secret Manager should definitely be investigated. Secret Manger should be favored over Berglas in most situations directly, since it has less operational overhead and complexity, however the per active secret and access costs associated with Secret Manager should be kept in mind.
If a native secret manager is desired, Secret Manger is the way to go. Leveraging features like Cloud IAM for secret-level access granularity is a good, cloud native way to provision access to secrets. In addition, you will not have to manage resources which can save a lot of operational costs.
The most feature rich and the solution of choice is HashiCorp Vault. It excels in most categories and has great community support. Arguably, it is much more than a secrets manager. It has been described as the “Swiss Army Knife” of cloud security because of its additional features such as brokering SSH access, being a Certificate Authority to generate X.509 certificates on demand, and performing encryption as a service. This is also the best option if you are considering a multi-cloud strategy in the future and would like to keep secrets in a centralized location.