Rotate Secrets
using AWS Lambda

Mateusz Wilczyński CTO

When storing secrets using technologies like AWS Secrets Manager, the rotation of certain secrets, like database credentials, is automated and easy to set up. When it comes to the custom secrets rotation, like the external API keys, the case is not so simple anymore.

In the previous article, we covered the basics of secret rotation including the rationale behind rotation, threats associated with secrets, and the rotation process. If you haven’t read it, check it out right now. Introduction to Secrets Rotation

Rotate custom secrets

Custom secrets stored in Secrets Manager can be rotated using a bespoke AWS Lambda function. During secret rotation, AWS calls the same Lambda function 4 times, each time with different parameters. The function is executed with the following parameters:
The process consists of 4 steps:

Step 1 createSecret – create a new version of the secret

This step is about generating a new secret like password in the lambda code. It will be stored in the Secrets Manager with the AWSPENDING staging label. If you are rotating external API keys, this step consists of making an HTTP request to the API to generate a new API key.

Step 2 setSecret – change the credentials in the database or service

If you are rotating external service API keys, you don’t have to do anything here. The API key has already been saved in the external API during the creation HTTP request in Step 1. In the case of generating a secret directly in the Lambda code in the first step, you have to set the generated secret in the database or service here.

Step 3 testSecret – Test the new secret version

In this step, you have an opportunity to ensure the new secret works as expected. Rotating external API keys, you should test that you can make successfully call the API using the new key. When rotating database credentials, you can make a test request to the database or service using the generated key.

Step 4 finishSecret – Finish the rotation

Up until now, the new secret was saved with the AWSPENDING staging label. This step is about changing the new secret label to AWSCURRENT, and the old secret to AWSPREVIOUS staging label. In the case of generating keys to external API, the process can look as presented below. 4 steps in secrets rotation using AWS Lambda

Expiring old secrets

To keep the description clean, I’ve skipped the process of expiring previous secrets from the above description, but this has to be implemented. Without expiring old API keys, they could still be used by the bad actor. You have to decide in which step to expire old API keys. Doing this in the last step can cause issues for the application that is using the API key, as it needs to fetch a new secret version from the Secret Manager when the old one gets expired. Expiring the previous secret in the first step would mean expiring it on the next rotation, increasing the time when the old API key is still valid. If it’s supported by the provider, probably the best option is to schedule the secret expiration in Step 4. This will give your application time to switch to a new secret and shorten the time when both secrets are valid.

Summary

AWS Secrets Manager provides quite an advanced and flexible way to rotate custom secrets. As we’ve seen, this process has a few disadvantages and limitations. You have to create and manage a custom AWS Lambda function. The External API needs to support the process of generating new keys by API. There are also concerns about expiring previous keys and managing the master key. As you know from the previous article, rotating secrets is often a must-have. Taking into account the disadvantages and limitations of rotating custom secrets by AWS Lambda, still, it seems to be a process more reliable than manual secrets rotation. Automatization is worth the effort.