AWS Secrets Manager: Rotation and Cross-Account Access

Hardcoded database passwords in source code, API keys committed to GitHub, credentials rotated by hand every quarter — these are the practices that turn a minor breach into a catastrophic one. AWS Secrets Manager eliminates all three problems by providing a fully managed, audited, and rotation-capable secrets store that integrates natively with the AWS service ecosystem.

This guide covers everything you need to operate Secrets Manager in production: how secrets are versioned, how to configure automatic rotation with Lambda, how to grant cross-account access without sharing credentials, how to route Secrets Manager API calls through a VPC endpoint, and how to inject secrets into ECS tasks, Lambda functions, Spring Boot applications, and Kubernetes workloads.

1. Secrets Manager Overview

What Secrets Manager Stores

Secrets Manager is designed to store any piece of sensitive configuration that your applications need at runtime:

  • Database credentials — username/password pairs for RDS, Aurora, Redshift, DocumentDB, and self-managed databases
  • API keys and tokens — third-party service keys (Stripe, Twilio, SendGrid), OAuth 2.0 client secrets, JWT signing keys
  • SSH keys and TLS certificates — private keys for service-to-service mTLS, SSH bastion credentials
  • License keys and configuration blobs — any opaque binary or text value up to 65,536 bytes per version

Each secret is stored as a JSON string by convention (though any UTF-8 text is valid). The standard RDS secret format looks like:

{
  "username": "appuser",
  "password": "s3cr3tP@ssword!",
  "engine": "mysql",
  "host": "mydb.cluster-xyz.us-east-1.rds.amazonaws.com",
  "port": 3306,
  "dbname": "production"
}

Pricing

Secrets Manager charges on two dimensions:

  • $0.40 per secret per month — prorated to the hour. A secret that exists for 15 days costs roughly $0.20.
  • $0.05 per 10,000 API calls — GetSecretValue, PutSecretValue, RotateSecret, etc. The first 10,000 calls per month are free.
Cost tip: At scale, caching secrets in-process is both a performance and cost optimization. A Lambda function that retrieves its DB password once per cold start instead of once per invocation can reduce API call costs by 99% while also eliminating 15–30 ms of latency per request.

Secrets Manager vs SSM Parameter Store

Both services can store secrets. The right choice depends on your requirements:

Feature Secrets Manager SSM Parameter Store (SecureString)
Automatic rotation Built-in + custom Lambda Manual or custom-built
Cross-account access Resource-based policy Not natively supported
Versioning with labels AWSCURRENT / AWSPENDING / AWSPREVIOUS Numeric versions only
RDS native integration Managed rotators None
Pricing $0.40/secret/month Free (standard) / $0.05/advanced/month
Best for Credentials requiring rotation, cross-account Config values, feature flags, non-rotating keys

See the AWS SSM Parameter Store guide for a deeper comparison. The short rule: if a secret needs to rotate or needs cross-account access, use Secrets Manager. Everything else can use Parameter Store for free.

2. Creating and Retrieving Secrets

Creating a Secret via CLI

The create-secret command accepts either a string value or a binary value, and optionally a customer-managed KMS key for envelope encryption (the default is the AWS-managed key aws/secretsmanager).

# Create a new secret with a customer-managed KMS key
aws secretsmanager create-secret \
  --name "prod/myapp/db-credentials" \
  --description "RDS MySQL credentials for the production app tier" \
  --kms-key-id "arn:aws:kms:us-east-1:123456789012:key/mrk-abc123" \
  --secret-string '{
    "username": "appuser",
    "password": "InitialP@ss!",
    "engine": "mysql",
    "host": "prod-db.cluster-xyz.us-east-1.rds.amazonaws.com",
    "port": 3306,
    "dbname": "appdb"
  }' \
  --tags '[
    {"Key":"Environment","Value":"production"},
    {"Key":"Team","Value":"platform"},
    {"Key":"CostCenter","Value":"engineering"}
  ]'

# Update the secret value (creates a new version, moves AWSCURRENT label)
aws secretsmanager put-secret-value \
  --secret-id "prod/myapp/db-credentials" \
  --secret-string '{
    "username": "appuser",
    "password": "RotatedP@ss!NewVersion",
    "engine": "mysql",
    "host": "prod-db.cluster-xyz.us-east-1.rds.amazonaws.com",
    "port": 3306,
    "dbname": "appdb"
  }'

Retrieving Secrets

# Retrieve the current secret value
aws secretsmanager get-secret-value \
  --secret-id "prod/myapp/db-credentials" \
  --query SecretString \
  --output text | jq .

# Retrieve a specific version by staging label
aws secretsmanager get-secret-value \
  --secret-id "prod/myapp/db-credentials" \
  --version-stage AWSPREVIOUS

# List all versions and their staging labels
aws secretsmanager list-secret-version-ids \
  --secret-id "prod/myapp/db-credentials"

Secret Versioning and Staging Labels

Every put-secret-value call creates a new version identified by a UUID. Secrets Manager moves staging labels between versions to track their lifecycle:

  • AWSCURRENT — the live version that get-secret-value returns by default
  • AWSPENDING — a new version created during rotation but not yet validated
  • AWSPREVIOUS — the previous AWSCURRENT, retained for rollback during rotation

You can also define custom labels (e.g., STABLE, CANARY) and attach them to any version for blue/green credential strategies.

Retention: Secrets Manager retains up to 100 versions per secret. Old versions without any staging label are eventually garbage-collected, but you can explicitly delete a version by removing all its labels.

3. Automatic Rotation

Built-in Rotation for AWS Databases

For RDS, Aurora, Redshift, and DocumentDB, Secrets Manager provides managed rotation functions that you do not need to write or maintain. When you enable rotation in the console or via CLI, AWS automatically creates a Lambda function in your account from a published Serverless Application Repository (SAR) template.

# Enable automatic rotation for an RDS secret using the managed rotator
aws secretsmanager rotate-secret \
  --secret-id "prod/myapp/db-credentials" \
  --rotation-rules '{
    "AutomaticallyAfterDays": 30
  }' \
  --rotate-immediately \
  --hosted-rotation-lambda-type MySQLSingleUser

# For multi-user rotation (creates a second user, alternates between them)
aws secretsmanager rotate-secret \
  --secret-id "prod/myapp/db-credentials" \
  --rotation-rules '{"AutomaticallyAfterDays": 30}' \
  --hosted-rotation-lambda-type MySQLMultiUser

Single-user rotation updates the password for the existing database user. There is a brief window (typically under 1 second) where the old password is invalid and the new one has not yet been committed. Multi-user rotation eliminates this window by alternating between two users — a better choice for high-availability applications.

Custom Lambda Rotation Function

For non-AWS databases, third-party API keys, or any secret that requires custom logic, you implement a Lambda function that Secrets Manager invokes. The function must handle four lifecycle steps:

  • createSecret — generate a new secret value and store it with the AWSPENDING label
  • setSecret — apply the new value at the target system (e.g., call the database's ALTER USER command)
  • testSecret — validate that the AWSPENDING value actually works against the target system
  • finishSecret — move the AWSCURRENT label to the AWSPENDING version and AWSPREVIOUS to the old AWSCURRENT
import boto3
import json
import logging
import string
import secrets as secrets_lib

logger = logging.getLogger()
logger.setLevel(logging.INFO)

client = boto3.client("secretsmanager")


def lambda_handler(event, context):
    """Entry point — routes to the correct rotation step."""
    arn = event["SecretId"]
    token = event["ClientRequestToken"]
    step = event["Step"]

    # Validate the secret exists and the version token is valid
    metadata = client.describe_secret(SecretId=arn)
    if not metadata.get("RotationEnabled"):
        raise ValueError(f"Rotation not enabled for secret {arn}")

    versions = metadata.get("VersionIdsToStages", {})
    if token not in versions:
        raise ValueError(f"Token {token} not found in versions for {arn}")

    if "AWSCURRENT" in versions[token]:
        logger.info("Token is already AWSCURRENT — rotation already done")
        return
    if "AWSPENDING" not in versions[token]:
        raise ValueError(f"Token {token} is not AWSPENDING")

    if step == "createSecret":
        create_secret(arn, token)
    elif step == "setSecret":
        set_secret(arn, token)
    elif step == "testSecret":
        test_secret(arn, token)
    elif step == "finishSecret":
        finish_secret(arn, token)
    else:
        raise ValueError(f"Unknown rotation step: {step}")


def create_secret(arn, token):
    """Generate and store a new secret value under AWSPENDING."""
    try:
        # Check if AWSPENDING already exists (idempotency)
        client.get_secret_value(SecretId=arn, VersionStage="AWSPENDING",
                                VersionId=token)
        logger.info("AWSPENDING already exists — skipping createSecret")
        return
    except client.exceptions.ResourceNotFoundException:
        pass

    # Get current secret to copy non-password fields
    current = json.loads(
        client.get_secret_value(SecretId=arn, VersionStage="AWSCURRENT")["SecretString"]
    )

    # Generate a new strong password
    alphabet = string.ascii_letters + string.digits + "!@#%^&*"
    new_password = "".join(secrets_lib.choice(alphabet) for _ in range(32))
    current["password"] = new_password

    client.put_secret_value(
        SecretId=arn,
        ClientRequestToken=token,
        SecretString=json.dumps(current),
        VersionStages=["AWSPENDING"],
    )
    logger.info("Created AWSPENDING version for %s", arn)


def set_secret(arn, token):
    """Apply the AWSPENDING password to the actual database."""
    pending = json.loads(
        client.get_secret_value(SecretId=arn, VersionStage="AWSPENDING",
                                VersionId=token)["SecretString"]
    )
    current = json.loads(
        client.get_secret_value(SecretId=arn, VersionStage="AWSCURRENT")["SecretString"]
    )

    # Connect with current credentials and update the password
    import pymysql
    conn = pymysql.connect(
        host=current["host"],
        user=current["username"],
        password=current["password"],
        db=current["dbname"],
        port=int(current["port"]),
        connect_timeout=5,
    )
    with conn.cursor() as cur:
        cur.execute(
            "ALTER USER %s@'%%' IDENTIFIED BY %s",
            (pending["username"], pending["password"]),
        )
    conn.commit()
    conn.close()
    logger.info("Password updated in database for user %s", pending["username"])


def test_secret(arn, token):
    """Verify that the AWSPENDING credentials actually work."""
    pending = json.loads(
        client.get_secret_value(SecretId=arn, VersionStage="AWSPENDING",
                                VersionId=token)["SecretString"]
    )
    import pymysql
    conn = pymysql.connect(
        host=pending["host"],
        user=pending["username"],
        password=pending["password"],
        db=pending["dbname"],
        port=int(pending["port"]),
        connect_timeout=5,
    )
    with conn.cursor() as cur:
        cur.execute("SELECT 1")
    conn.close()
    logger.info("AWSPENDING credentials validated successfully")


def finish_secret(arn, token):
    """Promote AWSPENDING to AWSCURRENT."""
    metadata = client.describe_secret(SecretId=arn)
    current_version = None
    for vid, stages in metadata["VersionIdsToStages"].items():
        if "AWSCURRENT" in stages:
            current_version = vid
            break

    if current_version == token:
        logger.info("Already AWSCURRENT — nothing to do")
        return

    client.update_secret_version_stage(
        SecretId=arn,
        VersionStage="AWSCURRENT",
        MoveToVersionId=token,
        RemoveFromVersionId=current_version,
    )
    logger.info("Rotation complete. New AWSCURRENT version: %s", token)
Lambda permissions: The rotation Lambda needs secretsmanager:GetSecretValue, secretsmanager:PutSecretValue, secretsmanager:UpdateSecretVersionStage, and secretsmanager:DescribeSecret. It also needs network access to your database — either deploy it in the same VPC or use a VPC endpoint.

Rotation Schedule Options

Starting in 2023, Secrets Manager supports both day-based and cron-based schedules:

# Rotate every 30 days
aws secretsmanager rotate-secret \
  --secret-id "prod/myapp/db-credentials" \
  --rotation-rules '{"AutomaticallyAfterDays": 30}'

# Rotate every Sunday at 02:00 UTC using cron expression
aws secretsmanager rotate-secret \
  --secret-id "prod/myapp/db-credentials" \
  --rotation-rules '{"ScheduleExpression": "cron(0 2 ? * SUN *)"}'

4. Cross-Account Access

A common enterprise pattern is to store secrets in a central security account and allow workload accounts to read them without copying credentials. This requires three components: a resource-based policy on the secret, a KMS key policy that grants cross-account decrypt, and an IAM role in the workload account.

Resource-Based Policy on the Secret

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowWorkloadAccountRead",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::987654321098:role/AppServerRole",
          "arn:aws:iam::987654321098:root"
        ]
      },
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:PrincipalOrgID": "o-abc123def456"
        }
      }
    },
    {
      "Sid": "DenyNonOrgAccess",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "secretsmanager:*",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalOrgID": "o-abc123def456"
        }
      }
    }
  ]
}

Apply the policy with:

aws secretsmanager put-resource-policy \
  --secret-id "prod/shared/stripe-api-key" \
  --resource-policy file://cross-account-policy.json \
  --block-public-policy

KMS Key Policy for Cross-Account Decryption

Because the secret value is envelope-encrypted with a KMS key in the security account, the workload account's role must also have kms:Decrypt permission on that key. Add a statement to the KMS key policy in the security account:

{
  "Sid": "AllowCrossAccountDecrypt",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::987654321098:role/AppServerRole"
  },
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey"
  ],
  "Resource": "*"
}
Important: KMS cross-account access requires the grant in the key policy AND an IAM policy on the consuming role. Both must allow the action — least privilege is enforced at both layers. See the IAM Roles and Policies guide for the full permission model.

Consuming the Cross-Account Secret

import boto3
import json

# The workload account assumes a role that has been granted access
sts = boto3.client("sts")
assumed = sts.assume_role(
    RoleArn="arn:aws:iam::123456789012:role/CrossAccountSecretsReader",
    RoleSessionName="app-secrets-session",
    DurationSeconds=3600
)

creds = assumed["Credentials"]
sm_client = boto3.client(
    "secretsmanager",
    region_name="us-east-1",
    aws_access_key_id=creds["AccessKeyId"],
    aws_secret_access_key=creds["SecretAccessKey"],
    aws_session_token=creds["SessionToken"]
)

response = sm_client.get_secret_value(
    SecretId="arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/shared/stripe-api-key"
)
secret = json.loads(response["SecretString"])
print(f"Retrieved API key: {secret['api_key'][:8]}...")

5. VPC Endpoints for Secrets Manager

By default, Secrets Manager API calls traverse the public internet, which means Lambda functions or EC2 instances in a private subnet require a NAT Gateway — a $0.045/hour service that also charges per-GB data transfer. A Secrets Manager VPC interface endpoint eliminates this dependency and keeps all traffic on the AWS backbone.

Creating the Interface Endpoint

# Create an interface endpoint for Secrets Manager
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-0abc12345def67890 \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.us-east-1.secretsmanager \
  --subnet-ids subnet-0aaa111 subnet-0bbb222 subnet-0ccc333 \
  --security-group-ids sg-0endpoint123 \
  --private-dns-enabled \
  --tag-specifications 'ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=secretsmanager-endpoint},{Key=Environment,Value=production}]'

The --private-dns-enabled flag is critical: it overrides the public DNS hostname secretsmanager.us-east-1.amazonaws.com within your VPC so that existing code needs zero changes — the SDK automatically routes through the endpoint.

Endpoint Security Group

# Allow HTTPS inbound from the Lambda/EC2 security group
aws ec2 authorize-security-group-ingress \
  --group-id sg-0endpoint123 \
  --protocol tcp \
  --port 443 \
  --source-group sg-0lambda456

Restrict the endpoint policy to allow only your account or specific roles for an additional defense-in-depth layer. See the VPC networking guide for endpoint policy patterns and PrivateLink architecture.

Cost impact: Interface endpoints cost $0.01/AZ/hour plus $0.01/GB data processed. For applications making thousands of GetSecretValue calls per hour (without caching), this typically saves $30–$100/month compared to a NAT Gateway, while also improving latency by 5–20 ms.

6. Integration Patterns

Python boto3 with In-Process Caching

Never call get_secret_value on every request. Cache the secret in-process with a TTL slightly shorter than the rotation period:

import boto3
import json
import time
import threading

_cache_lock = threading.Lock()
_secret_cache: dict = {}
CACHE_TTL_SECONDS = 300  # 5 minutes — well within any rotation window


def get_secret(secret_id: str, region: str = "us-east-1") -> dict:
    """
    Return secret as a dict, using a thread-safe in-process cache.
    On cache miss or TTL expiry, fetches from Secrets Manager.
    """
    now = time.monotonic()
    with _cache_lock:
        entry = _secret_cache.get(secret_id)
        if entry and (now - entry["fetched_at"]) < CACHE_TTL_SECONDS:
            return entry["value"]

    # Cache miss — fetch from Secrets Manager
    client = boto3.client("secretsmanager", region_name=region)
    response = client.get_secret_value(SecretId=secret_id)
    value = json.loads(response["SecretString"])

    with _cache_lock:
        _secret_cache[secret_id] = {"value": value, "fetched_at": now}

    return value


# Usage in a Lambda handler or application startup
def get_db_connection():
    creds = get_secret("prod/myapp/db-credentials")
    import pymysql
    return pymysql.connect(
        host=creds["host"],
        user=creds["username"],
        password=creds["password"],
        db=creds["dbname"],
        port=int(creds["port"])
    )
AWS Secrets Manager Agent: AWS also provides a sidecar agent (available since 2024) that handles caching at the process boundary. It exposes a local HTTP server on 127.0.0.1:2773, eliminating the need for AWS SDK credentials in your application containers. Consider it for polyglot environments.

ECS Task Definition — secretOptions

ECS natively injects secrets as environment variables using the secrets key in the container definition. ECS fetches the value at task start time and injects it; your application code reads a plain environment variable. See the ECS and containers guide for the full task definition reference.

{
  "family": "myapp-task",
  "taskRoleArn": "arn:aws:iam::123456789012:role/ECSTaskRole",
  "executionRoleArn": "arn:aws:iam::123456789012:role/ECSExecutionRole",
  "networkMode": "awsvpc",
  "containerDefinitions": [
    {
      "name": "myapp",
      "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp:latest",
      "essential": true,
      "portMappings": [{"containerPort": 8080, "protocol": "tcp"}],
      "secrets": [
        {
          "name": "DB_PASSWORD",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/myapp/db-credentials:password::"
        },
        {
          "name": "DB_HOST",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/myapp/db-credentials:host::"
        },
        {
          "name": "STRIPE_API_KEY",
          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/myapp/stripe-key"
        }
      ],
      "environment": [
        {"name": "SPRING_PROFILES_ACTIVE", "value": "production"},
        {"name": "SERVER_PORT", "value": "8080"}
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/myapp",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ],
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "512",
  "memory": "1024"
}

The valueFrom ARN supports a JSON key suffix (:password::) to extract a single field from a JSON secret, so you do not need to parse JSON inside your application for individual fields.

The ECS execution role (not the task role) must have permission to call secretsmanager:GetSecretValue and kms:Decrypt during task startup. Add this to the execution role policy:

{
  "Effect": "Allow",
  "Action": [
    "secretsmanager:GetSecretValue",
    "kms:Decrypt"
  ],
  "Resource": [
    "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/myapp/*",
    "arn:aws:kms:us-east-1:123456789012:key/mrk-abc123"
  ]
}

Lambda Environment Variable Injection

Lambda supports the same secretsmanager ARN syntax in environment variable definitions when you use the SSM/Secrets extension layer. Alternatively, use the AWS Parameters and Secrets Lambda Extension (layer ARN varies by region):

aws lambda update-function-configuration \
  --function-name my-api-function \
  --layers "arn:aws:lambda:us-east-1:177933569100:layer:AWS-Parameters-and-Secrets-Lambda-Extension:11" \
  --environment '{
    "Variables": {
      "PARAMETERS_SECRETS_EXTENSION_CACHE_ENABLED": "true",
      "PARAMETERS_SECRETS_EXTENSION_CACHE_SIZE": "1000",
      "PARAMETERS_SECRETS_EXTENSION_HTTP_PORT": "2773",
      "PARAMETERS_SECRETS_EXTENSION_LOG_LEVEL": "warn",
      "PARAMETERS_SECRETS_EXTENSION_MAX_CONNECTIONS": "3",
      "SECRETS_MANAGER_TTL": "300"
    }
  }'

With the extension running, your Lambda function fetches secrets via a local HTTP call — no SDK or IAM credential setup required inside your function code:

import os
import json
import urllib.request

def get_secret_via_extension(secret_id: str) -> dict:
    """Fetch secret from the local extension cache (no AWS SDK needed)."""
    port = os.environ.get("PARAMETERS_SECRETS_EXTENSION_HTTP_PORT", "2773")
    token = os.environ["AWS_SESSION_TOKEN"]
    url = f"http://localhost:{port}/secretsmanager/get?secretId={secret_id}"
    req = urllib.request.Request(url, headers={"X-Aws-Parameters-Secrets-Token": token})
    with urllib.request.urlopen(req) as resp:
        payload = json.loads(resp.read())
    return json.loads(payload["SecretString"])

Spring Boot Integration

The AWS SDK for Java v2 combined with Spring Cloud AWS makes secret injection straightforward. Add the dependency to your pom.xml:

<dependency>
    <groupId>io.awspring.cloud</groupId>
    <artifactId>spring-cloud-aws-secrets-manager-config</artifactId>
    <version>3.1.1</version>
</dependency>

Configure the bootstrap properties:

# application.properties
spring.cloud.aws.secretsmanager.enabled=true
spring.cloud.aws.secretsmanager.prefix=/secret
spring.cloud.aws.secretsmanager.default-context=application
spring.cloud.aws.secretsmanager.profile-separator=_
spring.cloud.aws.region.static=us-east-1

# Your app reads this as a property — Spring Cloud AWS resolves it automatically
# from the secret "prod/myapp/db-credentials" JSON key "password"
spring.datasource.password=${db.password}

Kubernetes ExternalSecrets Operator

For Kubernetes workloads, the External Secrets Operator (ESO) syncs Secrets Manager values into native Kubernetes Secrets on a configurable refresh interval:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-db-secret
  namespace: production
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: aws-secrets-manager-store
    kind: ClusterSecretStore
  target:
    name: myapp-db-credentials
    creationPolicy: Owner
    template:
      type: kubernetes.io/basic-auth
  data:
    - secretKey: username
      remoteRef:
        key: prod/myapp/db-credentials
        property: username
    - secretKey: password
      remoteRef:
        key: prod/myapp/db-credentials
        property: password

7. Best Practices

Secret Naming Conventions

A consistent naming scheme makes IAM policies and cost allocation far easier. Recommended pattern:

{environment}/{service}/{purpose}

Examples:
  prod/payments-api/stripe-secret-key
  prod/payments-api/db-credentials
  staging/user-service/jwt-signing-key
  dev/shared/internal-ca-cert

This structure lets you write IAM policy conditions like "secretsmanager:SecretId": "prod/payments-api/*" to restrict a role to only its own secrets.

Tagging Strategy

Tag every secret consistently for cost allocation and automation:

aws secretsmanager tag-resource \
  --secret-id "prod/myapp/db-credentials" \
  --tags '[
    {"Key":"Environment","Value":"production"},
    {"Key":"Service","Value":"myapp"},
    {"Key":"Owner","Value":"platform-team"},
    {"Key":"RotationEnabled","Value":"true"},
    {"Key":"CostCenter","Value":"CC-1042"},
    {"Key":"DataClassification","Value":"confidential"}
  ]'

Auditing with CloudTrail

All Secrets Manager API calls are recorded in CloudTrail as management events. Key events to monitor:

  • GetSecretValue — who is reading secrets, from which IP/role
  • PutSecretValue / UpdateSecret — unauthorized modifications
  • DeleteSecret — accidental or malicious deletion
  • RotateSecret — rotation invocations and their outcomes

Create a CloudWatch metric filter to alert on unexpected GetSecretValue calls from outside your VPC CIDR range:

aws logs put-metric-filter \
  --log-group-name "aws-cloudtrail-logs" \
  --filter-name "SecretsManagerExternalAccess" \
  --filter-pattern '{
    ($.eventSource = "secretsmanager.amazonaws.com") &&
    ($.eventName = "GetSecretValue") &&
    ($.sourceIPAddress != "10.*")
  }' \
  --metric-transformations \
    metricName=ExternalSecretsAccess,metricNamespace=Security/SecretsManager,metricValue=1

Alerting on Rotation Failures

Secrets Manager publishes rotation result events to CloudWatch Events (EventBridge). Create a rule to notify your on-call channel when rotation fails:

aws events put-rule \
  --name "SecretsManagerRotationFailed" \
  --event-pattern '{
    "source": ["aws.secretsmanager"],
    "detail-type": ["AWS API Call via CloudTrail"],
    "detail": {
      "eventName": ["RotationFailed"]
    }
  }' \
  --state ENABLED

# Target the rule to an SNS topic that triggers a PagerDuty/Slack webhook
aws events put-targets \
  --rule "SecretsManagerRotationFailed" \
  --targets '[{
    "Id": "notify-oncall",
    "Arn": "arn:aws:sns:us-east-1:123456789012:oncall-alerts"
  }]'

Additional Best Practices Summary

  • Always specify a customer-managed KMS key (--kms-key-id) for sensitive secrets — the default aws/secretsmanager key cannot be used for cross-account access
  • Enable --block-public-policy on all put-resource-policy calls to prevent accidental public exposure
  • Set a deletion window of 7–30 days (aws secretsmanager delete-secret --recovery-window-in-days 30) rather than immediate deletion to guard against accidents
  • Use the AWS security best practices framework to run periodic AWS Config rules that check for secrets lacking rotation or missing tags
  • Never log the raw output of get_secret_value — even in debug mode, rotate immediately if you suspect a secret has been logged
  • For secrets accessed by hundreds of Lambda cold starts simultaneously, consider the Secrets Manager Agent or the Parameters and Secrets Lambda Extension to distribute the caching burden across all concurrent execution environments

Summary

AWS Secrets Manager provides a production-grade foundation for secrets hygiene in AWS workloads. The key operational concepts covered in this guide are:

  • Versioning with staging labels — AWSCURRENT, AWSPENDING, and AWSPREVIOUS provide a clean rollback model during rotation
  • Automatic rotation — managed rotators for RDS/Aurora/Redshift; four-step Lambda protocol for any other target
  • Cross-account access — resource-based policy on the secret + KMS key policy grant; no credential sharing required
  • VPC endpoints — eliminate NAT Gateway dependency and keep traffic off the public internet
  • Integration patterns — ECS secretOptions, Lambda extension layer, Spring Cloud AWS, Kubernetes ESO
  • Operational controls — naming conventions, CloudTrail auditing, EventBridge rotation failure alerts

Combined with fine-grained IAM policies and a VPC endpoint, Secrets Manager eliminates the two most common credential security failures — hardcoded secrets and stale passwords — with minimal application code changes.

Related Articles

Stay Updated

Get the latest AWS and cloud engineering articles delivered to your inbox.