AWS Lambda Layers: Share Code, Reduce Bundle Size and Speed Up Deploys

What Are Lambda Layers and Why Do They Exist?

A Lambda deployment package is the ZIP archive (or container image) that AWS unpacks into the /var/task directory before invoking your handler. Lambda enforces hard size limits: 50 MB zipped for direct upload, 250 MB unzipped, and 512 MB of ephemeral /tmp storage. Those ceilings were designed for simple handlers — but modern serverless applications routinely pull in machine-learning libraries, HTTP clients, telemetry SDKs, and internal business logic utilities that collectively balloon well past the limit.

Lambda Layers are an independent ZIP archive that Lambda extracts into /opt before your handler runs. Your function's deployment package contains only your application code; it references the layer by Amazon Resource Name (ARN). The layer content is cached in the execution environment and shared across all functions that reference the same version — Lambda downloads the layer once per cold start and reuses it across warm invocations.

The concrete benefits are:

• Smaller deployment packages — a typical Python data-science function shrinks from 90 MB to under 2 MB when numpy/pandas live in a layer.
• Faster CI/CD — your pipeline only pushes the small function zip; the large dependency layer uploads once and is referenced thereafter.
• Shared dependencies with a single source of truth — 20 functions all reference layer version 7; patching a CVE means publishing version 8 and updating the ARN in one place.
• Separation of concerns — infrastructure teams own the layer (correct Python version, vetted package hashes); application teams own the function code.
• Custom runtimes — Lambda's built-in runtimes cover Python, Node, Java, Go, .NET, and Ruby. If you need Rust, PHP, or Bun, you package the runtime executable as a layer and attach it to any function.

Layers are versioned and immutable. Each publish-layer-version call increments a version number and returns a new ARN. Old versions remain available until you delete them, so you can always pin a function to a known-good version. There is no concept of a "latest" alias for layers — functions always reference an explicit version number in the ARN, which is intentional: it makes infrastructure-as-code diffs meaningful and prevents accidental layer updates from breaking production.

Layer Directory Structure by Runtime

Lambda is strict about where it looks for layer content. Each runtime maps /opt subdirectories to its native module search paths. If you put a Python package in the wrong folder, the import simply fails at runtime with no helpful error message. Here is the authoritative layout for every supported runtime:

# Python 3.x — packages must be in:
layer.zip
└── python/
    └── lib/
        └── python3.12/
            └── site-packages/
                ├── numpy/
                ├── pandas/
                └── ...

# Shorthand that also works (Lambda searches both):
layer.zip
└── python/
    ├── numpy/
    └── pandas/

# Node.js — node_modules at /opt/nodejs/node_modules:
layer.zip
└── nodejs/
    └── node_modules/
        ├── axios/
        └── lodash/

# Java — JARs at /opt/java/lib:
layer.zip
└── java/
    └── lib/
        ├── commons-lang3-3.14.0.jar
        └── jackson-databind-2.17.1.jar

# Ruby — gems at /opt/ruby/gems/:
layer.zip
└── ruby/
    └── gems/
        └── 3.2.0/
            └── gems/
                └── faraday-2.9.0/

# Custom runtime — executables and shared libs at /opt/bin or /opt/lib:
layer.zip
├── bin/
│   └── bootstrap          ← the runtime bootstrap executable
└── lib/
    └── libssl.so.3        ← shared libs the bootstrap links against

When Lambda initializes an execution environment it extracts each attached layer into /opt in the order they are listed (layer 1 first, layer 5 last). If two layers contain the same path, the later layer wins — a useful override mechanism for patching a single file in a large shared layer without rebuilding the whole thing.

You can verify the final /opt layout from inside a function by logging os.listdir('/opt') or sys.path during a test invocation. This is the fastest way to diagnose "module not found" errors that appear only in Lambda and not in local development.

Creating Layers: Python, Node.js, and Java

The golden rule of Lambda layer packaging: build on the same OS and architecture as the Lambda execution environment. Lambda runs on Amazon Linux 2 (x86_64 or arm64). If you build a Python layer on macOS, C-extension packages like numpy will contain macOS binaries and will crash at runtime. Use Docker or an EC2 Amazon Linux instance for all production layer builds.

Python Layer (pandas + numpy)

#!/bin/bash
# build-python-layer.sh
# Builds a Python 3.12 Lambda layer containing pandas and numpy
# Requires: Docker

set -euo pipefail

LAYER_NAME="data-science-layer"
PYTHON_VERSION="3.12"
OUTPUT_ZIP="layer-python-data-science.zip"

# Use the official Lambda build image — matches production OS exactly
docker run --rm \
  -v "$(pwd)":/build \
  -w /build \
  public.ecr.aws/sam/build-python${PYTHON_VERSION}:latest \
  bash -c "
    pip install \
      pandas==2.2.2 \
      numpy==1.26.4 \
      pyarrow==16.1.0 \
      -t python/lib/python${PYTHON_VERSION}/site-packages/ \
      --no-deps \
      --platform manylinux2014_x86_64 \
      --only-binary=:all: \
      --implementation cp \
      --python-version ${PYTHON_VERSION}
  "

# Zip the python/ directory (not the build dir itself)
zip -r9 "${OUTPUT_ZIP}" python/

# Print final size
echo "Layer zip size: $(du -sh ${OUTPUT_ZIP} | cut -f1)"
echo "Unzipped size:  $(du -sh python/ | cut -f1)"

# Clean up
rm -rf python/

# Alternative: use pip's --target with a requirements file
# requirements-layer.txt
pandas==2.2.2
numpy==1.26.4
pyarrow==16.1.0
scikit-learn==1.5.0

# Build using pip directly on Amazon Linux (e.g., in CodeBuild or EC2)
mkdir -p python/lib/python3.12/site-packages
pip install \
  -r requirements-layer.txt \
  -t python/lib/python3.12/site-packages/ \
  --no-cache-dir

zip -r9 layer-python-data-science.zip python/

Node.js Layer (axios + lodash + zod)

#!/bin/bash
# build-nodejs-layer.sh

mkdir -p nodejs/node_modules

# Install production dependencies only
cd nodejs
npm init -y
npm install axios@1.7.2 lodash@4.17.21 zod@3.23.8 --omit=dev
cd ..

# Remove dev artifacts and shrink
rm -rf nodejs/package.json nodejs/package-lock.json nodejs/node_modules/.package-lock.json

# Zip
zip -r9 layer-nodejs-utils.zip nodejs/

echo "Layer size: $(du -sh layer-nodejs-utils.zip | cut -f1)"
rm -rf nodejs/

// package.json for the layer (commit this to version control)
{
  "name": "lambda-utils-layer",
  "version": "1.0.0",
  "description": "Shared utilities for Lambda functions",
  "dependencies": {
    "axios": "^1.7.2",
    "lodash": "^4.17.21",
    "zod": "^3.23.8"
  },
  "engines": {
    "node": ">=20.0.0"
  }
}

Java Layer (commons-lang3 + Jackson)

#!/bin/bash
# build-java-layer.sh

mkdir -p java/lib

# Download JARs (use Maven or Gradle in real pipelines)
mvn dependency:copy \
  -Dartifact=org.apache.commons:commons-lang3:3.14.0:jar \
  -DoutputDirectory=java/lib/

mvn dependency:copy \
  -Dartifact=com.fasterxml.jackson.core:jackson-databind:2.17.1:jar \
  -DoutputDirectory=java/lib/

mvn dependency:copy \
  -Dartifact=com.fasterxml.jackson.core:jackson-core:2.17.1:jar \
  -DoutputDirectory=java/lib/

zip -r9 layer-java-commons.zip java/

echo "Layer size: $(du -sh layer-java-commons.zip | cut -f1)"

Publishing, Versioning, and Cross-Account Sharing

Publishing a layer creates a new immutable version and returns a version-specific ARN. The ARN format is always:

arn:aws:lambda:{region}:{account-id}:layer:{layer-name}:{version-number}

Publishing via AWS CLI

# Publish a new layer version
aws lambda publish-layer-version \
  --layer-name "data-science-layer" \
  --description "pandas 2.2.2 + numpy 1.26.4 + pyarrow 16.1.0 for Python 3.12" \
  --license-info "MIT" \
  --compatible-runtimes python3.12 python3.11 \
  --compatible-architectures x86_64 \
  --zip-file fileb://layer-python-data-science.zip \
  --region us-east-1

# Sample output:
# {
#   "LayerArn": "arn:aws:lambda:us-east-1:123456789012:layer:data-science-layer",
#   "LayerVersionArn": "arn:aws:lambda:us-east-1:123456789012:layer:data-science-layer:7",
#   "Version": 7,
#   "Description": "pandas 2.2.2 + numpy 1.26.4 + pyarrow 16.1.0 for Python 3.12",
#   "CreatedDate": "2026-06-08T10:23:45.000+0000"
# }

# List all versions of a layer
aws lambda list-layer-versions \
  --layer-name "data-science-layer" \
  --query 'LayerVersions[*].[Version,Description,CreatedDate]' \
  --output table

# Get specific version details
aws lambda get-layer-version \
  --layer-name "data-science-layer" \
  --version-number 7

Cross-Account Layer Sharing

Layers can be shared with specific AWS accounts, all accounts in your AWS Organization, or made public. Sharing is controlled by a resource-based policy attached to the layer version.

# Share with a specific account
aws lambda add-layer-version-permission \
  --layer-name "data-science-layer" \
  --version-number 7 \
  --statement-id "share-with-prod-account" \
  --action lambda:GetLayerVersion \
  --principal "987654321098"

# Share with your entire AWS Organization
aws lambda add-layer-version-permission \
  --layer-name "data-science-layer" \
  --version-number 7 \
  --statement-id "share-with-org" \
  --action lambda:GetLayerVersion \
  --principal "*" \
  --organization-id "o-abc123def456"

# Make a layer public (use with caution — open to all AWS accounts)
aws lambda add-layer-version-permission \
  --layer-name "data-science-layer" \
  --version-number 7 \
  --statement-id "public-access" \
  --action lambda:GetLayerVersion \
  --principal "*"

# Revoke a permission
aws lambda remove-layer-version-permission \
  --layer-name "data-science-layer" \
  --version-number 7 \
  --statement-id "share-with-prod-account"

# Delete old layer versions to avoid clutter (after migrating all functions)
aws lambda delete-layer-version \
  --layer-name "data-science-layer" \
  --version-number 5

Attaching Layers: Console, CLI, SAM, and Terraform

Once a layer is published you attach it to a function by including the layer version ARN in the function configuration. The approach differs depending on your toolchain.

AWS CLI

# Attach a single layer
aws lambda update-function-configuration \
  --function-name "my-data-function" \
  --layers "arn:aws:lambda:us-east-1:123456789012:layer:data-science-layer:7"

# Attach multiple layers (order matters — later layers override earlier ones for same paths)
aws lambda update-function-configuration \
  --function-name "my-data-function" \
  --layers \
    "arn:aws:lambda:us-east-1:123456789012:layer:data-science-layer:7" \
    "arn:aws:lambda:us-east-1:123456789012:layer:internal-utils:3" \
    "arn:aws:lambda:us-east-1:638966207318:layer:AWSLambdaPowertoolsPythonV3-python312-x86_64:7"

# Remove all layers (pass empty list)
aws lambda update-function-configuration \
  --function-name "my-data-function" \
  --layers

AWS SAM Template

# template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Globals:
  Function:
    Runtime: python3.12
    Architectures: [x86_64]
    Environment:
      Variables:
        LOG_LEVEL: INFO
        POWERTOOLS_SERVICE_NAME: !Sub "${AWS::StackName}"

Resources:
  # Define the layer
  DataScienceLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: data-science-layer
      Description: "pandas + numpy + pyarrow"
      ContentUri: layers/data-science/
      CompatibleRuntimes:
        - python3.12
      RetentionPolicy: Retain  # Keep old versions when stack updates
    Metadata:
      BuildMethod: python3.12   # SAM builds this layer using pip

  # Attach the layer to a function
  DataProcessorFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: data-processor
      CodeUri: src/data-processor/
      Handler: handler.lambda_handler
      MemorySize: 1024
      Timeout: 300
      Layers:
        - !Ref DataScienceLayer
        - "arn:aws:lambda:us-east-1:638966207318:layer:AWSLambdaPowertoolsPythonV3-python312-x86_64:7"
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref ProcessingTable

Terraform

# layers.tf

# Build and publish the layer
resource "aws_lambda_layer_version" "data_science" {
  layer_name          = "data-science-layer"
  filename            = "${path.module}/artifacts/layer-python-data-science.zip"
  source_code_hash    = filebase64sha256("${path.module}/artifacts/layer-python-data-science.zip")
  compatible_runtimes = ["python3.12", "python3.11"]
  compatible_architectures = ["x86_64"]
  description         = "pandas 2.2.2 + numpy 1.26.4 + pyarrow 16.1.0"
  license_info        = "MIT"
}

resource "aws_lambda_layer_version" "internal_utils" {
  layer_name       = "internal-utils"
  filename         = "${path.module}/artifacts/layer-internal-utils.zip"
  source_code_hash = filebase64sha256("${path.module}/artifacts/layer-internal-utils.zip")
  compatible_runtimes = ["python3.12"]
  description      = "Company shared utilities v3"
}

# Lambda function that references the layers
resource "aws_lambda_function" "data_processor" {
  function_name    = "data-processor"
  filename         = "${path.module}/artifacts/data-processor.zip"
  source_code_hash = filebase64sha256("${path.module}/artifacts/data-processor.zip")
  handler          = "handler.lambda_handler"
  runtime          = "python3.12"
  memory_size      = 1024
  timeout          = 300
  role             = aws_iam_role.lambda_exec.arn

  layers = [
    aws_lambda_layer_version.data_science.arn,
    aws_lambda_layer_version.internal_utils.arn,
    "arn:aws:lambda:us-east-1:638966207318:layer:AWSLambdaPowertoolsPythonV3-python312-x86_64:7"
  ]

  environment {
    variables = {
      LOG_LEVEL                = "INFO"
      POWERTOOLS_SERVICE_NAME  = "data-processor"
    }
  }

  tracing_config {
    mode = "Active"  # Required for PowerTools X-Ray tracing
  }
}

# Cross-account sharing permission
resource "aws_lambda_layer_version_permission" "share_with_prod" {
  layer_name     = aws_lambda_layer_version.data_science.layer_arn
  version_number = aws_lambda_layer_version.data_science.version
  statement_id   = "share-with-prod"
  action         = "lambda:GetLayerVersion"
  principal      = var.prod_account_id
}

Custom Runtimes via Layers

Lambda's custom runtime interface lets you run any language that can compile to a Linux binary. The runtime communicates with Lambda via a simple HTTP API at http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next. You package your runtime as a layer with a bootstrap executable in /opt/bin/ (or in your function's root directory).

Minimal Bash Custom Runtime (for illustration)

#!/bin/bash
# bootstrap — the Lambda custom runtime entry point
# Must be in /var/task/ (function zip) or /opt/bin/ (layer)
# Must be executable: chmod +x bootstrap

set -euo pipefail

# Lambda Runtime API base URL
RUNTIME_API="http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime"

# Source the handler file (set by the HANDLER env var or Lambda config)
source "${LAMBDA_TASK_ROOT}/${_HANDLER%%.*}.sh"
HANDLER_FUNCTION="${_HANDLER##*.}"

# Main polling loop
while true; do
  # 1. Get next invocation
  RESPONSE=$(curl -sS -D /tmp/headers.txt \
    "${RUNTIME_API}/invocation/next")

  REQUEST_ID=$(grep -i "lambda-runtime-aws-request-id" /tmp/headers.txt | tr -d '[:space:]' | cut -d: -f2)

  # 2. Call the handler
  RESULT=$(${HANDLER_FUNCTION} "${RESPONSE}" 2>&1) || {
    # 3a. Post an error response
    curl -sS -X POST \
      "${RUNTIME_API}/invocation/${REQUEST_ID}/error" \
      -H "Content-Type: application/json" \
      -d "{\"errorMessage\":\"${RESULT}\",\"errorType\":\"RuntimeError\"}"
    continue
  }

  # 3b. Post a successful response
  curl -sS -X POST \
    "${RUNTIME_API}/invocation/${REQUEST_ID}/response" \
    -H "Content-Type: application/json" \
    -d "${RESULT}"
done

Rust Runtime Layer (using the official aws-lambda-rust-runtime)

# Cargo.toml for the Rust Lambda function
[package]
name = "rust-lambda-function"
version = "0.1.0"
edition = "2021"

[[bin]]
name = "bootstrap"
path = "src/main.rs"

[dependencies]
lambda_runtime = "0.11"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tokio = { version = "1", features = ["full"] }

// src/main.rs — Rust Lambda handler
use lambda_runtime::{run, service_fn, Error, LambdaEvent};
use serde::{Deserialize, Serialize};

#[derive(Deserialize)]
struct Request {
    message: String,
}

#[derive(Serialize)]
struct Response {
    processed: String,
    length: usize,
}

async fn function_handler(event: LambdaEvent) -> Result {
    let processed = event.payload.message.to_uppercase();
    let length = processed.len();
    Ok(Response { processed, length })
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    run(service_fn(function_handler)).await
}

# Build the Rust binary for Amazon Linux (arm64 or x86_64)
# Install the target: rustup target add aarch64-unknown-linux-musl
cargo lambda build --release --arm64

# The binary is at target/lambda/rust-lambda-function/bootstrap
# Package as a function zip (no layer needed for Rust — the binary IS the runtime)
cp target/lambda/rust-lambda-function/bootstrap .
zip function.zip bootstrap

# Deploy as a custom runtime function
aws lambda create-function \
  --function-name rust-processor \
  --runtime provided.al2023 \
  --architectures arm64 \
  --handler bootstrap \
  --role arn:aws:iam::123456789012:role/lambda-exec-role \
  --zip-file fileb://function.zip

Lambda PowerTools Deep-Dive

AWS Lambda Powertools is an open-source developer toolkit that implements structured logging, distributed tracing, custom metrics, idempotency, batch processing, and more — all production-hardened, with zero-overhead when disabled. AWS publishes official PowerTools layers for Python and Java so you never have to build and manage them yourself.

Finding the Official PowerTools Layer ARNs

# List all official PowerTools layers in your region
aws lambda list-layers \
  --query "Layers[?contains(LayerName,'AWSLambdaPowertools')].[LayerName,LatestMatchingVersion.LayerVersionArn]" \
  --output table \
  --region us-east-1

# Get the latest Python 3.12 x86_64 PowerTools layer ARN
aws ssm get-parameter \
  --name "/aws/service/powertools/python/3.12/x86_64/latest" \
  --region us-east-1 \
  --query Parameter.Value \
  --output text

Structured Logging

# handler.py — using PowerTools Logger
import json
from aws_lambda_powertools import Logger, Tracer, Metrics
from aws_lambda_powertools.metrics import MetricUnit
from aws_lambda_powertools.utilities.typing import LambdaContext

logger = Logger(service="order-processor", level="INFO")
tracer = Tracer(service="order-processor")
metrics = Metrics(namespace="OrderService", service="order-processor")

@logger.inject_lambda_context(correlation_id_path="headers.x-correlation-id")
@tracer.capture_lambda_handler
@metrics.log_metrics(capture_cold_start_metric=True)
def lambda_handler(event: dict, context: LambdaContext) -> dict:
    order_id = event.get("orderId")

    # Structured log — every field is queryable in CloudWatch Logs Insights
    logger.info("Processing order", extra={
        "orderId": order_id,
        "customerId": event.get("customerId"),
        "itemCount": len(event.get("items", []))
    })

    # Add custom dimensions to all subsequent metrics
    metrics.add_dimension(name="Environment", value="prod")

    try:
        result = process_order(order_id, event["items"])

        # Record a custom metric
        metrics.add_metric(name="OrderProcessed", unit=MetricUnit.Count, value=1)
        metrics.add_metric(name="OrderValue", unit=MetricUnit.Count, value=result["totalValue"])

        logger.info("Order processed successfully", extra={"result": result})
        return {"statusCode": 200, "body": json.dumps(result)}

    except Exception as e:
        logger.exception("Order processing failed", extra={"orderId": order_id})
        metrics.add_metric(name="OrderFailed", unit=MetricUnit.Count, value=1)
        raise

@tracer.capture_method
def process_order(order_id: str, items: list) -> dict:
    # This method appears as a subsegment in X-Ray traces
    total = sum(item["price"] * item["quantity"] for item in items)
    return {"orderId": order_id, "totalValue": total, "itemCount": len(items)}

Idempotency with DynamoDB

# Idempotency — ensure exactly-once processing for SQS or API Gateway events
from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer,
    IdempotencyConfig,
    idempotent_function
)

# DynamoDB table for idempotency store (create with TTL enabled on 'expiration' attribute)
persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyStore")

idempotency_config = IdempotencyConfig(
    event_key_jmespath="body",           # use request body as idempotency key
    raise_on_no_idempotency_key=True,
    expires_after_seconds=3600           # deduplicate for 1 hour
)

@idempotent_function(
    data_keyword_argument="order",
    config=idempotency_config,
    persistence_store=persistence_layer
)
def create_order(order: dict) -> dict:
    # This function is safe to retry — duplicate calls with same order body
    # return the cached result without re-executing
    order_id = str(uuid.uuid4())
    # ... charge card, update inventory, send email ...
    return {"orderId": order_id, "status": "CREATED"}

Batch Processing with Partial Failure

# Process SQS batch records with automatic partial failure reporting
from aws_lambda_powertools.utilities.batch import (
    BatchProcessor, EventType, process_partial_response
)
from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord

processor = BatchProcessor(event_type=EventType.SQS)

def record_handler(record: SQSRecord):
    payload = record.json_body
    logger.info("Processing record", extra={"messageId": record.message_id})

    # Raise an exception to mark this specific record as failed
    # PowerTools automatically builds the batchItemFailures response
    if payload.get("invalid"):
        raise ValueError(f"Invalid payload in record {record.message_id}")

    # Process the valid record
    return save_to_dynamodb(payload)

@logger.inject_lambda_context
@tracer.capture_lambda_handler
def lambda_handler(event: dict, context: LambdaContext) -> dict:
    return process_partial_response(
        event=event,
        record_handler=record_handler,
        processor=processor,
        context=context
    )
    # Returns: {"batchItemFailures": [{"itemIdentifier": "msg-id-of-failed-record"}]}

PowerTools for Java

// Java Lambda with PowerTools annotations
import software.amazon.lambda.powertools.logging.Logging;
import software.amazon.lambda.powertools.metrics.Metrics;
import software.amazon.lambda.powertools.tracing.Tracing;
import software.amazon.lambda.powertools.logging.LoggingUtils;
import software.amazon.cloudwatchlogs.emf.model.Unit;

public class OrderHandler implements RequestHandler, Map> {

    @Logging(logEvent = true)   // logs the full incoming event as structured JSON
    @Tracing                    // creates X-Ray subsegment per method
    @Metrics(namespace = "OrderService", service = "order-handler", captureColdStart = true)
    @Override
    public Map handleRequest(Map event, Context context) {
        String orderId = (String) event.get("orderId");

        LoggingUtils.appendKey("orderId", orderId);  // add to all subsequent log messages

        MetricsUtils.addMetric("OrderReceived", 1, Unit.COUNT);

        Map result = processOrder(orderId, event);

        MetricsUtils.addMetric("OrderProcessed", 1, Unit.COUNT);
        return result;
    }

    @Tracing(segmentName = "##processOrder")
    private Map processOrder(String orderId, Map event) {
        // business logic — appears as X-Ray subsegment
        return Map.of("orderId", orderId, "status", "PROCESSED");
    }
}

CI/CD Pipeline for Layers with GitHub Actions

A production layer CI/CD pipeline needs to: detect changes to the layer's dependency manifest, build on Amazon Linux, run a smoke test, publish the new layer version, update all downstream functions, and notify the team. Here is a complete GitHub Actions workflow that does all of this.

# .github/workflows/publish-lambda-layer.yml
name: Build and Publish Lambda Layer

on:
  push:
    paths:
      - 'layers/data-science/requirements.txt'
      - 'layers/data-science/build.sh'
      - '.github/workflows/publish-lambda-layer.yml'
  workflow_dispatch:
    inputs:
      layer_name:
        description: 'Layer to build (data-science / nodejs-utils / java-commons)'
        required: true
        default: 'data-science'

env:
  AWS_REGION: us-east-1
  PYTHON_VERSION: "3.12"

jobs:
  build-and-publish:
    name: Build Layer and Publish
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write     # Required for OIDC auth with AWS

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Configure AWS credentials (OIDC — no long-lived keys)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-actions-layer-publisher
          aws-region: ${{ env.AWS_REGION }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build layer using Lambda build image
        run: |
          docker run --rm \
            -v "${{ github.workspace }}/layers/data-science":/build \
            -w /build \
            public.ecr.aws/sam/build-python${{ env.PYTHON_VERSION }}:latest \
            bash build.sh

          ls -lh layers/data-science/layer-python-data-science.zip

      - name: Smoke test — verify imports work in Lambda environment
        run: |
          docker run --rm \
            -v "${{ github.workspace }}/layers/data-science":/layer \
            public.ecr.aws/lambda/python:${{ env.PYTHON_VERSION }} \
            bash -c "
              unzip -q /layer/layer-python-data-science.zip -d /opt
              python3 -c '
                import sys
                sys.path.insert(0, \"/opt/python\")
                import numpy as np
                import pandas as pd
                import pyarrow as pa
                arr = np.array([1, 2, 3])
                df = pd.DataFrame({\"x\": arr})
                print(f\"numpy {np.__version__} OK\")
                print(f\"pandas {pd.__version__} OK\")
                print(f\"pyarrow {pa.__version__} OK\")
                assert len(df) == 3
                print(\"Smoke test PASSED\")
              '
            "

      - name: Publish layer version
        id: publish
        run: |
          LAYER_VERSION_ARN=$(aws lambda publish-layer-version \
            --layer-name "data-science-layer" \
            --description "pandas $(grep pandas layers/data-science/requirements.txt | cut -d= -f3) | commit ${{ github.sha }}" \
            --compatible-runtimes python${{ env.PYTHON_VERSION }} \
            --compatible-architectures x86_64 \
            --zip-file fileb://layers/data-science/layer-python-data-science.zip \
            --query LayerVersionArn \
            --output text)

          echo "layer_version_arn=${LAYER_VERSION_ARN}" >> "$GITHUB_OUTPUT"
          echo "Published: ${LAYER_VERSION_ARN}"

      - name: Update all functions using this layer
        run: |
          NEW_LAYER_ARN="${{ steps.publish.outputs.layer_version_arn }}"

          # Find all functions tagged with this layer
          FUNCTIONS=$(aws lambda list-functions \
            --query "Functions[?contains(Layers[].Arn, 'data-science-layer')].FunctionName" \
            --output text 2>/dev/null || echo "")

          if [ -z "${FUNCTIONS}" ]; then
            echo "No functions currently using this layer (first publish or none tagged)"
            exit 0
          fi

          for FUNCTION in ${FUNCTIONS}; do
            echo "Updating ${FUNCTION} to use ${NEW_LAYER_ARN}..."

            # Get current layers for this function, replace the old data-science layer
            CURRENT_LAYERS=$(aws lambda get-function-configuration \
              --function-name "${FUNCTION}" \
              --query "Layers[].Arn" \
              --output json)

            # Build new layers list with updated data-science layer ARN
            UPDATED_LAYERS=$(echo "${CURRENT_LAYERS}" | python3 -c "
              import sys, json
              layers = json.load(sys.stdin)
              new_layer = '${NEW_LAYER_ARN}'
              updated = [new_layer if 'data-science-layer' in l else l for l in layers]
              print(' '.join(updated))
            ")

            aws lambda update-function-configuration \
              --function-name "${FUNCTION}" \
              --layers ${UPDATED_LAYERS}

            echo "  -> ${FUNCTION} updated"
          done

      - name: Write layer ARN to SSM Parameter Store
        run: |
          aws ssm put-parameter \
            --name "/lambda/layers/data-science/latest-arn" \
            --value "${{ steps.publish.outputs.layer_version_arn }}" \
            --type String \
            --overwrite

      - name: Comment on commit with layer ARN
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.repos.createCommitComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              commit_sha: context.sha,
              body: `**Lambda Layer Published**\n\nARN: \`${{ steps.publish.outputs.layer_version_arn }}\`\n\nReference this ARN in your SAM/Terraform configs.`
            })

Best Practices and When NOT to Use Layers

Lambda Layers are a powerful primitive but are frequently misapplied. Here is what the most experienced serverless teams have learned from running layers in production at scale.

DO: Use Layers For

• Large, stable dependencies (numpy, pandas, OpenCV, ffmpeg) that rarely change and would otherwise dominate your function's zip size.
• Cross-function shared utilities — internal logging helpers, database clients, auth middleware used by 10+ functions.
• Compliance-vetted packages — security teams approve specific library versions; layers enforce the approved set across all functions.
• Custom runtimes — Rust, PHP, WASM, or any language Lambda doesn't support natively.
• AWS Lambda PowerTools — always use the AWS-managed layer; they patch it, not you.

DON'T: Use Layers For

• Application business logic — logic changes frequently; putting it in a layer makes deployments more complex, not simpler. Keep business logic in the function zip.
• Small dependencies (<1 MB) — the overhead of managing a layer version isn't worth it for a tiny package. Just bundle it.
• Per-function custom configurations — environment variables and SSM Parameter Store exist for this. Layers are for code, not config.
• Functions that are already container images — container images have their own layer caching mechanism (OCI layers). Lambda Layers don't apply to container image functions.

Layer Size Limits and Constraints

Lambda Layer Limits (per function invocation):
─────────────────────────────────────────────────────
Maximum layers per function:        5
Maximum unzipped size (all layers + function):  250 MB
Maximum zipped layer upload size:   50 MB (direct) / unlimited (S3)
Maximum layer description length:   256 characters
Layers cannot reference other layers
Layers are region-specific (copy to each region you deploy to)
Container image functions do NOT support layers
─────────────────────────────────────────────────────

Versioning Strategy

# Tag your layer versions with semantic version info in the description
# Use a naming convention in the layer name itself for major versions
# data-science-layer-v2 for a breaking change (Python 3.11 -> 3.12 upgrade)

# Always pin the EXACT version ARN in IaC — never resolve "latest" at deploy time
# BAD: resolve-layer-arn --layer-name data-science-layer --latest
# GOOD: arn:aws:lambda:us-east-1:123456789012:layer:data-science-layer:7

# Automate old version cleanup to avoid hitting the 75-layer-version limit per layer
aws lambda list-layer-versions \
  --layer-name "data-science-layer" \
  --query "LayerVersions[?Version<\`5\`].Version" \
  --output text | \
  xargs -I{} aws lambda delete-layer-version \
    --layer-name "data-science-layer" \
    --version-number {}

Multi-Region Layer Distribution

#!/bin/bash
# publish-layer-all-regions.sh
# Publish the same layer to all regions where you deploy functions

LAYER_NAME="data-science-layer"
LAYER_ZIP="layer-python-data-science.zip"
DESCRIPTION="pandas 2.2.2 + numpy 1.26.4 + pyarrow 16.1.0"
REGIONS=("us-east-1" "us-west-2" "eu-west-1" "ap-southeast-1")

declare -A LAYER_ARNS

for REGION in "${REGIONS[@]}"; do
  echo "Publishing to ${REGION}..."

  ARN=$(aws lambda publish-layer-version \
    --layer-name "${LAYER_NAME}" \
    --description "${DESCRIPTION}" \
    --compatible-runtimes python3.12 \
    --zip-file fileb://"${LAYER_ZIP}" \
    --region "${REGION}" \
    --query LayerVersionArn \
    --output text)

  LAYER_ARNS["${REGION}"]="${ARN}"
  echo "  ${REGION}: ${ARN}"
done

echo ""
echo "=== Layer ARNs by region ==="
for REGION in "${!LAYER_ARNS[@]}"; do
  echo "${REGION}: ${LAYER_ARNS[$REGION]}"
done

# Write to a JSON file for use by Terraform or CDK
python3 -c "
import json, sys
arns = dict(zip('${REGIONS[@]}'.split(), '${LAYER_ARNS[@]}'.split()))
print(json.dumps(arns, indent=2))
" > layer-arns-by-region.json

Frequently Asked Questions

Do Lambda Layers affect cold start time?

Layers are extracted to /opt during environment initialization — the same phase as your function's deployment package extraction. In practice, the difference between a 5 MB function zip + a 70 MB layer vs. a 75 MB function zip is negligible for cold start duration (both require the same amount of data to be extracted). What layers do improve is deploy time — your CI/CD pipeline only pushes the small function zip on most deploys, and layer extraction happens from AWS's internal network, not your VPC.

Can I update a Lambda Layer without redeploying my functions?

No — and this is by design. Layers are pinned by explicit version number in the function ARN. Publishing a new layer version does not automatically update any function. You must call update-function-configuration with the new layer ARN to adopt the update. This immutability guarantees that no function ever gets an unexpected change to its dependencies.

What happens when two layers contain the same file?

The later layer in the list wins. Lambda overlays layer content in order — layer 1 is extracted first, layer 5 last. If both layers contain /opt/python/utils.py, layer 5's version is the one that gets imported. This behavior enables a "patch layer" pattern: publish a small layer with only the changed file and put it last in the list, without rebuilding the large base layer.

Are layers shared across function invocations (execution environments)?

Yes — within a single execution environment, the layer content at /opt persists across multiple warm invocations. But each new execution environment (cold start) re-extracts the layers. There is no shared filesystem between different execution environments — layers are per-environment, not shared globally across concurrent invocations of the same function.

Can I use Lambda Layers with container image functions?

No. Lambda Layers are not supported for functions deployed as container images. Container images use OCI layer caching, which is conceptually similar but operates at the container registry level. For container-based functions, put your shared dependencies in a base Docker image and use FROM your-base-image:latest in your Dockerfile.

How do I find the ARN of AWS-published layers (PowerTools, etc.)?

AWS publishes PowerTools layer ARNs in the official documentation and via SSM Parameter Store. Query SSM for the canonical ARN: aws ssm get-parameter --name "/aws/service/powertools/python/3.12/x86_64/latest" --region us-east-1. You can also run aws lambda list-layers --compatible-runtime python3.12 to see all layers published by AWS in your region.