DEVOPS TRIBUNE ENGINEERING INTELLIGENCE FOR THE REAL WORLD
INCIDENT TRIAGE RUNBOOK · 17 MIN READ

CloudFront S3 403 Access Denied: Production SRE Triage Runbook

A systematic production guide to diagnosing and fixing HTTP 403 Access Denied errors between AWS CloudFront and Amazon S3: Origin Access Control (OAC), IAM bucket policies, subdirectory index.html routing, and viewer-request functions.

By DevOps Tribune Engineering Staff · Published November 17, 2026

Deploying static websites and single-page applications behind Amazon CloudFront and Amazon Simple Storage Service (S3) is an industry-standard architectural pattern. It provides global Anycast content delivery, DDoS protection via AWS Shield, automated SSL/TLS termination, and near-zero server maintenance costs. However, during new environment deployments or infrastructure migrations, platform engineers routinely encounter the dreaded 403 Access Denied error returned by CloudFront edge nodes.

The primary diagnostic challenge of the CloudFront 403 error is its ambiguity. Because Amazon S3 protects object metadata, if an unauthenticated requester queries an S3 object that does not exist, S3 deliberately returns 403 Forbidden instead of 404 Not Found to prevent malicious enumeration of private bucket keys. Consequently, a 403 status code can indicate a genuine IAM permissions failure, a missing S3 object, a missing subdirectory trailing slash, a malformed CloudFront Origin Access Control (OAC) policy, or an unhandled URL path in an edge function.

This comprehensive production runbook establishes a deterministic, step-by-step diagnostic framework for Site Reliability Engineers (SREs) and platform architects. We dissect the security architecture of Origin Access Control (OAC), examine S3 bucket policy conditions, explain the mechanics of subdirectory routing, implement a lightweight CloudFront viewer-request function, and provide complete, copy-paste-safe Terraform/OpenTofu configurations.


1. The Core Root Causes: Why CloudFront Returns 403 Forbidden

In production cloud environments, CloudFront-to-S3 403 errors stem from four distinct architectural failure domains:

  1. The Missing Object "Security Cloaking" Dilemma: The requested S3 key does not exist. However, because CloudFront's IAM principal lacks the s3:ListBucket permission, S3 refuses to disclose whether the object is absent and returns 403 Access Denied instead of 404 Not Found.
  2. Subdirectory Index Resolution Failure: CloudFront's "Default Root Object" setting applies exclusively to the root path (/). When a user browses to https://example.com/about/, CloudFront forwards a request to S3 for the key about/ rather than about/index.html. S3 treats about/ as an empty prefix (or non-existent key) and returns 403.
  3. Origin Access Control (OAC) Mismatches: The S3 bucket policy either lacks the required s3:GetObject grant for cloudfront.amazonaws.com, specifies an incorrect distribution ARN in the AWS:SourceArn condition, or has conflicting explicit deny statements.
  4. KMS Key Decryption Failures: The S3 bucket is encrypted using AWS Key Management Service (SSE-KMS) with a Customer Managed Key (CMK), but the KMS key policy does not grant kms:Decrypt permissions to the CloudFront service principal.
Golden Rule of CloudFront S3 Diagnostics Whenever CloudFront returns 403 Access Denied for a URL path containing a directory (e.g., /blog/ or /docs/), verify whether the actual object key is /blog/index.html. In 90% of static site deployments, the issue is not IAM permissions; it is the absence of an edge URL rewrite appending index.html.

2. Origin Access Control (OAC) vs Legacy Origin Access Identity (OAI)

Historically, securing S3 origins behind CloudFront relied on Origin Access Identity (OAI), a legacy mechanism where CloudFront created a virtual IAM user principal. AWS officially deprecated OAI in favor of Origin Access Control (OAC).

Why OAC is Mandatory for Modern Architectures

  • Full KMS Encryption Support: OAI cannot authenticate against S3 buckets encrypted with AWS KMS (SSE-KMS) Customer Managed Keys. OAC natively passes cryptographic request signatures to KMS.
  • Support for All AWS Regions: OAI does not function in regions launched after December 2022 (e.g., Zurich, Spain, Hyderabad, Melbourne). OAC is universally supported across all global AWS partitions.
  • Enhanced Security via SigV4: OAC uses AWS Signature Version 4 (SigV4) authentication, cryptographically signing every HTTP request between CloudFront edge locations and the S3 REST API endpoint.
  • Granular Resource Scoping: OAC allows S3 bucket policies to validate the exact CloudFront distribution ARN using the AWS:SourceArn condition, completely preventing other AWS customers from configuring origins pointing to your private bucket.

3. Production S3 Bucket Policy for CloudFront OAC

A production S3 origin bucket must have 100% of public access blocked at the bucket level (BlockPublicAcls, IgnorePublicAcls, BlockPublicPolicy, and RestrictPublicBuckets enabled). Access is granted exclusively through an explicit bucket policy validating the CloudFront distribution ARN.

Hardened S3 Bucket Policy Manifest

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCloudFrontServicePrincipalReadOnly",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudfront.amazonaws.com"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::devopstribune-production-origin-1794/*",
      "Condition": {
        "StringEquals": {
          "AWS:SourceArn": "arn:aws:cloudfront::347166961665:distribution/EX6PQD1ERDSMT"
        }
      }
    }
  ]
}

Critical Bucket Policy Pitfalls

  • Wildcard in Resource ARN: The resource must be arn:aws:s3:::bucket-name/*. If you omit the /* suffix, the policy applies to the bucket itself rather than the objects inside, causing all s3:GetObject requests to fail with 403.
  • Exact Distribution ARN Match: The AWS:SourceArn condition must match the exact AWS account ID and CloudFront distribution ID. A typo in the distribution ID will cause S3 to reject all incoming SigV4 signatures.
  • Conflicting Explicit Deny Statements: Check whether your organization enforces AWS Organizations Service Control Policies (SCPs) or bucket-level IP whitelists that inadvertently block the cloudfront.amazonaws.com service principal.

4. Resolving Subdirectory index.html Routing with CloudFront Functions

In standard web servers (such as Nginx, Apache, or Caddy), requesting a directory like /articles/ causes the web server to automatically locate and serve /articles/index.html. However, Amazon S3 is an object store, not a traditional POSIX filesystem. S3 treats articles/index.html as an arbitrary string key.

CloudFront's Default Root Object setting only appends index.html to requests for the absolute root path (https://www.devopstribune.com/ -> s3://bucket/index.html). It does not apply to subdirectories (https://www.devopstribune.com/about/ -> queries S3 for key about/ -> returns 403 Access Denied).

The Architectural Solution: CloudFront Viewer-Request Function

To resolve this without making the S3 bucket publicly readable via static website hosting endpoints, attach a lightweight JavaScript CloudFront Function to the viewer-request event. CloudFront Functions execute in sub-millisecond time at AWS edge locations, rewriting directory URIs before the request ever reaches the S3 origin.

// infrastructure/cloudfront/router.js
function handler(event) {
  var request = event.request;
  var rawHost = request.headers.host && request.headers.host.value;
  if (!rawHost) return deny();
  
  var host = rawHost.toLowerCase().replace(/:\d+$/, '');
  var uri = request.uri || '/';
  
  // 1. Security path sanitization
  if (uri.length > 2048 || uri.indexOf('\\') !== -1 || uri.indexOf('//') !== -1 ||
      /%(?:2f|5c|2e|00)/i.test(uri) || /%(?![0-9a-f]{2})/i.test(uri) ||
      /(^|\/)\.{1,2}(\/|$)/.test(uri)) {
    return deny();
  }

  // 2. 301 Redirect apex domain to canonical www domain
  if (host === 'devopstribune.com') {
    return {
      statusCode: 301,
      statusDescription: 'Moved Permanently',
      headers: {
        location: { value: 'https://www.devopstribune.com' + uri }
      }
    };
  }

  // 3. Normalize directory paths to append index.html
  var path = uri;
  if (path === '/') {
    path = '/index.html';
  } else if (path.charAt(path.length - 1) === '/') {
    path += 'index.html'; // Rewrite /about/ to /about/index.html
  } else if (path.lastIndexOf('.') < path.lastIndexOf('/')) {
    // Missing trailing slash: 301 Redirect /about to /about/
    return {
      statusCode: 301,
      statusDescription: 'Moved Permanently',
      headers: {
        location: { value: 'https://www.devopstribune.com' + path + '/' }
      }
    };
  }

  // 4. Map request to internal S3 origin prefix
  request.uri = '/sites/live' + path;
  return request;
}

function deny() {
  return {
    statusCode: 403,
    statusDescription: 'Forbidden',
    headers: { 'content-type': { value: 'text/plain' } }
  };
}

5. S3 Encryption with AWS KMS: Resolving KMS 403 Errors

If your organization enables server-side encryption with AWS Key Management Service (SSE-KMS) on the S3 bucket using a Customer Managed Key (CMK), CloudFront requires explicit cryptographic permissions. When CloudFront fetches an encrypted object from S3, S3 invokes KMS on CloudFront's behalf using the SigV4 credentials. If the KMS key policy does not trust CloudFront, the request aborts with an HTTP 403 Access Denied error.

Required AWS KMS Key Policy Statement

Add the following statement to your KMS Key Policy:

{
  "Sid": "AllowCloudFrontServicePrincipalToDecrypt",
  "Effect": "Allow",
  "Principal": {
    "Service": "cloudfront.amazonaws.com"
  },
  "Action": [
    "kms:Decrypt",
    "kms:GenerateDataKey*"
  ],
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "AWS:SourceArn": "arn:aws:cloudfront::347166961665:distribution/EX6PQD1ERDSMT"
    }
  }
}

6. CloudFront Error Caching and Negative TTL Traps

During an incident where an S3 permission issue is resolved, engineers frequently test the endpoint with curl or a web browser and observe that the 403 error persists. This is often caused by CloudFront Error Caching (Negative Caching).

By default, CloudFront caches HTTP 403 and 404 responses from origins for up to 300 seconds (5 minutes). Even after you fix the S3 bucket policy or upload the missing file, edge locations continue serving the cached 403 error response until the negative cache TTL expires.

Bypassing the Error Cache During Triage

  • Append a Cache-Busting Query String: Test the endpoint with a unique query string: curl -sI "https://www.devopstribune.com/about/?cb=12345". This forces the CloudFront edge node to fetch directly from S3 origin.
  • Configure Custom Error Response TTL to Zero: In your CloudFront distribution settings, configure a Custom Error Response rule for HTTP 403 with an Error Caching Minimum TTL of 0 or 10 seconds during active development.

7. Production AWS CLI Diagnostic Commands

When an active 403 incident occurs, execute the following commands in sequence to rapidly isolate the root cause:

# 1. Inspect HTTP response headers (Check x-cache header: Hit vs Miss)
curl -sI "https://www.devopstribune.com/articles/caddy-web-server-tutorial/"

# Expected healthy response:
# HTTP/2 200
# content-type: text/html
# x-cache: Hit from cloudfront (or Miss from cloudfront)

# 2. Verify whether the exact object key exists in S3
aws s3 ls s3://devopstribune-production-origin-1794/sites/live/articles/caddy-web-server-tutorial/index.html \
  --profile naveed

# 3. Retrieve and inspect live S3 bucket policy
aws s3api get-bucket-policy \
  --bucket devopstribune-production-origin-1794 \
  --query Policy --output text | jq .

# 4. Verify CloudFront OAC association on the target origin
aws cloudfront get-distribution-config \
  --id EX6PQD1ERDSMT \
  --query "DistributionConfig.Origins.Items[0].OriginAccessControlId"

# 5. Test CloudFront Function router locally
aws cloudfront test-function \
  --name devopstribune-viewer-router \
  --if-match "ETagValue" \
  --event-object fileb://test-event.json \
  --stage LIVE

8. Complete Production Infrastructure Definition (Terraform / OpenTofu)

The following production-hardened OpenTofu/Terraform code defines a zero-trust, private S3 bucket, CloudFront OAC, viewer-request function, and security response headers policy:

# infrastructure/terraform/s3_cloudfront_oac.tf

# 1. Private S3 Origin Bucket
resource "aws_s3_bucket" "site" {
  bucket = "devopstribune-production-origin-1794"
}

resource "aws_s3_bucket_public_access_block" "site" {
  bucket = aws_s3_bucket.site.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_server_side_encryption_configuration" "site" {
  bucket = aws_s3_bucket.site.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

# 2. CloudFront Origin Access Control (OAC)
resource "aws_cloudfront_origin_access_control" "oac" {
  name                              = "devopstribune-oac"
  description                       = "OAC for private S3 website distribution"
  origin_access_control_origin_type = "s3"
  signing_behavior                  = "always"
  signing_protocol                  = "sigv4"
}

# 3. CloudFront Viewer-Request Function (Trailing Slash & URL Router)
resource "aws_cloudfront_function" "router" {
  name    = "devopstribune-viewer-router"
  runtime = "cloudfront-js-2.0"
  comment = "Appends index.html to directory requests and enforces trailing slashes"
  publish = true
  code    = file("${path.module}/../cloudfront/router.js")
}

# 4. S3 Bucket Policy Granting OAC Access
resource "aws_s3_bucket_policy" "site" {
  bucket = aws_s3_bucket.site.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "AllowCloudFrontOAC"
        Effect    = "Allow"
        Principal = {
          Service = "cloudfront.amazonaws.com"
        }
        Action   = "s3:GetObject"
        Resource = "${aws_s3_bucket.site.arn}/*"
        Condition = {
          StringEquals = {
            "AWS:SourceArn" = aws_cloudfront_distribution.site.arn
          }
        }
      }
    ]
  })
}

9. Step-by-Step Diagnostic Flowchart

Follow this systematic decision tree to diagnose any CloudFront S3 403 incident:

  1. Step 1: Check the URL Path. Does the requested URL end in a slash (/about/) or no slash (/about)?
    • If no slash and no file extension: Does your viewer-request function redirect to /about/? If not, CloudFront queries key about and S3 returns 403.
    • If ending in slash: Does your viewer-request function rewrite to /about/index.html? If not, S3 returns 403.
  2. Step 2: Check S3 Key Existence. Run aws s3 ls s3://bucket/path/index.html. If the file does not exist, upload it. S3 returns 403 for missing keys when public list access is disabled.
  3. Step 3: Check S3 Bucket Policy. Verify that s3:GetObject is granted to cloudfront.amazonaws.com with an AWS:SourceArn condition matching your distribution.
  4. Step 4: Check S3 Origin Domain Name. Ensure your CloudFront distribution is pointing to the S3 REST API endpoint (bucket.s3.us-east-1.amazonaws.com) and NOT the static website hosting endpoint (bucket.s3-website...). OAC only functions with REST endpoints.
  5. Step 5: Check Negative Cache TTL. Bypass edge cache using ?nocache=1 to ensure you are not viewing a stale cached 403 error.

10. Cross-Account Architecture: CloudFront in Account A, S3 in Account B

In enterprise multi-account environments (governed by AWS Organizations), platform teams often separate edge networking from data storage. A central networking account hosts the CloudFront distribution, while application workloads store static assets inside dedicated workload account S3 buckets. This introduces unique cross-account permission requirements.

Cross-Account OAC Bucket Policy

When CloudFront is in Account 111111111111 and the S3 bucket is in Account 222222222222, the S3 bucket policy must explicitly grant permissions to the external CloudFront distribution ARN:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCrossAccountCloudFrontOAC",
      "Effect": "Allow",
      "Principal": {
        "Service": "cloudfront.amazonaws.com"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::workload-storage-account-b/*",
      "Condition": {
        "StringEquals": {
          "AWS:SourceArn": "arn:aws:cloudfront::111111111111:distribution/EDISTRIBUTIONID"
        }
      }
    }
  ]
}

Object Ownership Pitfall: BucketOwnerEnforced

Historically, when external CI/CD pipelines uploaded files to S3, the uploading IAM identity owned the object rather than the bucket owner, causing CloudFront OAC to fail with 403 Access Denied even with a valid bucket policy. Always enforce S3 Object Ownership to BucketOwnerEnforced in the destination bucket, permanently disabling legacy ACLs:

resource "aws_s3_bucket_ownership_controls" "site" {
  bucket = aws_s3_bucket.site.id

  rule {
    object_ownership = "BucketOwnerEnforced"
  }
}

11. CloudWatch Metrics and Athena Log Analysis for 403 Incidents

During an active production incident, platform teams must isolate whether 403 errors originate from the CloudFront edge layer or from the S3 origin. Inspect CloudFront standard access logs stored in S3 using Amazon Athena queries:

-- Amazon Athena Query to Isolate CloudFront 403 Errors
SELECT 
    date,
    time,
    cs_method,
    cs_uri_stem,
    sc_status,
    sc_sub_status,
    x_edge_result_type,
    x_edge_response_result_type,
    cs_user_agent
FROM cloudfront_logs
WHERE sc_status = 403
  AND date = CURRENT_DATE
ORDER BY time DESC
LIMIT 50;

Interpreting Athena log results:

  • x_edge_response_result_type = Error with x_edge_result_type = Miss: CloudFront contacted S3, and S3 explicitly returned 403. Focus triage on S3 bucket policy, key existence, or KMS permissions.
  • x_edge_response_result_type = Error with x_edge_result_type = Hit: CloudFront served a cached negative 403 response. Invalidate the edge cache or bypass TTL with query parameters.
  • x_edge_result_type = FunctionGeneratedResponse: The CloudFront Function or Lambda@Edge viewer-request router executed an explicit deny(). Focus triage on routing logic and URI security regexes.

12. Production Postmortem Checklist: Preventing Recurrent 403 Outages

Following the resolution of a CloudFront S3 403 incident, SRE teams should complete a blameless postmortem to implement systemic architectural safeguards against recurrence:

  • Automated Route Validation in CI/CD: Add integration tests to your deployment pipeline (e.g., in GitHub Actions) that verify every generated static directory contains an index.html before uploading to S3.
  • CloudFront Function Unit Testing: Integrate automated unit tests for your viewer-request JavaScript router (using Node.js assert or Mocha) to validate trailing slash normalization and security regexes on every commit.
  • Terraform Drift Detection: Schedule daily or weekly automated tofu plan -detailed-exitcode runs to ensure out-of-band AWS console modifications do not silently detach Origin Access Control from S3 origins.
  • Custom Error Response Pages: Configure CloudFront Custom Error Responses for HTTP 403 to serve a branded, user-friendly error page (such as /404.html) with an Error Caching Minimum TTL of 10 seconds, preventing prolonged user impact during transient origin glitches.
  • Alarm on 4xx Error Rates: Configure CloudWatch alarms on CloudFront metric 4xxErrorRate exceeding 1.0% over a 5-minute evaluation period, alerting on-call engineers via PagerDuty or Slack before customers submit support tickets.

13. Frequently Asked Questions

Why does S3 return 403 Forbidden instead of 404 Not Found when a file is missing?

By default, anonymous requesters (and CloudFront without s3:ListBucket permissions) are not permitted to list bucket contents. If S3 returned 404 for missing keys and 403 for existing unauthorized keys, an attacker could enumerate private bucket filenames by observing status code differences. Returning 403 for all unauthorized or non-existent objects preserves security cloaking.

Can I use CloudFront OAC with S3 Static Website Hosting endpoints?

No. Origin Access Control (OAC) and legacy OAI operate strictly with S3 REST API endpoints (e.g., bucket.s3.region.amazonaws.com). If you configure CloudFront with an S3 website endpoint (bucket.s3-website.region.amazonaws.com), OAC SigV4 headers are ignored, and the bucket must be made completely public to the internet.

What is the performance difference between CloudFront Functions and Lambda@Edge?

CloudFront Functions execute in the lightweight process space of edge cache servers in under 1 millisecond at 1/6th the cost of Lambda@Edge, making them ideal for URL rewrites and header mutations. Lambda@Edge runs in full Node.js or Python VM runtimes in regional edge caches with higher execution latencies (10-50ms), reserved for complex tasks like network calls or request body modifications.

Related Edge & Cloud Infrastructure Guides