Highly Available and Resilient Multi-AWS VPC Architectures: Design Patterns for Private Subnets and Inter-VPC Communication

To build highly available and resilient multi-AWS VPC architectures, deploy workloads in private subnets with NAT Gateways per Availability Zone, use Transit Gateway for scalable inter-VPC communication, and distribute traffic across multiple AZs using load balancers and Route 53 health checks.

According to the bregman-arie/devops-exercises repository, enterprise-grade AWS networking requires isolating compute resources while maintaining robust connectivity between VPCs. The source code analysis reveals specific patterns for implementing fault-tolerant infrastructure that survives individual Availability Zone failures and supports complex multi-VPC topologies.

Private Subnets and NAT Gateways: The Foundation of High Availability

Isolating Workloads in Private Subnets

Place all workload instances in private subnets to ensure they are never exposed directly to the Internet. As documented in topics/cloud/README.md, instances should be accessed exclusively through load balancers or bastion hosts, remaining "off the internet (in a private subnet behind a NAT)." This approach eliminates public IP exposure while permitting necessary outbound traffic for updates and package downloads.

NAT Gateway Distribution Across Availability Zones

Deploy one NAT Gateway per Availability Zone to eliminate single points of failure for egress traffic. This pattern ensures that if one AZ becomes unavailable, instances in private subnets of surviving AZs maintain outbound connectivity through their local NAT Gateway. The repository emphasizes that distributing NAT Gateways across multiple AZs prevents a failure in one zone from cutting off egress for the entire VPC.

High-Availability Networking Primitives

Internet Gateway Constraints and Configuration

Each VPC can attach to only one Internet Gateway (IGW), as explicitly noted in topics/aws/README.md (lines 84-86). While this creates a single point of egress for public traffic, you mitigate risks by deploying public-facing components across multiple AZs behind redundant load balancers. The IGW attaches to the VPC and routes public subnet traffic, but never directly to private subnet instances.

Load Balancer Distribution and Auto-Scaling

Deploy multiple Application Load Balancers (ALBs) or Network Load Balancers (NLBs) across Availability Zones, registering targets exclusively from private subnets. Combine this with EC2 Auto-Scaling groups configured with a minimum of two instances across at least two AZs. This configuration guarantees capacity during AZ failures and provides DNS-based failover through Route 53 health-checked alias records that route traffic to healthy endpoints, with fallback to secondary regions when necessary.

Security Groups versus Network ACLs

Implement layered security using Security Groups for instance-level allow-lists and Network ACLs (NACLs) for subnet-level deny/allow boundaries. The repository distinguishes these concepts in topics/aws/README.md (lines 55-60), recommending Security Groups for stateful firewall rules specific to individual instances and NACLs for broader subnet traffic policies.

Inter-VPC Communication Patterns

VPC Peering for Direct Connectivity

Use VPC Peering for simple, same-region connectivity between two VPCs. This creates a direct routing path for private IPv4/IPv6 traffic, but note the critical limitation documented in topics/aws/README.md (lines 96-99): peering maintains a one-to-one relationship with no transitive routing support. If VPC A peers with VPC B, and VPC B peers with VPC C, VPC A cannot reach VPC C through B.

resource "aws_vpc_peering_connection" "peer_ab" {
  vpc_id      = aws_vpc.vpc_a.id
  peer_vpc_id = aws_vpc.vpc_b.id
  auto_accept = true
  tags = { Name = "A-to-B-Peering" }
}

resource "aws_route" "a_to_b" {
  route_table_id            = aws_vpc.vpc_a.main_route_table_id
  destination_cidr_block    = aws_vpc.vpc_b.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.peer_ab.id
}

AWS Transit Gateway for Scalable Hub-and-Spoke

Implement AWS Transit Gateway when managing many VPCs (per-environment or per-team configurations) requiring transitive routing. This centralized hub connects dozens of VPCs and on-premises networks through a single routing table, supporting bandwidth-based routing and inter-region peering for cross-region high availability. Unlike VPC Peering, Transit Gateway allows VPC A to communicate with VPC C through the hub without direct peering between them.

resource "aws_ec2_transit_gateway" "core" {
  description     = "Core TGW for multi-VPC hub"
  amazon_side_asn = 64512
}

resource "aws_ec2_transit_gateway_vpc_attachment" "vpc_a" {
  transit_gateway_id = aws_ec2_transit_gateway.core.id
  vpc_id             = aws_vpc.vpc_a.id
  subnet_ids         = [aws_subnet.public_a.id, aws_subnet.public_b.id]
}

AWS PrivateLink for Service-Oriented Access

Expose specific services (RDS instances, internal APIs) via Interface VPC Endpoints using AWS PrivateLink. This pattern keeps traffic within the AWS network, eliminating the need for VPC peering for service consumption while maintaining strict network isolation.

Shared Services VPC Architecture

Create a dedicated Shared Services VPC hosting centralized resources such as NAT Gateways, shared DNS resolvers, and security-inspection appliances. Workload VPCs attach to this hub via Transit Gateway or peering, obtaining a single point for egress/NAT and centralized security policy enforcement. This design reduces operational overhead by consolidating shared infrastructure into one managed network boundary.

Operational Considerations for Resilience

Subnet Sizing and IP Address Management

AWS reserves five IP addresses per subnet (network address, broadcast address, and three for AWS internal use). The repository references the "Kratos" example in topics/aws/README.md (lines 77-81), emphasizing the need to allocate sufficient CIDR blocks to accommodate future growth without requiring complex re-addressing.

Monitoring and Failover Testing

Enable VPC Flow Logs and CloudWatch Alarms on NAT Gateway and Transit Gateway health metrics. Implement IAM least-privilege policies granting only required permissions for IGW and NAT management. Regularly simulate AZ failures by stopping instances in private subnets to verify traffic continues routing through remaining AZs and NAT Gateways.

Infrastructure as Code Examples

Terraform VPC with Multi-AZ NAT Gateways

The topics/terraform/exercises/vpc_subnet_creation/solution.md file provides a complete implementation of a highly available VPC with public and private subnets across multiple Availability Zones:

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  tags = { Name = "multi-az-vpc" }
}

resource "aws_subnet" "public_a" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.0.0.0/24"
  availability_zone       = "us-east-1a"
  map_public_ip_on_launch = true
}

resource "aws_subnet" "public_b" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.0.1.0/24"
  availability_zone       = "us-east-1b"
  map_public_ip_on_launch = true
}

resource "aws_subnet" "private_a" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.10.0/24"
  availability_zone = "us-east-1a"
}

resource "aws_subnet" "private_b" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.11.0/24"
  availability_zone = "us-east-1b"
}

resource "aws_eip" "nat_a" { vpc = true }
resource "aws_nat_gateway" "nat_a" {
  allocation_id = aws_eip.nat_a.id
  subnet_id     = aws_subnet.public_a.id
}

resource "aws_eip" "nat_b" { vpc = true }
resource "aws_nat_gateway" "nat_b" {
  allocation_id = aws_eip.nat_b.id
  subnet_id     = aws_subnet.public_b.id
}

resource "aws_route_table" "private_a" {
  vpc_id = aws_vpc.main.id
  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat_a.id
  }
}

resource "aws_route_table_association" "private_a" {
  subnet_id      = aws_subnet.private_a.id
  route_table_id = aws_route_table.private_a.id
}

This configuration establishes redundant egress paths, ensuring private subnet instances in us-east-1a route through nat_a while instances in us-east-1b use nat_b.

Summary

• Private subnet isolation is mandatory for secure workloads; combine with NAT Gateways per AZ to ensure highly available outbound connectivity without public IP exposure.
• AWS Transit Gateway provides the only scalable solution for transitive routing between multiple VPCs, while VPC Peering suffices for simple two-VPC scenarios.
• Load balancers spanning multiple AZs with Route 53 health checks deliver automatic failover capabilities for public-facing services.
• Internet Gateways are limited to one per VPC, requiring careful architecture of public subnets to avoid congestion points.
• Subnet sizing must account for AWS reserving five IP addresses per CIDR block, as documented in the repository's AWS exercises.
• Shared Services VPCs centralize common infrastructure like NAT and DNS, reducing operational complexity across large multi-VPC estates.

Frequently Asked Questions

What is the difference between VPC Peering and Transit Gateway for inter-VPC communication?

VPC Peering creates a direct, one-to-one network connection between exactly two VPCs with no transitive routing capability, meaning traffic cannot flow through an intermediary VPC to reach a third. Transit Gateway acts as a centralized hub that supports transitive routing, allowing dozens of VPCs to communicate through a single attachment point. According to topics/aws/README.md (lines 96-99), choose peering for simple two-VPC scenarios and Transit Gateway when managing complex multi-VPC architectures requiring many-to-many connectivity.

Why should I deploy a NAT Gateway in every Availability Zone?

Deploying one NAT Gateway per AZ eliminates a single point of failure for outbound internet traffic from private subnets. If an AZ experiences an outage, only the NAT Gateway in that specific zone fails; instances in private subnets of healthy AZs continue routing egress traffic through their local NAT Gateway. This pattern ensures your highly available and resilient multi-AWS VPC architecture maintains internet connectivity (for updates, patches, and external API calls) even during partial infrastructure failures.

How do Security Groups and Network ACLs work together in a multi-VPC architecture?

Security Groups operate at the instance level as stateful firewalls, automatically allowing return traffic for established connections. Network ACLs function at the subnet level as stateless filters, evaluating both inbound and outbound rules independently. The repository's topics/aws/README.md (lines 55-60) recommends using Security Groups for granular, instance-specific access controls while employing NACLs for broad subnet-level security boundaries, creating a defense-in-depth strategy across your VPC infrastructure.

What are the IP addressing considerations when designing subnets for high availability?

AWS reserves five IP addresses in every subnet: the network address, broadcast address, and three addresses for AWS internal use (router, DNS, and future use). When planning CIDR blocks for highly available architectures spanning multiple AZs, allocate sufficiently large subnets to accommodate future scaling without exhausting available IPs. The repository references this consideration in topics/aws/README.md (lines 77-81) within the "Kratos" example, emphasizing that re-addressing production subnets requires significant downtime and should be avoided through proper initial sizing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}