Cloud networking (VPC)

What makes a subnet public or private

A VPC is a private IP range you carve into subnets. Nothing about a subnet is inherently public or private. The distinction is a property of the subnet's route table: a subnet is public when its route table sends default traffic (0.0.0.0/0) to an internet gateway, and private when it does not.

# Public subnet route table
10.0.0.0/16   local            # traffic inside the VPC
0.0.0.0/0     igw-0abc...      # everything else to the internet gateway

# Private subnet route table
10.0.0.0/16   local
0.0.0.0/0     nat-0def...      # outbound via a NAT gateway, no direct IGW route

Wrong "It has a public IP, so it can reach the internet." A public IPv4 address does nothing without a route to an internet gateway. An instance in a subnet whose route table has only the local route is stranded no matter what address it holds or how open its security group is. Right reachability is routing first. The address and the firewall rules only matter once a path exists.

Internet gateway vs NAT gateway

An internet gateway (IGW) is the VPC's door to the internet. It allows two-way traffic: an instance with a public IP in a public subnet can both initiate outbound connections and receive inbound ones.

A NAT gateway is for private subnets that need to reach out but must never be reachable from outside. It maps many private source addresses to one address and, in AWS terms, lets "instances in a private subnet connect to services outside your VPC but external services can't initiate a connection with those instances." It is one-directional by design: outbound works, unsolicited inbound is dropped.

The wiring people get wrong: a NAT gateway is not attached to an instance. It lives in a public subnet, holds an Elastic IP, and instances in private subnets route their 0.0.0.0/0 to it. The NAT gateway then routes to the internet gateway. So a private instance's packet goes instance to NAT gateway to IGW to internet, and the reply retraces the path.

Wrong "Give the instance a NAT gateway and it's on the internet." NAT only carries connections the instance starts. If you need inbound (a public web server), that instance belongs in a public subnet behind an IGW, or behind a load balancer that is.

Security groups vs NACLs, and the order to debug in

Two firewalls sit between a packet and your instance, and they behave differently.

When a connection hangs or times out, debug in this order:

1. Routing. Does the source subnet's route table have a path to the destination, and the destination's back to the source? No path, nothing else matters.
2. Network ACL. It is stateless, so it must allow both the request and the response. Replies come back to the client's ephemeral source port (roughly 1024-65535), not the service port, so an outbound NACL rule that only allows 443 drops every response. This asymmetry is the single most common NACL bug.
3. Security group. It is stateful, so if it allowed the inbound request, the reply is allowed automatically. You rarely need a return rule here, which is why a one-way hang usually is not the security group.

Wrong "Inbound is allowed on the NACL, so the response can get back out." NACLs remember nothing. An allowed inbound request does not imply an allowed outbound reply; you add the ephemeral-port outbound rule yourself.

Why reading S3 through a NAT gateway costs money

Put a workload in a private subnet, have it read from S3, and its traffic exits through the NAT gateway. NAT gateways bill two ways: an hourly charge and a per-GB data-processing charge (about $0.045/GB in US Regions). Move several TB through it and the per-GB line dominates the bill. The S3 data itself is in the same Region and never needed the public internet.

The fix is a gateway VPC endpoint for S3 (it also serves DynamoDB). It is free, adds no hourly or per-GB cost, and installs an AWS-managed prefix-list route into your subnet's route table so S3 traffic goes straight to the service and skips the NAT gateway.

Wrong "S3 is on the internet, so the NAT charge is unavoidable." A gateway endpoint keeps that traffic on the AWS network and off the metered NAT path, for no charge. Any private subnet doing bulk S3 or DynamoDB work should have one.

When a NAT gateway runs out of ports

A NAT gateway looks infinite until a fan-out workload hits a port ceiling. Each IPv4 address on the gateway supports up to 55,000 simultaneous connections to each unique destination, where a unique destination is the combination of destination IP, destination port, and protocol. Ten thousand workers all opening connections to the same api.example.com:443 share that one bucket; cross it and new connections fail while the gateway emits the ErrorPortAllocation metric and increments PacketsDropCount.

The knobs, in order of reach:

• Spread destinations. The 55,000 limit is per unique destination, so traffic to many distinct endpoints rarely exhausts it; a single hot destination does.
• Add addresses. You can attach up to 8 IPv4 addresses (1 primary, 7 secondary) to a NAT gateway, multiplying the per-destination ceiling.
• Split subnets. A NAT gateway scales to 100 Gbps and 10 million packets/second on its own; past that you shard resources across subnets, one NAT gateway each.

Watch ErrorPortAllocation in CloudWatch before it becomes an incident. It is the metric that tells you the fan-out, not the bandwidth, is the problem.

Gateway endpoints vs interface endpoints

Both keep traffic off the internet, but they are different mechanisms with different bills.

A gateway endpoint serves only S3 and DynamoDB. It is a route-table construct: you get an AWS-managed prefix list as a destination pointing at the endpoint, and it costs nothing. It is Region-scoped and cannot be reached across a peering connection, a VPN, or from on-premises.

An interface endpoint (AWS PrivateLink) is an elastic network interface with a private IP inside your subnet. It fronts most AWS services and your own or partner services, and it is reachable across peering, VPN, and Direct Connect because it is just a private IP. It bills an hourly charge per AZ (about $0.01/hour per ENI) plus per-GB processing (about $0.01/GB).

For bulk S3 from inside one VPC, the gateway endpoint is both simpler and free. Reach for the interface endpoint when you need the service over a hybrid link, from a peered VPC, or for a service gateway endpoints do not cover.

Wrong "A gateway endpoint lets my on-premises servers reach S3 privately." Gateway endpoints do not extend past the VPC; a peered VPC, a VPN, or Direct Connect cannot use them. Private S3 access over a hybrid link is an interface endpoint job.

Why peering does not scale and transit gateway does

VPC peering is a one-to-one, non-transitive link. Three properties bite as the topology grows:

• Non-transitive. If A peers B and B peers C, A cannot reach C through B. Every pair that must talk needs its own connection, so N fully-connected VPCs need N(N-1)/2 peerings.
• No overlapping CIDRs. You cannot peer two VPCs with matching or overlapping ranges, even if you only intend to use the non-overlapping parts. Address planning up front is not optional.
• No edge-to-edge routing. A peered VPC cannot borrow your internet gateway, NAT gateway, VPN, Direct Connect, or gateway endpoint. Each VPC provides its own egress.

A transit gateway replaces the mesh with a hub. It is a Regional virtual router: VPCs, VPN connections, and Direct Connect gateways attach to it, and it routes transitively between attachments. Each attachment associates with exactly one transit gateway route table, so you get segmentation for free (put prod and dev attachments on separate route tables and they do not route to each other). VPC attachments need static routes into the transit gateway route table; VPN and Direct Connect attachments propagate routes over BGP.

# Instead of peering A-B, A-C, B-C, D-...  (a growing mesh)
A ─┐
B ─┼─ transit gateway (hub, routes between attachments)
C ─┘

The tradeoff is cost and a hop: a transit gateway bills per attachment-hour and per-GB processed, and traffic takes one extra logical hop through the hub. For two VPCs, a single peering connection is cheaper and lower-latency. The hub earns its keep once you have several VPCs plus hybrid links to interconnect.

Hybrid links at survey depth

Two ways to reach a VPC from your own data center, and interviewers want the tradeoff, not the config:

• Site-to-Site VPN is IPsec over the public internet, delivered as two tunnels to a virtual private gateway or a transit gateway. Quick to stand up, cheap, but latency and throughput ride the public internet.
• Direct Connect is a dedicated physical link into AWS. Consistent latency, higher and more predictable bandwidth, no public internet in the path, but it takes weeks to provision and costs more. Common pattern: Direct Connect for the steady load, a VPN tunnel as the encrypted backup.

One caller-facing failure mode worth carrying from the middle notes into production reviews: because network ACLs are stateless, a subnet that allows a service port inbound but not the ephemeral range (1024-65535) outbound will accept connections and then drop every reply. Security groups never show this because they are stateful. When a connection opens but hangs, suspect the NACL's return path before the security group.