Day 07: The Manual Load Balancer (Nginx Reverse Proxy)

🎯 The Objective

In the cloud, we often click "Create Load Balancer" and treat it as a magic black box. But what is actually happening inside?

Today, we break the magic. We will build a Layer 7 Load Balancer from scratch using Nginx. Our goal is to understand the First Principles:

1. The Reverse Proxy: The "Gateway" that terminates your connection and opens a new one to the backend.
2. The Upstream: The "Backend Service" that manages a pool of IPs.
3. Resilience: How the system survives a zonal outage without the user noticing.

🏗️ Phase 1: The Architecture (Manager vs. Workers)

We are building a High Availability cluster in the asia-south1 (Mumbai) region.

• The Workers (Backends): Two simple Python web servers. We place them in Different Zones (Zone A and Zone B). This simulates two separate physical data centers.
• The Manager (Load Balancer): An Nginx server in Zone C. It acts as the single entry point.

Step 1: The Setup We spin up 3 VMs. The backends run a startup script that creates a custom index.html so we can identify them.

[IMAGE PLACEHOLDER: vm-setup.png]

The Nginx Logic (The Brain): We configure Nginx (/etc/nginx/sites-available/default) to act as the proxy. This file maps the cloud concepts to code:

# 1. THE BACKEND SERVICE (Upstream Block)
# This is our pool of healthy workers.
upstream my_backend_pool {
    server 10.128.0.2;  # Backend A (Zone A)
    server 10.128.0.3;  # Backend B (Zone B)
}

# 2. THE URL MAP (Location Block)
server {
    listen 80;
    location / {
        # 3. THE PROXY (Forwarding Rule)
        proxy_pass http://my_backend_pool;
    }
}

🔄 Phase 2: Round Robin & Failover

1. Round Robin Test: By default, Nginx uses Round Robin. When we curl the Manager's IP, it takes turns: Backend A -> Backend B -> Backend A -> Backend B.

2. The "Kill" Test: We simulated a Data Center failure by stopping Backend A:

gcloud compute instances stop backend-a --zone=asia-south1-a

We immediately hit the Manager again.

Result: Nginx detected the failure (Passive Health Check), marked A as "Dead," and instantly routed 100% of traffic to Backend B. The user experienced zero downtime.

🧠 Phase 3: Advanced Logic (Sticky & Weighted)

We explored two critical configurations:

1. Sticky Sessions (Session Affinity)

We added ip_hash; to the upstream block.

Effect: My specific IP address was "glued" to Backend A.

The Trade-off: While this preserves user state (like a shopping cart), it breaks true load balancing. If one office with 5,000 users shares an IP, they all hammer one server.

Modern Fix: Keep the LB "Stateless" and store the cart in Redis.

2. Weighted Balancing (Canary)

We assigned weight=3 to Backend A and weight=1 to Backend B.

Effect: Backend A took 75% of the traffic. This is exactly how Canary Deployments work (slowly shifting traffic to a new version).

🔬 Deep Dive: The Logic of a Packet

To be a Security Architect, you must understand both the Network Layers (Encryption/Termination) and the Cloud Components (Routing/Decision Making).

The Balanced Flow:

1. The Entry (Anycast): The user sends an HTTPS request. It lands at the nearest Google Edge location (not necessarily where your servers are).

2. Termination (The Handshake): The Target Proxy accepts the connection.

   • Layer Detail: It performs the TLS Handshake here, decrypting the traffic so it can inspect the Layer 7 data.

3. Security (The Gatekeeper): Cloud Armor inspects the now-decrypted request.

   • Decision: Is this SQL Injection? Is the IP banned? -> Block or Allow.

4. Routing (The Brain): The URL Map looks at the path (e.g., /app).

   • Decision: Does this path exist? Which "Backend Service" handles it?

5. Selection (The Manager): The Backend Service looks at the Instance Group.

   • Logic: Filter out unhealthy VMs -> Apply Round Robin -> Pick VM-A.

6. The Handoff (Internal Network): The Proxy initiates a New TCP Connection to VM-A using its Internal IP.

   • Note: The source IP is now the Proxy's IP, not the User's IP.

7. Processing: VM-A processes the request and hands the response back to the Proxy to be re-encrypted and delivered.

📝 Key Takeaways

• L7 Awareness: A Layer 7 LB doesn't just forward packets; it terminates the connection, reads the request, and makes intelligent decisions (Routing).
• Passive Health Checks: The LB actively removes failed servers from rotation without needing external monitoring tools.
• Stateless Architecture: Sticky Sessions are a patch; External State (Redis) is the solution.