NGINX Configuration

The cloud-init provisioning deploys a fully performance-optimized NGINX CDN edge configuration. This page documents every configuration layer, from kernel tuning through cache behavior to vendor header injection. All settings were verified under a 48-hour continuous load test peaking at 172,540 req/s.

Configuration Files

File	Purpose
/etc/sysctl.d/99-cdn-tuning.conf	Linux kernel network tuning
/etc/systemd/system/nginx.service.d/override.conf	File descriptor limits for NGINX
/etc/security/limits.d/99-nginx.conf	OS-level limits for www-data user
/etc/nginx/nginx.conf	NGINX main config (workers, buffers, gzip, logging)
/etc/nginx/conf.d/cdn-edge.conf	CDN proxy config (cache, upstream, headers)

Kernel Tuning

Applied via /etc/sysctl.d/99-cdn-tuning.conf at boot.

Parameter	Value	Default	Purpose
net.core.somaxconn	65535	4096	Listen backlog queue size
net.core.netdev_max_backlog	65535	1000	Per-CPU incoming packet backlog
net.ipv4.tcp_max_syn_backlog	65535	256	SYN request queue (prevents drops under burst)
net.ipv4.tcp_tw_reuse	1	2	Reuse TIME_WAIT sockets for outgoing connections
net.ipv4.ip_local_port_range	1024-65535	32768-60999	Ephemeral port range (64K vs 28K)
net.core.rmem_max	16 MB	212 KB	Maximum receive socket buffer
net.core.wmem_max	16 MB	212 KB	Maximum send socket buffer
net.ipv4.tcp_rmem	4K/87K/16M	4K/131K/6M	Per-socket receive buffer (min/default/max)
net.ipv4.tcp_wmem	4K/65K/16M	4K/16K/4M	Per-socket send buffer (min/default/max)
net.ipv4.tcp_fin_timeout	15	60	FIN_WAIT_2 timeout (faster socket cleanup)
net.ipv4.tcp_keepalive_time	300	7200	Start keepalive probes after 5 min idle
net.ipv4.tcp_slow_start_after_idle	0	1	Keep congestion window warm on idle connections
net.ipv4.tcp_max_tw_buckets	2000000	~65536	Maximum TIME_WAIT sockets
fs.file-max	2097152	varies	System-wide file descriptor limit
vm.swappiness	10	60	Prefer RAM over swap for cache data

NGINX Main Config

Workers and Connections

worker_processes auto;           # 4 workers on D4s_v5 (1 per vCPU)
worker_rlimit_nofile 65535;      # Per-worker file descriptor limit

events {
    use epoll;                   # Linux-optimized event model
    worker_connections 8192;     # 4 workers x 8192 = 32,768 max concurrent
    multi_accept on;             # Accept all pending connections per event loop
    accept_mutex off;            # Not needed with epoll + reuseport
}

Systemd override at /etc/systemd/system/nginx.service.d/override.conf sets LimitNOFILE=65535 to match.

Proxy Buffers

proxy_buffering on;
proxy_buffer_size 16k;           # Header buffer (handles large CDN headers)
proxy_buffers 64 16k;            # 1 MB per connection (64 x 16k)
proxy_busy_buffers_size 256k;    # Can send 256k to client while still reading

The 256k busy buffer size is critical — it must exceed the largest cached response (Juice Shop is 75 KB). The original 64k setting caused serialization under load.

Gzip Compression

gzip on;
gzip_comp_level 4;               # Balance: ~85% of max compression at ~40% CPU
gzip_min_length 256;             # Skip tiny responses (gzip header overhead)
gzip_vary on;                    # Vary: Accept-Encoding for correct caching
gzip_proxied any;                # Compress all proxied responses
gzip_types text/plain text/css text/javascript text/xml
           application/json application/javascript application/xml
           application/xml+rss application/atom+xml
           application/ld+json application/manifest+json
           image/svg+xml;

Open File Cache

open_file_cache max=200000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;

Caches file descriptors and metadata for cached objects, eliminating stat() and open() syscalls on hot files.

Client Keepalive

keepalive_timeout 65;
keepalive_requests 100000;       # Was 1000 — caused connection recycling at 90K req/s

At 90K req/s with keepalive_requests 1000, connections recycled every 11 seconds, generating TIME_WAIT sockets. Increasing to 100,000 eliminated this entirely.

Access Logging

log_format cdn '$remote_addr [$time_local] "$request" $status $body_bytes_sent $upstream_cache_status $request_time';
access_log /var/log/nginx/access.log cdn buffer=256k flush=5s;

The cdn log format includes $upstream_cache_status (HIT/MISS) and $request_time for latency analysis. Buffered logging (256k/5s flush) reduces I/O overhead under high load.

Upstream Keepalive

upstream origin_backend {
    server 20.12.78.159:80;
    keepalive 256;               # Persistent connections per worker to origin
    keepalive_timeout 60s;
    keepalive_requests 1000;
}

With 4 workers x 256 keepalive = 1,024 warm connections to the origin. Combined with proxy_http_version 1.1 and proxy_set_header Connection "", this eliminates TCP handshake overhead on every cache miss.

Cache Configuration

Cache Path

proxy_cache_path /var/cache/nginx/cdn
                 levels=1:2
                 keys_zone=cdn_cache:32m
                 max_size=25g
                 inactive=24h
                 use_temp_path=off;

Parameter	Value	Purpose
keys_zone=cdn_cache:32m	32 MB shared memory	Stores ~256,000 cache keys and metadata
max_size=25g	25 GB disk limit	Uses most of the 30 GB OS disk; LRU eviction at limit
inactive=24h	24-hour inactivity timeout	Content not accessed for 24h is evicted (not the same as TTL)
use_temp_path=off	Write directly to cache dir	Avoids cross-filesystem file moves

Cache Behavior

proxy_cache cdn_cache;
proxy_cache_valid 200 301 302 4h;
proxy_cache_valid 404 1m;
proxy_cache_key "$scheme$host$request_uri";
proxy_cache_lock on;
proxy_cache_lock_age 3s;
proxy_cache_lock_timeout 3s;
proxy_cache_background_update on;
proxy_cache_use_stale updating error timeout http_500 http_502 http_503 http_504;
proxy_ignore_headers Set-Cookie Cache-Control Expires Vary;

proxy_hide_header X-Cache-Status;
proxy_hide_header Vary;

Directive	Value	Purpose
proxy_cache_valid 200 301 302 4h	4-hour TTL	Cached content valid for 4 hours (increased to reduce refresh wave throughput dips)
proxy_cache_lock on	Thundering herd prevention	Only 1 request goes to origin per uncached URL; others wait for cache fill
proxy_cache_lock_age 3s	Lock timeout	If first request hasn’t completed in 3s, allow another through
proxy_cache_background_update on	Zero-latency refresh	Serves stale content immediately while refreshing in background
proxy_cache_use_stale	Resilience	Serves stale content during origin errors (500/502/503/504) or while updating
proxy_ignore_headers	Force caching	Ignores origin Set-Cookie, Cache-Control, Expires, Vary — the CDN decides TTL and Vary behavior (Juice Shop sends max-age=0 and triple Vary headers which prevented effective caching)
proxy_hide_header X-Cache-Status	Strip origin header	Origin NGINX adds its own X-Cache-Status — strip it so only the CDN’s cache status is visible
proxy_hide_header Vary	Prevent cache fragmentation	Origin sends multiple Vary: Accept-Encoding headers. Stripping them prevents cache key fragmentation across Accept-Encoding permutations. NGINX’s gzip_vary on adds the correct single Vary header automatically

Proxy Timeouts

proxy_read_timeout 30s;
proxy_connect_timeout 10s;
proxy_send_timeout 15s;

Gives origin more response time under load while still failing fast on connection issues.

Server Block

server {
    listen 80 reuseport;        # Kernel distributes connections across all 4 workers
    server_name _;
}

reuseport enables SO_REUSEPORT — the kernel distributes incoming connections directly to worker processes, eliminating accept mutex contention.

CDN Vendor Headers

The simulator injects headers from all five major CDN vendors simultaneously. This allows F5 XC to be configured with any vendor’s “Trusted Client IP Header” and see realistic header payloads regardless of which CDN is being simulated.

Industry Standard Headers (All CDNs)

Header	Value	Purpose
X-Forwarded-For	Client IP chain	Standard forwarded IP
X-Forwarded-Proto	http or https	Original client protocol
X-Forwarded-Host	Original hostname	Original Host header
X-Forwarded-Port	Server port	Original port
X-Real-IP	Client IP	Single client IP (nginx convention)
Via	1.1 cdn-simulator	Proxy identification
Forwarded	RFC 7239 format	Standardized forwarding header
CDN-Loop	cdn-simulator	Loop detection

Akamai Headers

Header	Value	Purpose
True-Client-IP	Client IP	Original end-user IP address
X-Akamai-Edgescape	Compound geo string	georegion, country_code, region_code, city, dma, pmsa, msa, areacode, county, fips, lat, long, timezone, zip, continent, throughput, bw, network, asnum, network_type
X-Akamai-Device-Characteristics	Device properties	brand_name, model_name, is_mobile, is_tablet, is_wireless_device, device_os, device_os_version, resolution_width, resolution_height
X-Akamai-Request-ID	Request UUID	Unique request identifier

Cloudflare Headers

Header	Value	Purpose
CF-Connecting-IP	Client IP	True client IP (always present)
CF-IPCountry	US	Two-letter country code
cf-ipcity	San Jose	Client city
cf-ipcontinent	NA	Continent code
cf-iplatitude / cf-iplongitude	Coordinates	Geolocation
cf-region / cf-region-code	California / CA	Region info
cf-metro-code	807	US metro code
cf-postal-code	95113	Postal code
cf-timezone	America/Los_Angeles	IANA timezone
Cf-Ray	{request_id}-SJC	Unique ray ID with POP IATA code
CF-Visitor	{"scheme":"https"}	Visitor protocol info
cf-bot-score	85	Bot score (1=bot, 99=human)
cf-verified-bot	false	Known good bot flag
cf-ja3-hash	e7d705a3286e19ea42f587b344ee6865	JA3 TLS fingerprint
cf-ja4	t13d1516h2_8daaf6152771_b0da82dd1658	JA4 TLS fingerprint

Amazon CloudFront Headers

Header	Value	Purpose
CloudFront-Viewer-Address	IP:port	Client IP and source port
CloudFront-Viewer-Country	US	Country code
CloudFront-Viewer-Country-Name	United States	Full country name
CloudFront-Viewer-Country-Region	CA	Region code
CloudFront-Viewer-Country-Region-Name	California	Full region name
CloudFront-Viewer-City	San Jose	Client city
CloudFront-Viewer-Postal-Code	95113	Postal code
CloudFront-Viewer-Latitude / Longitude	37.33530 / -121.89300	Geolocation
CloudFront-Viewer-Time-Zone	America/Los_Angeles	IANA timezone
CloudFront-Viewer-Metro-Code	807	US metro code
CloudFront-Viewer-ASN	7018	Autonomous System Number
CloudFront-Viewer-Http-Version	2.0	Client HTTP version
CloudFront-Forwarded-Proto	https	Original protocol
CloudFront-Viewer-TLS	TLSv1.3:TLS_AES_128_GCM_SHA256:sessionResumed	TLS details
CloudFront-Viewer-JA3-Fingerprint	e7d705a3286e19ea42f587b344ee6865	JA3 TLS fingerprint
CloudFront-Is-Desktop-Viewer	true/false	Device detection
CloudFront-Is-Mobile-Viewer	true/false	Device detection
CloudFront-Is-Tablet-Viewer	true/false	Device detection
CloudFront-Is-SmartTV-Viewer	false	Device detection
X-Amz-Cf-Id	Encoded ID	CloudFront request identifier

Fastly Headers

Header	Value	Purpose
Fastly-Client-IP	Client IP	True client IP
Fastly-SSL	1	Connection was over TLS
Fastly-Client	1	Client-facing request (not shield)
Fastly-FF	cache-sjc3120-SJC	Cache node identification
X-Geo-Country-Code	US	Country (VCL variable convention)
X-Geo-Country-Code3	USA	Three-letter country code
X-Geo-Country-Name	United States	Full country name
X-Geo-City	San Jose	Client city
X-Geo-Region	CA	Region code
X-Geo-Continent-Code	NA	Continent
X-Geo-Latitude / X-Geo-Longitude	37.3353 / -121.8938	Geolocation
X-Geo-Postal-Code	95113	Postal code
X-Geo-Metro-Code	807	US metro code
X-Geo-ASN	7018	Autonomous System Number
X-Geo-Conn-Speed	broadband	Connection speed class
X-Geo-Conn-Type	wired	Connection type

Azure Front Door Headers

Header	Value	Purpose
X-Azure-ClientIP	Client IP	Client IP address
X-Azure-SocketIP	Client IP	TCP socket source IP
X-Azure-Ref	Encoded ref string	Unique request reference for troubleshooting
X-Azure-FDID	a0a0a0a0-bbbb-cccc-dddd-e1e1e1e1e1e1	Front Door resource identifier
X-Azure-RequestChain	hops=1	Loop detection hop count

Response Headers (Added to Client Responses)

Header	Values	Purpose
X-Cache-Status	HIT, MISS, EXPIRED, STALE, UPDATING	Cache behavior for this request
X-CDN-Edge	cdn-simulator	Identifies this edge node
X-CDN-POP	SJC	Simulated Point of Presence IATA code
X-Served-By	cache-sjc3120-SJC	Simulated cache node in Fastly format
X-Request-ID	UUID (per-request)	Unique request identifier

Device Detection

The simulator detects device type from the User-Agent header using NGINX map directives:

• Mobile: Matches iPhone, Android (non-tablet), iPod, BlackBerry, Opera Mini, IEMobile
• Tablet: Matches iPad, Android tablet, Kindle, PlayBook
• Desktop: Default when neither mobile nor tablet matches

Device type is reflected in:

• CloudFront-Is-Desktop-Viewer / CloudFront-Is-Mobile-Viewer / CloudFront-Is-Tablet-Viewer
• X-Akamai-Device-Characteristics (is_mobile, is_tablet, is_wireless_device fields)

Operations

Changing the Origin Server

ssh azureuser@<PUBLIC_IP>

# Update upstream server
sudo sed -i 's|server .*;|server NEW_HOST:80;|' /etc/nginx/conf.d/cdn-edge.conf

# Clear cache and reload
sudo rm -rf /var/cache/nginx/cdn/*
sudo nginx -t && sudo systemctl reload nginx

Or update origin_host in terraform.tfvars and run terraform apply to reprovision.

Clearing the Cache

ssh azureuser@<PUBLIC_IP>
sudo rm -rf /var/cache/nginx/cdn/*
sudo systemctl reload nginx

Checking Cache Stats

# Object count and disk usage
sudo find /var/cache/nginx/cdn -type f | wc -l
sudo du -sh /var/cache/nginx/cdn

# Recent access log with cache status
tail -20 /var/log/nginx/access.log

Monitoring Under Load

# Real-time connections and socket states
ss -s

# NGINX worker CPU usage
top -bn1 | grep nginx

# Upstream keepalive connections
ss -tn state established dst <ORIGIN_IP> | wc -l

# TIME_WAIT socket count
ss -tn state time-wait | wc -l

Performance Benchmark Results

Verified under 48-hour continuous load test:

Metric	Value
Peak throughput (cached)	172,540 req/s
Sustained throughput (cached)	85,000-103,000 req/s
Peak connections	15,000 concurrent
Cache hit ratio	100% (when warmed)
Memory under load	1.2 GB stable (8% of 16 GB)
CPU at peak	100% (4 cores - CPU is the ceiling)
Errors during 48h test	0
Memory leaks	None detected
Connection leaks	None detected