My Thoughts...

Introduction

Multi-cloud networking is complex, but necessary when you want to optimize cost, performance, or geographic redundancy. In this guide, I’ll walk through how I routed traffic from Azure to Linode through Equinix ExpressRoute, including the challenges, missteps, and lessons learned.

This setup enables low-latency, private, and reliable data transfer between Azure and Linode via Equinix’s fabric and BGP configuration.

Architecture Overview

• Azure: Virtual Network (VNet) with ExpressRoute Gateway
• Equinix: Virtual Device and Fabric connection
• Linode: LKE (Linode Kubernetes Engine) with a private subnet
• Protocol: BGP over VLAN

Key Goals:

• Avoid internet hops
• Enable deterministic routing
• Support redundancy

Step 1: Provision ExpressRoute in Azure

1. Create a Virtual Network Gateway with ExpressRoute SKU
2. Link it to a subnet within your Azure VNet
3. Create an ExpressRoute Circuit
4. Choose Equinix as the provider and set the peering location

📌 Tip: Make sure you don’t enable Microsoft peering if you’re only routing to Linode. Use Private peering.

Step 2: Create the Equinix Fabric Connection

1. Log into the Equinix Fabric Portal
2. Create a connection between Azure and a virtual device (Cisco CSR or Palo Alto works well)
3. Assign VLAN tags to each side:
4. A-side: Azure (e.g., 10.10.1.1/30)
5. Z-side: Linode (e.g., 10.10.2.1/30)

💡 Watch for overlapping subnets between Azure and Linode! This caused initial route flaps.

Step 3: Configure Linode for BGP

Linode doesn’t natively support BGP, so you’ll need to:

1. Deploy a router VM (e.g., VyOS or FRRouting)
2. Assign it a static IP on your private Linode subnet
3. Configure BGP neighbor with the Equinix Z-side
4. Advertise Linode CIDRs

Step 4: Exchange Routes with Azure

In Azure: - Use Get-AzExpressRouteCircuit to confirm peering is active - Check learned routes via Get-AzRouteTable - Ensure route tables are associated with subnets in your VNet

From Linode: - Ping test Azure private IPs - Run traceroute to confirm Equinix path is taken

Common Pitfalls

• ❌ Missing BGP ASN on one side — causes peering rejection
• ❌ Incorrect VLAN tags — traffic drops silently
• ❌ Unroutable return path — remember to update both route tables
• ❌ UDP/ICMP tests failing — ExpressRoute doesn’t forward all protocol types by default

Bonus: Test End-to-End Application Traffic

1. Deploy a sample app in Linode’s LKE
2. Use Azure Container App or App Gateway to hit the endpoint
3. Use tcpdump to confirm traffic path
4. Add latency monitoring via Prometheus or Grafana

Conclusion

Routing traffic between Azure and Linode via Equinix ExpressRoute is absolutely possible—but requires surgical attention to BGP, subnets, and physical routing topology.

Once working, the benefits are huge: consistent performance, lower latency, and private, secure inter-cloud communication.

Let me know if you’d like a Terraform version of this configuration or a reusable BGP starter template!

Introduction

Handling millions of requests per second (RPS) through a Kubernetes cluster is not just a matter of adding replicas—it demands deliberate optimization of ingress design, connection handling, autoscaling, and network I/O. This post distills key strategies we used to scale an HAProxy-based ingress to consistently handle 1M+ RPS on Azure Kubernetes Service (AKS).

Ingress Stack Overview

We used the following stack: - HAProxy (with custom config map) - Azure Kubernetes Service (AKS) - Horizontal Pod Autoscaler (HPA) - Node pools tuned for low-latency - Prometheus + Grafana for observability

HAProxy Configuration Essentials

In ConfigMap:

maxconn-global: "100000"
maxconn-server: "10000"
nbthread: "4"
timeout-client: 50s
timeout-connect: 5s
timeout-queue: 5s
timeout-http-request: 10s
timeout-http-keep-alive: 1s
ssl-options: no-sslv3 no-tls-tickets no-tlsv10 no-tlsv11
ssl-ciphers: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256

These settings ensure fast connection handling, low latency, and strict SSL policies.

Scaling Strategy

• Use Dedicated Node Pools for ingress controllers (separate from app workloads)
• Set PodDisruptionBudgets to avoid draining ingress under load
• Use topologySpreadConstraints or podAntiAffinity to prevent all ingress pods landing on one node

HPA Tweaks

• Custom metric: sum(rate(requests[1m]))
• Stabilization window: 30s
• Cooldown: 60s

Ensure metrics server and Prometheus Adapter are tuned to avoid lag in metrics reporting.

Connection and Network Limits

AKS nodes have system limits: - conntrack table: Default is ~131072. You'll need to tune this with sysctl or use node images with extended limits - NIC throughput: Scale with Standard_Dv4/Dv5 node series - Watch out for ConntrackFull and ReadOnlyFilesystem errors on nodes under stress

Observability

Key metrics to monitor: - RPS per pod - Latency P95/P99 - Dropped connections - conntrack usage

Recommended tools: - Prometheus: with haproxy_exporter - Grafana: custom dashboards with alerts - Kubernetes Events: monitor for pod eviction or failed scheduling

Bonus: Simulate Load Without Overcommit

Use wrk, vegeta, or k6 to simulate realistic traffic:

wrk -t12 -c1000 -d60s --latency https://your-ingress.example.com

This helps avoid triggering false autoscaler signals while still stressing the ingress layer.

Conclusion

Building a high-throughput ingress isn’t just about more pods—it’s about smarter topology, system-level tuning, and proactive observability. With the right HAProxy configuration and node awareness, Kubernetes ingress can scale to serve millions of requests per second reliably.

Let me know if you'd like a Helm chart, Terraform config, or Azure-specific node tuning guide to go with this.