From a Single EC2 Instance to a 14-Node Multi-Cloud K3s Cluster: Networking, Firewalls, Self-Healing, and DevOps
Back when I first started this blog, everything ran on one tuned EC2 t1.micro instance sharing memory between Apache, PHP, and MySQL. Over the years that single VM grew into a 14-node multi-cloud Kubernetes (K3s) cluster running across Oracle Cloud Infrastructure and Google Cloud Platform Always Free tiers. Instead of relying on expensive managed services, every layer of networking, firewall filtering, ingress TLS, media storage, observability, and disaster recovery is built declaratively and maintained by closed-loop self-healing automation.
14-Node Multi-Cloud Compute and Namespace Isolation
Rather than concentrating every service on one cloud account or one CPU architecture, capacity is spread across three OCI tenancies and two independent GCP projects so no single outage takes everything offline:
- ARM64 Control Plane and Core Workloads (6 OCI Ampere A1 nodes, 2 OCPU and 12 GB RAM each): Hosts the cluster control plane, database workloads, multi-replica WordPress pods, observability backends, and main ingress routing.
- AMD64 Micro and Edge Capacity (6 OCI Micro and 2 GCP
e2-micronodes): Distributes cluster quorum across providers, adds regional edge capacity in North America (us-east1andus-west1), and allows scheduling botharm64andamd64containers. - Guardrails by Default: Every namespace enforces
ResourceQuota,LimitRange, least-privilege RBAC, and Pod Security Standards (baseline). Floating:latestimage tags are disallowed in favor of fixed semantic tags or immutable digests so deployments stay predictable.
Cross-Cloud WireGuard Mesh and Layered Firewalls
Running pods across two cloud providers requires strict packet-path control and multiple firewall boundaries:
- Encrypted WireGuard Mesh and MTU Alignment: Node-to-node traffic traverses a kernel-space WireGuard mesh across public internet links. Tunnel MTUs are aligned explicitly for the outer encapsulation and VXLAN/Flannel pod overlay so TLS handshakes never stall on path MTU discovery issues.
- Host Firewall Hardening (
nftablesandiptables): Alongside strict port allowlists on every node, firewall rules explicitly drop pod traffic targeting cloud instance metadata services to prevent SSRF credential exposure. - Zero-Trust Kubernetes
NetworkPolicy: Namespaces default to denying all unmatched ingress and egress flows, opening only documented paths for DNS, Traefik ingress, database ports, observability scrapes, and S3 object storage endpoints.
Edge Security, HA WordPress with S3 Sync, and Self-Hosted Web Tools
In front of the cluster, every HTTP request passes through multiple verification and caching controls before touching an application pod:
- Cloudflare Strict TLS and Authenticated Origin Pulls (mTLS): Edge traffic connects to Traefik ingress with TLS Strict mode, rate limiting, WAF rules, and origin mTLS client-certificate verification so backend nodes reject direct unauthenticated scans.
- Apache Hardening and Canonical SEO Routing: Native Apache rules enforce canonical redirects to
https://gerson.barreiros.nom.br, blockxmlrpc.php, comment submission endpoints, and hidden dotfiles, prevent script execution inside writable upload directories, and purge stale Cloudflare cache automatically after updates. - HA Stateful Blog and OCI Object Storage Media Sync: WordPress runs with multiple replicas scheduled across distinct nodes, backed by an isolated MariaDB database namespace (alongside CloudNativePG capabilities in the data layer) and Apache metrics sidecars. Uploaded media is redirected via HTTP 301 to OCI Object Storage while periodic CronJobs perform bidirectional additive sync across pod volumes without needing expensive shared block storage.
- Isolated Digital Tools Platform (
/ferramentas/): The same cluster also hosts 19 privacy-first, zero-tracking web utilities across dedicated subdomains (covering PDF editing, IT-Tools, CyberChef, encrypted notes with PrivateBin, local image compression with Squoosh and Mazanoke, P2P file sharing via PairDrop, Excalidraw diagrams, EXIF stripping, and Display Tester), all protected by wildcard/SAN TLS certificates via automated DNS-01 challenges.
Three-Layer Self-Healing, Full Observability, and Three-Region DR
Running production workloads on small Always Free VMs leaves zero room for stuck pods, runaway logs, or unmonitored disk growth. Recovery and visibility run continuously:
- Host-Level Watchdogs (Every 15 Minutes): A
systemdtimer across every node verifies Kubernetes service and container runtime responsiveness (crictl info), auto-prunes unused container images and oversized pod logs above 70% root disk utilization without touching persistent PVC data, and restores metadata firewall rules if anything drifts. - Cluster Manifest Reconciliation and Autonomy Controller: Declarative operations bundles self-apply on the control plane, while an in-cluster controller runs every 60 seconds to keep CoreDNS and metrics healthy, purge
Evictedor stuck pods, audit TLS renewal windows, check canonical site availability, and emit structured JSON logs. - Full Telemetry Stack: VictoriaMetrics and node, Apache, and blackbox exporters track golden signals; Loki and Promtail centralize cluster logs; Tempo and OpenTelemetry collect distributed traces; and vmalert, Alertmanager, and Grafana dashboards tie metrics, backup status, and security checks together.
- Three-Location Multi-Cloud Backups and Weekly DR Rehearsals: Daily database, etcd, and media snapshots are verified locally with
gzip -tbefore replicating to three separate locations: OCI Object Storage S3 plus two independent GCP Cloud Storage buckets inus-east1andus-west1, complemented by automated disk snapshots on GCP edge nodes and weekly sandbox restore rehearsals (--dry-run=server).
Moving from one small cloud instance to a multi-cloud cluster changed the tooling, but the operational goal stayed practical: keep every component documented in code, monitor the signals that catch failures early, and automate recovery cleanly.
Comments are Disabled