Comprehensive Hybrid Cloud Infrastructure Setup and Integration Guide

Hybrid Cloud Infrastructure Setup

Imagine: you have a legacy system in a data center with dozens of hardware and license dependencies. Moving it to the cloud is a multi-month risk. But you want to run new microservices on Kubernetes on AWS. The solution is hybrid infrastructure. We configure it turnkey so that on-premise and cloud work as a single cluster with a shared network, DNS, and IAM contour.

Where things usually break

The most common problem is IP address inconsistency. The on-premise network lives in 10.0.0.0/8, the cloud VPC in 172.16.0.0/12, and at first glance everything is fine. But when you add Transit Gateway, peering to multiple VPCs, and corporate VPN, overlapping CIDRs start to silently confuse the router. Packets drop, services become unavailable, and neither side sees an error at its level.

The second pain point is DNS. In a hybrid environment, db.internal.company.com from a cloud pod must resolve to the IP of an on-premise database, and api.svc.cluster.local from an on-premise application must resolve to the ClusterIP of a Kubernetes service. This is solved via Route 53 Resolver with forwarding rules to on-premise DNS, and a Conditional Forwarder on the on-premise side to Route 53. If you don't configure both directions, half the resolutions will fail with NXDOMAIN.

Authentication is also nontrivial. Cloud applications need credentials for on-premise systems and vice versa. IAM Roles for Service Accounts (IRSA) in EKS works great inside AWS, but for on-premise resources you need either Vault with AWS auth backend or secrets via AWS Secrets Manager replicated into the cluster with External Secrets Operator.

Direct Connect or VPN: what to choose?

Source: AWS Direct Connect and Site-to-Site VPN documentation

For production workloads with critical latency, Direct Connect is up to 10x more reliable than VPN. For dev/staging or low volumes, VPN is sufficient.

How we build the hybrid network

Network connectivity is the first thing we design. We use Transit Gateway as a central hub. All VPCs connect to it, not through peering with each other. This scales: add a new VPC — attach it to TGW, routes to the rest appear automatically.

Network topology for a mid-sized project:

On-Premise (10.10.0.0/16)
│
│ Direct Connect / VPN
▼
AWS Transit Gateway
├── Production VPC (10.20.0.0/16)
│   ├── Private Subnet (10.20.1.0/24) — EKS nodes
│   └── Private Subnet (10.20.2.0/24) — RDS
├── Staging VPC (10.30.0.0/16)
└── Shared Services VPC (10.40.0.0/16)
    ├── Route 53 Resolver Inbound Endpoint
    └── Route 53 Resolver Outbound Endpoint

Configuration management with Terraform

All network infrastructure is in code. Example Terraform for Route 53 Resolver:

resource "aws_route53_resolver_endpoint" "outbound" {
  name = "hybrid-outbound-resolver"
  direction = "OUTBOUND"
  security_group_ids = [aws_security_group.resolver.id]

  ip_address {
    subnet_id = aws_subnet.shared_services_a.id
  }

  ip_address {
    subnet_id = aws_subnet.shared_services_b.id
  }
}

resource "aws_route53_resolver_rule" "onprem" {
  domain_name = "internal.company.com"
  name = "forward-to-onprem"
  rule_type = "FORWARD"
  resolver_endpoint_id = aws_route53_resolver_endpoint.outbound.id

  target_ip {
    ip = "10.10.0.53"
    port = 53
  }

  target_ip {
    ip = "10.10.0.54"
    port = 53
  }
}

Routes in the Transit Gateway Route Table are also declarative — no manual clicks in the console that nobody documents.

Kubernetes in a hybrid environment

An EKS cluster with a node group in private subnets has no direct internet access. To pull images from ECR, you need VPC Endpoints for ECR and S3 (image layers are stored in S3). For access to on-premise resources, routes via TGW are used. CoreDNS in the cluster is configured via a ConfigMap with Corefile:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
        }
        forward internal.company.com 10.10.0.53 10.10.0.54
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }