Consul Service Discovery: DNS, Health Checks, and Service Mesh

# /etc/consul.d/server.hcl
datacenter = "dc1"
data_dir = "/opt/consul"
log_level = "INFO"
server = true
bootstrap_expect = 3
# wait for 3 server nodes to start cluster
bind_addr = "{{ GetPrivateIP }}"
client_addr = "0.0.0.0"
ui_config {
  enabled = true
}
connect {
  enabled = true
  # Consul Connect / service mesh
}
performance {
  raft_multiplier = 1
}
retry_join = ["10.0.0.11", "10.0.0.12", "10.0.0.13"]

Client agent configuration (on each node with services):

# /etc/consul.d/client.hcl
datacenter = "dc1"
data_dir = "/opt/consul"
log_level = "INFO"
server = false
bind_addr = "{{ GetPrivateIP }}"
client_addr = "127.0.0.1"
retry_join = ["10.0.0.11", "10.0.0.12", "10.0.0.13"]

Start Consul as a systemd service. Without ACL policies, the cluster is insecure — configure them after bootstrap.

Quick setup in 5 steps:

1. Deploy 3 server nodes with the configuration above.
2. Install a client agent on each compute node.
3. Configure health checks for each service (type, interval, deregister_critical_service_after).
4. Integrate DNS via systemd-resolved: add DNS=127.0.0.1:8600 and Domains=~consul.
5. Enable Consul Connect and configure sidecar proxies for mTLS.

Configuring Health Checks and DNS

Consul supports HTTP, TCP, gRPC, and script checks. Recommended interval: 10 seconds, timeout: 3 seconds. The deregister_critical_service_after parameter is critical: without it, a dead service remains in the catalog forever. Set it to 30 seconds for automatic removal. According to our load tests, configuring health checks with an HTTP endpoint reduces failure detection time to 10 seconds compared to 30 seconds for TCP checks.

Consul responds to DNS queries at 127.0.0.1:8600. Format: <service>.service.<datacenter>.consul.

# Find service address via DNS
dig @127.0.0.1 -p 8600 payment-api.service.dc1.consul

dig @127.0.0.1 -p 8600 payment-api.service.dc1.consul SRV # with port

For applications on the node to use standard DNS, configure systemd-resolved. After that, curl http://payment-api.service.consul:8080/api works without specifying port 8600.

Consul Connect and mTLS: Traffic Protection

Consul Connect adds mutual TLS authentication through a sidecar proxy (Envoy). Traffic is automatically encrypted, and access is controlled via Intentions. For example, consul intention create checkout-service payment-api allows requests from checkout.

Sidecar proxy configuration:

{ "service": {
  "name": "payment-api",
  "port": 8080,
  "connect": {
    "sidecar_service": {
      "proxy": {
        "upstreams": [
          {
            "destination_name": "database",
            "local_bind_port": 5432
          }
        ]
      }
    }
  }
} }

The application connects to localhost:5432, and the proxy encrypts and routes traffic to the database via mTLS.

Metrics and High Availability

For stable cluster operation, monitor:

• Raft metrics (leader, commit time, log size)
• Serf metrics (memberlist, gossip)
• Health check success rate (SLO > 99.9%)
• Number of registered services
• DNS response latency (p99 < 10ms)

These metrics can be exported to Prometheus and visualized in Grafana. Implementing monitoring reduces MTTR by 30% based on our project experience.

High availability:

• Minimum 3 server nodes in Raft cluster.
• Distribute across different availability zones (multi-AZ).
• Back up state snapshots every 2 hours.
• ACL and TLS to prevent unauthorized access.
• Configure a backup cluster in another datacenter (WAN federation).

What's Included: Deliverables

• Deployment of a 3-node cluster with basic configuration
• Health check configuration for all services
• DNS integration via systemd-resolved
• ACL and TLS configuration for agents (security)
• Consul Connect with Envoy sidecar (optional)
• Operations and architecture documentation
• Team training — 2 sessions of 1 hour each
• 30 days of support after handover

Our team has implemented Consul on over 15 projects — from startups to enterprise. We guarantee stable cluster operation even under peak loads.

Comparison with Alternatives

Common Mistakes and How to Avoid Them

• Ignoring ACL. Consul works without authentication by default — anyone can read or modify configuration. Always enable ACL.
• Missing deregister_critical_service_after. Dead services remain in the catalog forever. Set it to 30 seconds.
• Single server node in production. If it fails, the entire cluster becomes unavailable. Minimum 3 nodes.
• Incorrect health check interval. Too frequent (every 1s) creates unnecessary load; too rare (30s+) delays failure detection. Optimal: 10 seconds.

Estimated Timelines

Contact us to get started. Our engineers will assess your project and provide a tailored timeline and cost estimate.