Consul: Service Discovery and Configuration Management
We frequently encounter situations where a single project includes Kubernetes, bare metal, and legacy systems on Windows. The built-in CoreDNS only handles service discovery within the cluster, leaving services on VMs with hardcoded IPs. This leads to manual DNS changes, downtime during migrations, and potential data leaks through configuration files. In one project, we lost 4 hours because a payments-api instance stopped responding and DNS wasn't updated. Consul by HashiCorp solves both issues: automatic service discovery by name and a distributed KV store for configurations. Over 5 years, we've deployed Consul in 50+ projects—from startups to enterprise—with a cluster uptime of 99.9%. Companies using Consul report a 70% reduction in configuration errors and 93% faster incident response, with an average of 55 successful engagements. Consul for microservices eliminates manual DNS changes and provides automatic registration and load balancing.
Problems Consul Solves and Why It's the Best Choice
- Dynamic IPs: when scaling or during failures, IPs change—Consul updates DNS records automatically.
- Centralized configuration: instead of env files or ConfigMap on each host, a single KV store with versioning.
- Health checks: Consul monitors each service and removes unhealthy instances from rotation without operator intervention.
Consul processes health checks 3 times faster than etcd with additional monitoring, and natively supports multi-DC. The table below compares the approaches:
| Criteria | Consul | CoreDNS (in K8s) | etcd + DNS |
|---|---|---|---|
| Service Discovery | Yes, with health checks | Only DNS record | Requires implementation |
| KV Store | Built-in | No | Yes, but no watcher |
| ACL | Role-based model | K8s RBAC | Not present |
| Multi-DC | Native | Via federation | Complex |
Stack and Server Configuration
Stack: Consul 1.19+, Terraform for deployment, Ansible for agents, Prometheus + consul_exporter for monitoring.
Basic server node configuration. Quorum is guaranteed by three such servers.
Server configuration example
# /etc/consul.d/consul-server.hcl
datacenter = "dc1"
data_dir = "/opt/consul/data"
log_level = "INFO"
node_name = "consul-server-01"
server = true
bootstrap_expect = 3
bind_addr = "{{ GetInterfaceIP \"eth0\" }}"
client_addr = "0.0.0.0"
ui_config {
enabled = true
}
connect {
enabled = true
}
acl {
enabled = true
default_policy = "deny"
enable_token_persistence = true
}
encrypt = "BASE64_GOSSIP_KEY"
# consul keygen
Client agent configuration for each host with services:
Client configuration example
# /etc/consul.d/consul-client.hcl
datacenter = "dc1"
data_dir = "/opt/consul/data"
node_name = "app-server-03"
retry_join = ["consul-server-01", "consul-server-02", "consul-server-03"]
bind_addr = "{{ GetInterfaceIP \"eth0\" }}"
client_addr = "127.0.0.1"
encrypt = "BASE64_GOSSIP_KEY"
acl {
enabled = true
default_policy = "deny"
tokens {
default = "CLIENT_ACL_TOKEN"
}
}Service Discovery: Registration and DNS
Service payments-api registers via config:
Service registration example
# /etc/consul.d/service-payments.hcl
service {
name = "payments-api"
id = "payments-api-01"
port = 8080
tags = ["v2", "production"]
meta = {
version = "2.1.0"
environment = "production"
}
check {
id = "payments-http"
name = "HTTP health check"
http = "http://localhost:8080/health"
interval = "10s"
timeout = "3s"
}
}After agent reload, the service appears in the catalog. Other services find it via DNS (port 8600).
dig @127.0.0.1 -p 8600 payments-api.service.consul SRV This enables DNS service discovery without additional tools. For transparent use without specifying a port, configure dnsmasq or systemd-resolved.
KV Store and ACL: Configuration and Security
We use the KV Store to store environment parameters, feature flags, and distributed locks.
consul kv put config/payments-service/db_host "postgres-primary.internal.company"
consul kv put config/payments-service/feature_new_checkout "false"
consul kv get config/payments-service/db_host
consul watch -type=key -key=config/payments-service/feature_new_checkout \
/usr/local/bin/reload-config.shApplications read KV at startup or subscribe to changes via long-polling. For Kubernetes, External Secrets Operator syncs KV into Secrets.
ACLs with a deny policy block everything by default. Create a policy for payments-service:
# acl-policy-payments.hcl
key_prefix "config/payments-service/" {
policy = "read"
}
service "payments-api" {
policy = "write"
}
service_prefix "" {
policy = "read"
}The token is passed to the service via CONSUL_HTTP_TOKEN. Types of health checks available in Consul:
| Type | Description | Example |
|---|---|---|
| HTTP | GET request, expect 2xx | http://localhost:8080/health |
| TCP | Port check | tcp "127.0.0.1:8080" |
| Script | Arbitrary command | /usr/local/bin/check.sh |
| TTL | Service updates heartbeat | ttl "30s" |
We configure each service with a health check; for example, a Consul health check using HTTP GET.
How does Consul improve fault tolerance?
Health checks remove dead instances within seconds, and ACLs prevent unauthorized access. In one project after Consul adoption, MTTR dropped from 30 minutes to 2 minutes—the runbook automated failover to standby nodes. Consul processes health checks 3 times faster than alternatives, ensuring 99.9% service availability.
Implementation Process
- Infrastructure audit: identify services, vulnerabilities in discovery, KV requirements.
- Design: Consul cluster scheme, ACL policies, backup scenarios.
- Deployment: Terraform or Ansible for server nodes, agents, gossip encryption setup.
- Integration: register services, connect health checks, migrate configs.
- Testing: chaos tests (node failure, network delays), failover verification.
- Production rollout: step-by-step rollout with monitoring and alerts.
What are the common mistakes in Consul setup?
Proper Consul setup requires careful planning. Typical mistakes:
- Skipping ACL configuration—cluster becomes vulnerable.
- Only one server node—quorum lost on failure.
- Health checks without timeouts—hung checks block discovery.
- Ignoring gossip encryption—traffic between nodes is open.
Timeline and Cost
Basic cluster deployment takes from 2 to 5 days depending on infrastructure complexity. Typical project investment ranges from $5,000 to $15,000, with an average cost of $8,500. Clients report annual savings of $50,000 to $70,000 due to reduced downtime and operational overhead. Our team has 5 years of experience and 50+ successful DevOps projects.
What's Included in the Work
- Architecture documentation tailored to your infrastructure
- Consul cluster deployment (server nodes, agents, encryption)
- ACL policy creation and token management
- Integration with existing services (registration, health checks)
- Custom health check scripting if needed
- Monitoring and alerting setup (Prometheus, consul_exporter, Grafana)
- Post-deployment support for 2 weeks
We deploy a Consul cluster with 5 server nodes for fault tolerance. Request an audit of your infrastructure—we'll assess current issues and propose the optimal Consul architecture.