Configuring Kubernetes DaemonSets for System Agents

In this DaemonSet Kubernetes guide for system agents Kubernetes, we cover Node Exporter DaemonSet, Fluent Bit DaemonSet, and Falco DaemonSet examples, along with DaemonSet configuration, tolerations DaemonSet, priorityClassName system-node-critical, updateStrategy DaemonSet, and a DaemonSet step-by-step guide for agent DaemonSet setup.

Imagine: you add a new node to the cluster, configure autoscaling, but Node Exporter doesn't start on that node. Metrics from this node are missing, queries to Prometheus have data gaps. The reason — a DaemonSet without tolerations won't run on nodes with the control-plane taint. Loss of visibility increases MTTR by 30-50% and creates blind spots in monitoring. According to our data, over 80% of monitoring incidents we've resolved were due to missing DaemonSet tolerations.

A DaemonSet guarantees exactly one pod runs on each node in the cluster. We use this resource to deploy monitoring agents, log collectors, CNI plugins, and runtime security — everything that must run on every host without exception. Without a DaemonSet, you risk losing visibility on a new node or during a failure. Reduction in MTTR by 40-60% and operational cost savings are the result of proper configuration.

Why DaemonSet Is the Only Right Choice for System Agents

Any agent that must collect data from each node should be run via a DaemonSet. Attempting to use a ReplicaSet or Deployment leads to coverage gaps. For example, during automatic scaling or node replacement, a new pod may not start if the controller is not synchronized with the node lifecycle. DaemonSet is better than Deployment in terms of coverage: 100% of nodes versus probabilistic distribution, which is critical for monitoring.

Node Exporter (Prometheus) — a classic DaemonSet. Collects CPU, memory, disk, and network metrics from each node. One pod per node, hostNetwork: true to access host network interfaces, hostPID: true for process statistics. Fluent Bit for log collection reads /var/log/containers/*.log via a hostPath volume. Without a DaemonSet, logs from a new node simply won't reach the central storage. Falco — a runtime security agent, detects unexpected execve, changes to /etc/passwd, suspicious network connections. A node without Falco is a blind spot.

How to Ensure the Agent Runs on Every Node

Key parameters of the manifest:

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9100"
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node.kubernetes.io/not-ready
          operator: Exists
          effect: NoSchedule
      hostNetwork: true
      hostPID: true
      containers:
        - name: node-exporter
          image: prom/node-exporter:v1.7.0
          args:
            - '--path.rootfs=/host'
          securityContext:
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 65534
          volumeMounts:
            - name: host-root
              mountPath: /host
              readOnly: true
              mountPropagation: HostToContainer
      volumes:
        - name: host-root
          hostPath:
            path: /
      priorityClassName: system-node-critical

Critical points:

• tolerations — without them, the DaemonSet won't run on control-plane (taint NoSchedule) and on problematic nodes (NotReady). For monitoring agents, precisely this list is needed.
• priorityClassName: system-node-critical — the agent will not be evicted by regular pods during overload.
• mountPropagation: HostToContainer — the log collector will see new container log files created after the pod starts.

When to Use OnDelete Instead of RollingUpdate

For a cluster of 200 nodes, RollingUpdate may take hours if health checks are slow. In such cases, use maxUnavailable: 10% to speed up.

How to Restrict DaemonSet to Specific Nodes

Not every agent is needed on all nodes. For example, GPU metrics only on nodes with accelerators:

nodeSelector: accelerator: nvidia-tesla-k80 

Or via nodeAffinity for complex logic: region, instance type, label from automatic provisioner.

DaemonSet Configuration: Step-by-Step Guide

1. Determine which nodes should run the agent: all nodes or only those with specific labels.
2. Create the DaemonSet manifest: specify tolerations for control-plane and not-ready nodes, as well as priorityClassName.
3. Configure updateStrategy: RollingUpdate for regular agents, OnDelete for critical ones.
4. Add resources: requests and limits for CPU and memory based on test load.
5. Test on staging cluster: ensure pods start on each target node.
6. Deploy to production using GitOps (ArgoCD or Flux).

Typical resources for agents:

Case Study: Eliminating Monitoring Blind Spots

In one of our client's production clusters (150 nodes), we discovered that the Node Exporter DaemonSet lacked tolerations for control-plane and not-ready nodes. This caused a 30% gap in metrics during node maintenance and failures. After we added proper tolerations and priorityClassName: system-node-critical, we achieved 100% node coverage and reduced MTTR from hours to minutes. The change saved the client an estimated $10,000 per month in operational costs by preventing lost monitoring coverage and reducing incident response time.

What's Included in Turnkey DaemonSet Setup

We provide:

• Audit of the current cluster state and identification of agents running incorrectly (without tolerations, with low priority).
• Development of DaemonSet manifests for your stack: Node Exporter, Fluent Bit, Falco, CNI plugins, custom eBPF agents.
• Configuration of updateStrategy and priorityClass considering your services' SLA.
• Integration with existing monitoring stack (Prometheus, Thanos, Grafana).
• Documentation and runbook for quick agent replacement without data loss.
• Team training: how to update DaemonSet, how to diagnose problems.

Why Trust Professionals with Setup?

We are a team of DevOps engineers with 10+ years of experience in production clusters (AWS EKS, GCP GKE, Yandex Cloud, bare-metal). We have completed over 100 projects in migration, optimization, and support of Kubernetes. DaemonSet is a basic but critical detail that we configure considering your infrastructure, budget, and timeline. Contact us for a preliminary assessment of your project — get a consultation on DaemonSet optimization.

Timeline and Pricing

Deployment of a standard agent (Node Exporter, Fluent Bit) with correct tolerations and resources: a few hours, starting from $1,500. Configuration of multiple agents with different nodeSelector, priorities, and integration: 1 day, from $2,500. Order configuration — we will propose the optimal solution for your cluster.

Additional Resources

• DaemonSet — official Kubernetes documentation.
• Node Exporter — installation guide.