Setting Up Continuous Profiling (Pyroscope/Parca)

Traces show a request took 800 ms. Metrics confirm p99 tripled after a deploy. But neither explains which line of code consumes 40% CPU. For that, you need a profiler running continuously in production. Continuous Profiling — the constant recording of profiles (flamegraphs) of all processes. Over 5 years of practice, we've found hundreds of bottlenecks: from JSON serialization in hot paths to memory leaks in middleware. Implementing Pyroscope or Parca not only finds problems but also prevents regressions with every deploy. Discuss Continuous Profiling for your project and get a consultation.

Why Continuous Profiling Over One-Shot Profilers?

Let's see what continuous monitoring brings. A one-shot py-spy top gives a snapshot here and now. Continuous profiling gives history. If today p95 CPU is 70% and a week ago it was 50%, the flamegraph shows which commit caused the jump. Without history, you'll guess when degradation started. Moreover, Pyroscope and Parca help investigate incidents 2–3 times faster thanks to a ready time axis. You can see how the flamegraph changed with each deploy — reducing MTTR from hours to minutes. Based on our projects, debugging time savings reach 40%, and infrastructure costs drop by 15–20%.

What Problems Does Continuous Profiling Solve?

Let's look at specific scenarios.

Memory Leaks Invisible in Metrics

Standard container_memory_usage_bytes shows growth but not the culprit. A heap profiler (Pyroscope ProfileInuseObjects) pinpoints exactly which object and which function holds memory. In one project, we found a leak in middleware that raised the process RSS from 200 MB to 2 GB in 6 hours — the flamegraph immediately pointed to code misusing context.Context.

CPU Regressions After Deploy

Roll out a new version — CPU spikes 300%. A flamegraph 'before' and 'after' shows which function slowed down. Typical finds: accidental debug mode, log level changes, a new dependency with heavy regexps.

Correlation with Traces

When a slow trace indicates a 500 ms span, the flamegraph for that period shows which function inside the span took 80% of the time. Pyroscope supports middleware for OpenTelemetry and Jaeger — profile.id is written to the span automatically.

Which Tool to Choose: Pyroscope or Parca?

Get expert advice on tool selection and setup — contact us for a consultation.

How We Set Up Continuous Profiling

Pyroscope: Pull Mode (eBPF Agent)

The Pyroscope Agent runs as a DaemonSet, reading profiles directly from the kernel via eBPF without application changes. Requires Kubernetes 1.19+ and Linux kernel 4.9+.

# pyroscope-values.yaml for Helm pyroscope:
replicaCount: 1
storage:
  backend: s3
  s3:
    bucket: my-pyroscope-profiles
    region: eu-west-1
agent:
  enabled: true
  tolerations:
    - operator: Exists # deploy to all nodes, including system
config:
  discovery.relabel "pods" {} # auto-discovery of pods

Pyroscope: Push Mode (SDK)

The application explicitly sends profiles to Pyroscope. This gives finer control over labels and lower overhead for specific services.

Go SDK:

import "github.com/grafana/pyroscope-go"

func main() {
    pyroscope.Start(pyroscope.Config{
        ApplicationName: "order-service",
        ServerAddress: "http://pyroscope:4040",
        Tags: map[string]string{
            "version": os.Getenv("APP_VERSION"),
            "region": os.Getenv("REGION"),
        },
        ProfileTypes: []pyroscope.ProfileType{
            pyroscope.ProfileCPU,
            pyroscope.ProfileAllocObjects,
            pyroscope.ProfileAllocSpace,
            pyroscope.ProfileInuseObjects,
            pyroscope.ProfileInuseSpace,
        },
    })
    // ...
}

ProfileAllocObjects and ProfileInuseObjects are heap profiles for finding memory leaks. They helped find the middleware leak that raised RSS from 200MB to 2GB in 6 hours. Explore Pyroscope for deeper understanding.

Parca: eBPF-Oriented Alternative

Parca is 100% built on eBPF, more minimalistic, and requires no code changes.

helm install parca parca/parca \
  --set parca.config.object_storage.bucket.type=S3 \
  --set parca.config.object_storage.bucket.config.bucket=my-parca-profiles

Implementation Process

What’s Included in the Work

• Architecture and configuration documentation (runbook).
• Access to dashboards and profile sources.
• Team training (30 min demo session).
• Support for 2 weeks after implementation.

Common Implementation Mistakes

• Sampling too frequently (1000 Hz). Causes excessive overhead (up to 10%). We recommend 100 Hz for CPU, 50 Hz for heap.
• Storing profiles on local PVC. Easily lost; use S3 with a retention policy.
• Missing trace correlation. Without profile.id, the flamegraph sits isolated — drastically losing value.
• Not excluding eBPF agent from nodes with high I/O load. In some cases it can cause latency; use tolerations for isolation.

Setup time ranges from 1 day (eBPF pull-mode with no application changes) to 3 days (push-mode SDK for all services plus trace correlation and Grafana datasource). Pricing is custom after a detailed assessment. To get a consultation on implementation, contact us — we’ll evaluate your project and propose the optimal architecture.