🔥 Kubectl Prof

Profile your Kubernetes applications with zero overhead and zero modifications 🚀

kubectl-prof is a powerful kubectl plugin that enables low-overhead profiling of applications running in Kubernetes environments. Generate FlameGraphs, JFR files, thread dumps, heap dumps, and many other diagnostic outputs without modifying your pods.

✨ Key Features:

• 🎯 Zero modification - Profile running pods without any changes to your deployment
• 🌐 Multi-language support - Java, Go, Python, Ruby, Node.js, Rust, Clang/Clang++
• 📊 Multiple output formats - FlameGraphs, JFR, SpeedScope, thread dumps, heap dumps, and more
• ⚡ Low overhead - Minimal impact on running applications
• 🔄 Continuous profiling - Support for both discrete and continuous profiling modes

This is an open source fork of kubectl-flame with enhanced features and bug fixes.

📋 Table of Contents

• Requirements
• Quick Start
• Installation
• Usage
  • Java Profiling
  • Python Profiling
  • Go Profiling
  • Node.js Profiling
  • Ruby Profiling
  • Clang/Clang++ Profiling
  • Advanced Usage
• How It Works
• Building from Source
• Contributing
• License

📋 Requirements

Supported Languages 💻

Language	Status	Tools Available
☕ Java (JVM)	✅ Fully Supported	async-profiler, jcmd
🐹 Go	✅ Fully Supported	eBPF profiling
🐍 Python	✅ Fully Supported	py-spy
💎 Ruby	✅ Fully Supported	rbspy
📗 Node.js	✅ Fully Supported	eBPF profiling, perf
🦀 Rust	✅ Fully Supported	eBPF profiling
⚙️ Clang/Clang++	✅ Fully Supported	eBPF profiling, perf

Container Runtimes 🐳

• Containerd - --runtime=containerd (default)
• CRI-O - --runtime=crio

🚀 Quick Start

Profile a Java application for 1 minute and save the FlameGraph:

kubectl prof my-pod -t 1m -l java

Profile a Python application and save to a specific location:

kubectl prof my-pod -t 1m -l python --local-path=/tmp

Profile multiple pods using a label selector:

kubectl prof --selector app=myapp -t 5m -l java -o jfr

📖 Usage

☕ Java Profiling

Basic FlameGraph Generation

Profile a Java application for 5 minutes and generate a FlameGraph:

kubectl prof my-pod -t 5m -l java -o flamegraph --local-path=/tmp

💡 Tip: If --local-path is omitted, the FlameGraph will be saved to the current directory.

Alpine-based Containers

For Java applications running in Alpine-based containers, use the --alpine flag:

kubectl prof mypod -t 1m -l java -o flamegraph --alpine

⚠️ Note: The --alpine flag is only required for Java applications.

JFR Output Generation

Using jcmd (default for JFR):

kubectl prof mypod -t 5m -l java -o jfr

Using async-profiler:

kubectl prof mypod -t 5m -l java -o jfr --tool async-profiler

Thread Dump

Generate a thread dump using jcmd:

kubectl prof mypod -l java -o threaddump

Heap Dump

Generate a heap dump in hprof format:

kubectl prof mypod -l java -o heapdump --tool jcmd

Heap Histogram

Generate a heap histogram:

kubectl prof mypod -l java -o heaphistogram --tool jcmd

Available Event Types for Java

When using async-profiler, you can specify different event types:

# CPU profiling (default: ctimer)
kubectl prof mypod -t 5m -l java -e cpu

# Memory allocation profiling
kubectl prof mypod -t 5m -l java -e alloc

# Lock contention profiling
kubectl prof mypod -t 5m -l java -e lock

Supported events: cpu, alloc, lock, cache-misses, wall, itimer, ctimer

---

🐍 Python Profiling

FlameGraph Generation

kubectl prof mypod -t 1m -l python -o flamegraph --local-path=/tmp

Thread Dump

kubectl prof mypod -l python -o threaddump --local-path=/tmp

SpeedScope Format

Generate a SpeedScope compatible file:

kubectl prof mypod -t 1m -l python -o speedscope --local-path=/tmp

---

🐹 Go Profiling

Profile a Go application for 1 minute:

kubectl prof mypod -t 1m -l go -o flamegraph

---

📗 Node.js Profiling

FlameGraph Generation

kubectl prof mypod -t 1m -l node -o flamegraph

💡 Tip: For JavaScript symbols to be resolved, run your Node.js process with the --perf-basic-prof flag.

Heap Snapshot

Generate a heap snapshot:

kubectl prof mypod -l node -o heapsnapshot

⚠️ Requirements: Your Node.js app must be run with --heapsnapshot-signal=SIGUSR2 (default) or --heapsnapshot-signal=SIGUSR1.

If using SIGUSR1:

kubectl prof mypod -l node -o heapsnapshot --node-heap-snapshot-signal=10

📚 Learn more: Node.js Heap Snapshots

---

💎 Ruby Profiling

Profile a Ruby application:

kubectl prof mypod -t 1m -l ruby -o flamegraph

Available output formats:

• flamegraph - FlameGraph visualization
• speedscope - SpeedScope format
• callgrind - Callgrind format

---

⚙️ Clang/Clang++ Profiling

Clang:

kubectl prof mypod -t 1m -l clang -o flamegraph

Clang++:

kubectl prof mypod -t 1m -l clang++ -o flamegraph

---

🎯 Advanced Usage

Specify Container Runtime

kubectl prof mypod -t 1m -l java --runtime crio

Supported runtimes: containerd (default), crio

Continuous Profiling

Profile continuously at 60-second intervals for 5 minutes:

kubectl prof mypod -l java -t 5m --interval 60s

📝 Note: In continuous mode, a new result is produced every interval. Only the last result is available by default.

Custom Resource Limits

Set CPU and memory limits for the profiling agent pod:

kubectl prof mypod -l java -t 5m \
  --cpu-limits=1 \
  --cpu-requests=100m \
  --mem-limits=200Mi \
  --mem-requests=100Mi

Cross-Namespace Profiling

Profile a pod in a different namespace:

kubectl prof mypod -n profiling \
  --service-account=profiler \
  --target-namespace=my-apps \
  -l go

Custom Agent Image

Use a custom profiling agent image:

kubectl prof mypod -l java -t 5m \
  --image=localhost/my-agent-image-jvm:latest \
  --image-pull-policy=IfNotPresent \
  --runtime containerd

Profile Multiple Pods with Label Selector

Profile all pods matching a label selector:

kubectl prof --selector app=myapp -t 5m -l java -o jfr

⚠️ ATTENTION: Use this option with caution as it will profile ALL pods matching the selector.

Control concurrent profiling jobs:

kubectl prof --selector app=myapp -t 5m -l java -o jfr --pool-size-profiling-jobs 5

Target Specific Process

By default, kubectl-prof attempts to profile all processes in the container. To target a specific process:

Using PID:

kubectl prof mypod -l java --pid 1234

Using process name:

kubectl prof mypod -l java --pgrep java-app-process

Capabilities Configuration

For Java profiling, kubectl-prof uses PERFMON and SYSLOG capabilities by default. To use SYS_ADMIN:

kubectl prof my-pod -t 5m -l java --capabilities=SYS_ADMIN

Add multiple capabilities:

kubectl prof my-pod -t 5m -l java \
  --capabilities=SYS_ADMIN \
  --capabilities=PERFMON

Node Tolerations

Profile pods on nodes with taints by specifying tolerations:

Tolerate specific taint:

kubectl prof my-pod -t 5m -l java \
  --tolerations=node.kubernetes.io/disk-pressure=true:NoSchedule

Multiple tolerations:

kubectl prof my-pod -t 5m -l java \
  --tolerations=node.kubernetes.io/disk-pressure=true:NoSchedule \
  --tolerations=node.kubernetes.io/memory-pressure:NoExecute \
  --tolerations=dedicated=profiling:PreferNoSchedule

Toleration formats:

• key=value:effect - Full specification
• key:effect - Any value
• key - Defaults to NoSchedule

---

📚 Get Help

For a complete list of options:

kubectl prof --help

📦 Installation

Using Krew (Recommended) 🔌

Krew is the plugin manager for kubectl.

1. Install Krew (if not already installed)

2. Add kubectl-prof repository and install:

kubectl krew index add kubectl-prof https://github.com/josepdcs/kubectl-prof
kubectl krew search kubectl-prof
kubectl krew install kubectl-prof/prof
kubectl prof --help

---

Pre-built Binaries 📥

Download pre-built binaries from the releases page.

Linux x86_64

wget https://github.com/josepdcs/kubectl-prof/releases/download/1.8.1/kubectl-prof_1.8.1_linux_amd64.tar.gz
tar xvfz kubectl-prof_1.8.1_linux_amd64.tar.gz
sudo install kubectl-prof /usr/local/bin/

macOS

wget https://github.com/josepdcs/kubectl-prof/releases/download/1.8.1/kubectl-prof_1.8.1_darwin_amd64.tar.gz
tar xvfz kubectl-prof_1.8.1_darwin_amd64.tar.gz
sudo install kubectl-prof /usr/local/bin/

Windows

Download the Windows binary from the releases page and add it to your PATH.

🔨 Building from Source

Prerequisites

• Go 1.21 or higher
• Make
• Docker (for building agent containers)

Build Steps

1. Clone and install dependencies:

go get -d github.com/josepdcs/kubectl-prof
cd $GOPATH/src/github.com/josepdcs/kubectl-prof
make install-deps

2. Build the binary:

make build

The binary will be available in ./bin/kubectl-prof

3. Build agent containers (optional):

Modify the DOCKER_BASE_IMAGE property in Makefile, then run:

make build-docker-agents

🔧 How It Works

kubectl-prof launches a Kubernetes Job on the same node as the target pod. The profiling is performed using specialized tools based on the programming language:

Profiling Tools by Language

☕ Java (JVM)

async-profiler - For FlameGraphs and JFR files

• FlameGraphs: --tool async-profiler -o flamegraph (default)
• JFR files: --tool async-profiler -o jfr
• Collapsed/Raw: --tool async-profiler -o collapsed or -o raw
• Event types: cpu, alloc, lock, cache-misses, wall, itimer, ctimer (default)

jcmd - For JFR, thread dumps, heap dumps

• JFR files: --tool jcmd -o jfr (default for jcmd)
• Thread dumps: --tool jcmd -o threaddump
• Heap dumps: --tool jcmd -o heapdump
• Heap histogram: --tool jcmd -o heaphistogram

🐍 Python

py-spy - Low-overhead Python profiler

• FlameGraphs: -o flamegraph (default)
• Thread dumps: -o threaddump
• SpeedScope: -o speedscope
• Raw output: -o raw

🐹 Go & 🦀 Rust

eBPF profiling - Kernel-level profiling

• FlameGraphs: -o flamegraph (default)
• Raw output: -o raw

💎 Ruby

rbspy - Ruby sampling profiler

• FlameGraphs: -o flamegraph (default)
• SpeedScope: -o speedscope
• Callgrind: -o callgrind

📗 Node.js

eBPF profiling (recommended) and perf

• FlameGraphs: -o flamegraph (default)
• Raw output: -o raw
• Heap snapshot: -o heapsnapshot

💡 Tip: For JavaScript symbol resolution, run Node.js with --perf-basic-prof flag
💡 Tip: For heap snapshots, run Node.js with --heapsnapshot-signal flag

⚙️ Clang/Clang++

perf (default) and eBPF profiling

• FlameGraphs: -o flamegraph
• Raw output: -o raw

---

📊 Raw Output Format

The raw output is a text file containing profiling data that can be:

• Used to generate FlameGraphs manually
• Visualized at speedscope.app

---

🔄 Profiling Modes

Discrete Mode (default)

• Single profiling session
• Result available when profiling completes
• Usage: -t 5m

Continuous Mode

• Multiple results at regular intervals
• Only the last result is available by default
• Client responsible for storing all results
• Usage: -t 5m --interval 60s

---

🎯 Process Targeting

By default, kubectl-prof profiles all processes in the target container matching the specified language.

Warning example:

⚠ Detected more than one PID to profile: [2508 2509]. 
  It will attempt to profile all of them. 
  Use the --pid flag to profile a specific PID.

Target a specific process:

• By PID: --pid 1234
• By name: --pgrep process-name

---

🔐 Capabilities

For Java profiling, kubectl-prof uses PERFMON and SYSLOG capabilities by default.

According to the Kernel documentation, these capabilities should be sufficient for collecting performance samples.

To use SYS_ADMIN instead:

kubectl prof my-pod -t 5m -l java --capabilities=SYS_ADMIN

Add multiple capabilities:

kubectl prof my-pod -t 5m -l java \
  --capabilities=SYS_ADMIN \
  --capabilities=PERFMON

---

🏷️ Node Tolerations

By default, the profiling agent pod is scheduled only on nodes without taints. For nodes with taints, specify tolerations:

Toleration formats:

• key=value:effect - Full specification
• key:effect - Any value
• key - Defaults to NoSchedule

Examples:

# Single toleration
kubectl prof my-pod -t 5m -l java \
  --tolerations=node.kubernetes.io/disk-pressure=true:NoSchedule

# Multiple tolerations
kubectl prof my-pod -t 5m -l java \
  --tolerations=node.kubernetes.io/disk-pressure=true:NoSchedule \
  --tolerations=node.kubernetes.io/memory-pressure:NoExecute \
  --tolerations=dedicated=profiling:PreferNoSchedule

🤝 Contributing

We welcome contributions! Please refer to Contributing.md for information about how to get involved.

We welcome:

• 🐛 Bug reports
• 💡 Feature requests
• 📝 Documentation improvements
• 🔧 Pull requests

---

👥 Maintainers

• Josep Damià Carbonell Seguí - [email protected]

Special Thanks 🙏

Original author of kubectl-flame:

• Eden Federman - [email protected]
• Verizon Media Code

---

📄 License

This project is licensed under the terms of the Apache 2.0 open source license. Please refer to LICENSE for the full terms.