Linux

Linux Process and Performance Troubleshooting for Busy Servers

When a Linux server becomes slow, the first instinct is often to look at CPU percentage.

That is useful, but it is not enough. A server can feel slow because of CPU saturation, memory pressure, storage latency, network delays, or a single application thread doing too much work.

A good investigation starts broad and becomes more specific.

Check the load and uptime

Start with:

uptime

The load average is a measure of runnable and waiting work, not a direct percentage of CPU usage.

On a many-core server, a load of 4 means something very different from a load of 4 on a two-core machine.

Check the CPU count.

nproc
lscpu

Find the busy processes

Use ps for a snapshot.

ps -eo pid,ppid,user,%cpu,%mem,stat,etime,cmd --sort=-%cpu | head -20

This gives you a useful first ranking without requiring an interactive tool.

For memory-heavy processes, sort differently.

ps -eo pid,user,%mem,rss,vsz,cmd --sort=-%mem | head -20

RSS is generally more useful than virtual size when you are trying to understand physical memory pressure.

Watch the system live

top is available on almost every Linux server.

top

Press P to sort by CPU and M to sort by memory in many implementations.

The exact keyboard commands can vary, so check the help if needed.

Look at one process

Once you have a suspicious PID, inspect it directly.

ps -p 1234 -o pid,ppid,user,%cpu,%mem,stat,etime,cmd

You can also inspect its threads.

top -H -p 1234

This is important when an application is mostly single-threaded.

The process may show high CPU while only one thread is actually busy.

Understand load average

Load average is often misunderstood.

On Linux it includes tasks that are runnable and tasks stuck in uninterruptible sleep, which commonly includes certain I/O waits.

That means high load does not automatically mean high CPU usage.

Compare load with CPU utilization, I/O statistics, and process states before deciding what the bottleneck is.

Check memory pressure

Use:

free -h

Linux uses otherwise unused memory for cache, so seeing low “free” memory is not automatically a problem.

Pay attention to available memory and swap activity instead.

You can inspect swap with:

swapon --show

And watch memory behavior over time:

vmstat 1

Check I/O at the same time

A slow application may be waiting on storage rather than consuming CPU.

Use:

iostat -xz 1

If device utilization and latency rise while the application slows down, storage becomes a stronger suspect.

Do not stop there. Identify which process is responsible.

pidstat -d 1

Use perf when the question gets deeper

When a process is CPU-bound and ordinary tools are not enough, perf can show where CPU time is going.

A simple system-wide sample is:

sudo perf top

For a specific command:

sudo perf record -g -- ./my-program
sudo perf report

The call graph can reveal whether time is spent in application code, libraries, synchronization, memory operations, or kernel work.

Sampling is not magic

A profile tells you where sampled CPU time was observed. It does not automatically explain why that code exists or whether changing it will improve the whole workload.

For example, a JSON-heavy service may spend significant CPU time allocating memory or formatting strings.

The correct fix might be application-level batching rather than a kernel setting.

Check context switches and threads

Highly concurrent workloads can spend meaningful time switching between threads.

pidstat can provide additional process statistics.

pidstat -w 1

If a workload has an unusually high number of context switches, investigate whether concurrency is useful or simply creating contention.

More threads do not automatically mean more throughput.

Inspect systemd services

If the busy process belongs to a service, connect the PID back to systemd.

systemctl status myapp

Then inspect recent logs.

journalctl -u myapp -b -n 100

Performance problems often become obvious when an application log is viewed next to CPU and I/O metrics.

Avoid the “CPU is only 30%” trap

A process can be limited by one busy core while the machine reports plenty of total CPU capacity.

If a server has 32 logical CPUs and one thread is saturated, overall utilization may look low.

That is why per-thread inspection matters for latency-sensitive workloads.

Build a performance timeline

Do not collect random commands forever. Pick a short interval and record what changes.

date
uptime
free -h
iostat -xz 1 3
vmstat 1 3

Then capture the top processes.

ps -eo pid,%cpu,%mem,stat,cmd --sort=-%cpu | head -15

Repeat during the slow period and compare the results.

The practical workflow

Start with load, CPU, memory, and I/O.

Find the process that correlates with the slowdown.

Determine whether the process is CPU-bound, blocked on I/O, short on memory, or waiting on another service.

Only then move to perf, application profiling, kernel tracing, or configuration changes.

The goal is not to collect the most metrics. The goal is to reduce uncertainty until the bottleneck is specific enough to fix.