Essential Linux Troubleshooting Commands for Production Issues (2026)

When a production alert rings at 2:00 AM saying *"Application server unresponsive, latency exceeding 10 seconds"*, you cannot rely on a GUI. You must SSH into the Linux box and systematically isolate the root cause within minutes.

In my years managing production infrastructure and mentoring DevOps and Application Support engineers, the difference between a junior and senior engineer is troubleshooting methodology.

Here is the systematic methodology and essential command toolkit used by production engineers.

---

The Golden Troubleshooting Flow

Always isolate the resource bottleneck in this order:

1. CPU: Is the system compute-bound?
2. Memory: Is the system out of RAM or aggressively swapping?
3. Disk I/O: Is disk throughput or IOPS saturated?
4. Network: Are sockets exhausted or packet drops occurring?

---

1. Investigating CPU Spikes

Command 1: uptime & Understanding Load Average

```bash

$ uptime

14:22:01 up 45 days, 3:12, 2 users, load average: 8.52, 4.21, 2.10

```

How to interpret: The three numbers represent load averages over 1, 5, and 15 minutes.

• Check core count: nproc
• If you have a 4-core server and load average is 8.52, your CPU is over-saturated by over 200%. If it is 2.10, CPU has ample headroom.

Command 2: top / htop

• Press P in top to sort by CPU usage.
• Press c to display full command paths.
• Check the `%wa` (I/O Wait) metric! If CPU is 90% in %wa, the CPU is not busy calculating; it is stalled waiting for disk or network.

Command 3: pidstat 1

```bash

$ pidstat 1 5

```

Prints process CPU statistics every 1 second for 5 iterations. Helps identify erratic processes that spike intermittently.

---

2. Investigating Memory Leaks & OOM Killer

Command 1: free -h

```bash

$ free -h

total used free shared buff/cache available

Mem: 15Gi 13Gi 850Mi 120Mi 1.8Gi 1.6Gi

Swap: 2.0Gi 1.8Gi 200Mi

```

Crucial Rule: Look at the `available` column, NOT free. Linux uses spare memory for file buffers and caching (buff/cache), which it can reclaim instantly when applications demand it.

Command 2: Checking if OOM (Out Of Memory) Killer killed your process

```bash

$ dmesg -T | grep -i -E "killed process|oom"

```

If Linux runs completely out of memory and swap, the kernel's OOM Killer will forcibly terminate the process with the highest oom_score (often Java or Node.js).

---

3. Investigating Disk Space & I/O Bottlenecks

Command 1: df -h (Disk Space Exhaustion)

```bash

$ df -h

Filesystem Size Used Avail Use% Mounted on

/dev/nvme0n1p1 50G 49G 1.2G 98% /

```

If disk usage reaches 100%, services like MySQL, Nginx, or Docker will immediately fail to write logs and crash.

Command 2: Finding Largest Directories & Files

```bash

$ du -sh /var/log/* | sort -hr | head -n 10

$ find / -type f -size +500M -exec ls -lh {} ; 2>/dev/null

```

Command 3: df -i (Inodes Exhaustion)

Sometimes df -h shows 50% free space, but applications still complain *"No space left on device"*. Check inode usage:

```bash

$ df -i

```

Millions of tiny cache files or session logs can consume all inodes even when gigabytes of disk storage remain free.

Command 4: iostat -xz 1 (I/O Saturation)

```bash

$ iostat -xz 1

```

Look at the `%util` column. If %util is consistently near 100%, your storage disk (EBS/SSD) is at capacity.

---

4. Investigating Network & Port Issues

Command 1: ss -tulpn (Active Listening Ports)

```bash

$ ss -tulpn

```

Replaces legacy netstat. Shows which process is listening on which port and whether a service failed to bind.

Command 2: curl -Iv https://localhost:8080

Test service reachability locally from within the server to verify whether the issue is inside the application or at the external firewall/load balancer.

Command 3: tcpdump for Packet Analysis

```bash

$ sudo tcpdump -i eth0 port 443 -n -c 50

```

---

Real Production Scenario: "Server Unresponsive"

When you get an alert, run this 60-second diagnosis sequence:

```bash

uptime

free -h

df -h

ps aux --sort=-%cpu | head -n 6

ps aux --sort=-%mem | head -n 6

dmesg -T | tail -n 25

```

Mastering these commands transforms you from an engineer who guesses to one who diagnoses with surgical precision.