Small Linux Habits That Prevent Big Production Problems

Linux administration experience is not measured only by how many commands someone knows. On real servers, what often matters more is the order in which commands are used, what gets checked before anything is changed, and what evidence is preserved when an incident occurs. Habits that take only a few seconds can separate a fast diagnosis from an hour of blind trial and error, and they can also prevent a rushed intervention from destroying the very clues needed to find the root cause.

The key Linux habits in 30 seconds

• Before restarting a service, capture its state, logs, sockets, and available resources.
• journalctl becomes far more useful when filtered by service, boot, priority, or time range.
• For storage issues, findmnt and lsblk often provide more context than df alone.
• In scripts, set -e is not enough: exit codes, pipelines, and pipefail still matter.
• Comparing recent changes, preserving evidence, and adding dry runs can prevent many operational mistakes.

There is nothing particularly sophisticated about these practices. That is exactly why they work. They do not require changing distributions, deploying a new observability platform, or learning another language. They fit into commands administrators already use every day.

The real change is moving from a reactive terminal workflow, where commands are tried until something appears to work, to a process where you observe, narrow down, verify, and only then change the system.

1. Read the system before touching it

When a service goes down, one of the most common instincts is still:

systemctl restart nginx

That may fix the symptom.

It may also erase part of the state that explained the problem, generate fresh log entries, and completely alter the scene before anyone has inspected it.

A better order would be:

systemctl status nginx
journalctl -u nginx --since "-15 min"
ss -lntp
df -h
free -hCode language: JavaScript (javascript)

systemctl status gives you the current state of the unit and recent journal entries. journalctl lets you inspect the exact time window you care about. ss confirms which sockets are actually listening.

Only then does it make sense to decide whether a restart is appropriate.

The principle is simple but extremely valuable: capture first, intervene second.

2. Always ask what changed

Many apparently mysterious incidents have very ordinary causes:

• a package update;
• a deployment;
• a configuration change;
• a new certificate;
• a DNS modification;
• a firewall rule;
• a new kernel;
• a filesystem mounted differently.

That is why one of the first questions should be:

What changed since this last worked?

If the configuration is versioned:

git status
git diff
git log --since="2 days ago" --onelineCode language: JavaScript (javascript)

On Debian and Ubuntu, recent package changes can be investigated through APT and dpkg logs. RPM-based distributions provide equivalent history through their package managers.

On systems managed through automation, Ansible logs, CI/CD history, or deployment platforms may be even more useful.

Investigating recent change reduces the search space dramatically. It is usually more efficient than starting with an elaborate theory about memory, networking, or kernel behavior without any supporting evidence.

3. Learn to filter journalctl, not just run it

Running:

journalctl -xe

can return so much information that the actual problem disappears into the noise.

The systemd journal supports much more focused queries.

For a single service:

journalctl -u nginx

For the last 20 minutes:

journalctl -u nginx --since "-20 min"Code language: JavaScript (javascript)

For the current boot only:

journalctl -b

For the previous boot:

journalctl -b -1

For errors and more severe messages:

journalctl -p err -b

For kernel messages only:

journalctl -k -b

To search for a pattern:

journalctl -u nginx -g "timeout"Code language: JavaScript (javascript)

The ability to inspect the previous boot is especially useful when a server has just restarted. Instead of losing the incident trail, you can review what happened immediately beforehand.

The useful habit is not simply “check the logs.” It is check the right logs for the right time window.

4. Check failed services as part of every review

One small command deserves to be used much more often:

systemctl --failed

It quickly shows units that ended up in the failed state.

This can reveal problems that have not yet triggered an obvious outage: a mount that failed, an auxiliary service that broke, or a unit that failed during boot without immediately affecting users.

systemd keeps the failed state available for inspection until the unit is restarted, stopped, or explicitly reset.

That is why you should not automatically run:

systemctl reset-failed

before investigating.

Clearing failed state without looking at it first is another way of deleting evidence.

5. Do not rely only on df -h for storage problems

df -h answers one question very well: how much space is being used on mounted filesystems.

But storage issues can exist at other layers.

Add:

lsblk

to understand block devices, partitions, and their relationships, and:

findmnt

to inspect mounted filesystems.

To find which filesystem contains a specific path:

findmnt --target /var/lib/dockerCode language: JavaScript (javascript)

This is much more precise than visually scanning dozens of mounts.

On systems with LVM, multiple disks, bind mounts, containers, NFS, or cloud volumes, distinguishing device, filesystem, and mount point prevents many mistakes.

It is also worth remembering that names such as /dev/sda and /dev/sdb may change when hardware changes. For persistent configuration, UUIDs or labels are often more appropriate where supported.

6. Verify what is actually listening

When an application says that “the port is open,” the operating system can tell you whether that is really true.

A particularly useful command is:

ss -lntp

It shows listening TCP sockets and, with sufficient permissions, the associated processes.

For UDP:

ss -lunp

For active TCP connections:

ss -tnp

This is more reliable than assuming an application is listening because its configuration says it should be.

If nginx is expected to listen on :443 but ss shows nothing, the problem exists before external networking enters the picture.

If it is listening only on:

127.0.0.1:443Code language: CSS (css)

when it should be reachable externally, you already have a strong clue.

A good sysadmin habit is to verify effective state, not only declared configuration.

7. Distinguish snapshots from trends

top, free, and df show what is happening at one moment.

Performance issues do not always happen during that moment.

To observe behavior over time:

vmstat 1

This can show runnable processes, memory, swap, block activity, and CPU usage at regular intervals.

For I/O, when sysstat is installed:

iostat -xz 1

And sar becomes particularly valuable when historical collection is already enabled.

One detail with vmstat matters: the first line reports averages since boot, while subsequent lines reflect the requested interval.

Misreading that difference can lead to bad conclusions.

A CPU spike at 95% for one second may not matter. A saturation pattern recurring every five minutes probably does.

8. Check the kernel when the application explanation makes no sense

Not every application failure originates inside the application.

If you see:

• I/O errors;
• disappearing devices;
• network interfaces resetting;
• processes being killed unexpectedly;
• filesystem problems;
• memory pressure;

check the kernel messages.

A convenient modern approach is:

journalctl -k -b

You can also filter them:

journalctl -k -b -p warning

A service that simply appears as “killed” may have been terminated by the OOM killer. A slow disk may actually be generating device errors. A network interface may be renegotiating its link.

Application logs explain what the application knows.

Kernel logs can explain what happened to the application from outside it.

9. Read the exit code before declaring success

In Bash, exit status 0 means success and any non-zero value represents some kind of failure.

The most recent exit code is available with:

echo $?Code language: PHP (php)

But in scripts it is usually clearer to test the command directly:

if rsync -a /data/ /backup/data/; then
    echo "Backup completed"
else
    echo "Backup failed" >&2
    exit 1
fiCode language: PHP (php)

This is cleaner than running another command and checking $? afterwards.

It is also useful to remember a couple of common Bash exit codes: 127 usually means the command was not found, while 126 means it was found but could not be executed.

The lesson is simple: an automation job that finished is not necessarily an automation job that finished successfully.

10. set -e helps, but it is not error handling

A common recommendation is:

set -eCode language: JavaScript (javascript)

The intention is good: stop the script when a command fails.

The problem is that Bash has exceptions. -e behaves differently inside conditions, && and || lists, certain pipelines, and other contexts.

For small scripts, this is also common:

set -Eeuo pipefailCode language: JavaScript (javascript)

-u helps catch undefined variables, and pipefail prevents a pipeline from being considered successful just because its final command succeeded.

For example:

grep "important" missing-file | sortCode language: JavaScript (javascript)

without pipefail may return the exit status of sort, even though grep failed.

With:

set -o pipefailCode language: JavaScript (javascript)

that failure can propagate correctly.

Even then, these options do not replace explicit checks when different failures require different responses.

A backup script should be able to distinguish between a recoverable error, an unmounted destination, and an incomplete copy. Simply aborting on every non-zero exit code is often not enough.

11. Never assume cron or systemd has your shell environment

The classic “it works in my terminal” problem appears constantly in automation.

An interactive shell may have:

• a custom PATH;
• environment variables;
• loaded SSH keys;
• a specific working directory;
• aliases;
• shell functions;
• credentials already available.

Cron or a systemd unit may have none of those things.

Using absolute paths can help:

/usr/bin/python3 /opt/scripts/backup.py

but an even better practice is to define the environment the process actually needs.

For example:

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export PATHCode language: JavaScript (javascript)

And do not depend on a particular current working directory:

cd /opt/myapp || exit 1Code language: PHP (php)

or use absolute paths throughout.

Reproducibility matters more than convenience in an interactive shell.

12. Before a destructive command, run a non-destructive version

One of the most useful habits is separating selection from action.

Before:

find /backup -type f -mtime +30 -deleteCode language: JavaScript (javascript)

try:

find /backup -type f -mtime +30 -printCode language: PHP (php)

First verify exactly which files match.

Then add -delete.

With rsync, before synchronizing with deletions:

rsync -a --delete --dry-run /source/ /destination/Code language: JavaScript (javascript)

Only afterwards:

rsync -a --delete /source/ /destination/Code language: JavaScript (javascript)

This matters because many dangerous mistakes are not caused by the destructive command itself.

They are caused by selecting the wrong objects.

A preview lets you validate that selection first.

13. Preserve state before changing configuration

Before editing a critical configuration, keep a copy or, better still, version it.

For example:

cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.pre-change

For services that provide configuration validators, there is an even better habit: test before reloading.

For nginx:

nginx -t

If the configuration is valid:

systemctl reload nginx

When a service supports reload, this can avoid the interruption caused by a full restart.

The general pattern should be:

edit
→ validate
→ compare
→ reload
→ verify

rather than:

edit
→ restart
→ hope

A small change in procedure can reduce risk considerably.

14. Know the difference between reload and restart

Not every configuration change requires killing the process.

For services that implement configuration reload correctly:

systemctl reload nginx

may be preferable to:

systemctl restart nginx

restart stops and starts the unit again. reload asks the running service to re-read configuration without necessarily terminating the main process.

Not every service supports reload, and its exact semantics depend on the daemon, so this must be checked.

But using restart automatically for every configuration change introduces unnecessary interruption.

The useful habit is asking:

Do I really need to restart the process?

15. Inspect a process before killing it

Another dangerous reflex is:

kill -9 PID

SIGKILL cannot be caught by the process and gives it no opportunity to clean up resources.

It is usually better to start with normal termination:

kill PID

which sends SIGTERM.

Before even doing that:

ps -fp PID

and, where useful:

cat /proc/PID/status

help confirm exactly which process you are dealing with.

There are situations where SIGKILL is necessary. It should not be the first attempt just because it “always works.”

Mature Linux administration has a lot to do with this principle: use the smallest intervention necessary.

16. Use systemd-analyze when the problem is boot time

If a server takes too long to start, checking top afterwards will not explain what happened.

systemd includes dedicated tools:

systemd-analyze time

for overall startup timing.

Then:

systemd-analyze blame

shows units sorted by how long they took to initialize.

And:

systemd-analyze critical-chain

shows the critical dependency chain.

One warning matters here: blame should not be interpreted as an automatic list of guilty services. A service may appear slow because it was waiting for another dependency. systemd’s own documentation points out this limitation.

That is why critical-chain often provides better context.

Again, the useful habit is choosing the tool that answers the specific question instead of measuring whatever happens to be available.

17. Do not casually parse output meant for humans

Many Linux commands produce output that is convenient for people to read but not guaranteed to remain identical forever.

Tools such as lsblk and findmnt, for example, recommend specifying columns explicitly when used in scripts.

Instead of depending on the visual layout of:

lsblk

an automation can request exactly what it needs:

lsblk -o NAME,TYPE,FSTYPE,SIZE,MOUNTPOINTS

Likewise:

findmnt -o SOURCE,TARGET,FSTYPE,OPTIONS

This prevents scripts from breaking when an update adds a field, changes ordering, or adjusts formatting.

Humans appreciate pretty output.

Scripts need explicit contracts.

18. Capture a small diagnostic snapshot before intervention

A useful habit to automate is collecting basic system state when an incident starts.

For example:

date
hostname
uptime
free -h
df -h
lsblk
systemctl --failed
ss -lntp
journalctl -p err -b --no-pager

This does not need to become a full monitoring suite.

A script can save everything into a timestamped file:

/opt/tools/capture-state.sh > "/var/tmp/state-$(date +%Y%m%d-%H%M%S).log" 2>&1Code language: JavaScript (javascript)

Then, even if a service or the whole machine has to be restarted afterwards, the pre-change state is preserved.

This is especially useful during incidents where multiple people are making changes in parallel.

19. Automate checks, not only repairs

Automation often gets framed as “automatically fixing things.”

But some of the most valuable automation simply detects when reality differs from expectations.

For example:

systemctl is-active --quiet nginx || echo "nginx is not active"Code language: PHP (php)

Other useful checks include:

• free disk space;
• certificates approaching expiration;
• recent backups;
• expected mounts;
• failed units;
• required listeners;
• time synchronization;
• RAID state.

Automatic remediation can sometimes turn one unexpected condition into an even harder-to-debug one.

Automatic detection, on the other hand, can alert you before the issue reaches users.

20. Keep an incident journal and a command notebook

The least technical habit can become one of the most valuable.

After an incident, record:

• initial symptom;
• timestamp;
• first evidence;
• recent changes;
• commands used;
• root cause;
• fix;
• how the same problem could be detected earlier;
• what automation could prevent it from recurring.

There is no need for a complex platform. A private Git repository full of Markdown files can be enough.

After a year, it becomes something more useful than a collection of notes: a manual of the real failures your infrastructure has experienced.

A good administrator does not rely only on remembering how something was fixed six months ago.

They make the system remember for them.

Commands matter, but the order matters more

Two administrators can know exactly the same commands and still get very different results.

One works like this:

restart
→ test
→ change something
→ test again

The other works like this:

observe
→ narrow down
→ preserve evidence
→ form a hypothesis
→ validate
→ make the smallest change
→ verify

The difference is not knowing more Linux.

It is following a process that reduces the chances of destroying information, introducing new problems, or confusing correlation with causation.

It also explains why simple tools such as journalctl, ss, findmnt, vmstat, and systemctl --failed remain useful even in infrastructures already equipped with Prometheus, Grafana, OpenTelemetry, and advanced observability platforms.

When a machine is in front of you and something is broken, you still need to know how to interrogate it.

The best habits eventually look boring: check before deleting, validate before reloading, preserve before changing, and read before restarting.

That is exactly why they work so well in production.

Frequently Asked Questions

Which commands should I run first during a Linux incident?

It depends on the symptom, but systemctl status, journalctl, systemctl --failed, ss, df, free, findmnt, and vmstat can quickly reveal service, log, network, storage, memory, and load information. The most important thing is collecting evidence before changing the system.

Should I use set -e in Bash scripts?

It can be useful, but it should not be treated as complete error handling. Bash applies exceptions to -e, and pipefail, explicit checks, and specific error handling are still necessary in important scripts.

Why use findmnt if df already exists?

They answer different questions. df reports filesystem usage, while findmnt helps identify and inspect mounts, sources, targets, filesystem types, and mount options.

Is it better to restart or reload a service?

If the daemon supports a safe configuration reload, reload may avoid a full interruption. It depends on the service. Before either operation, validate the configuration and inspect the current state.