Auditing a Linux Server Before You Trust It: A Sysadmin Checklist

A Linux server can be responding normally, serving traffic, and running for months without any obvious errors while still carrying forgotten accounts, old SSH keys, unknown repositories, unnecessary services, or configurations nobody can explain. When a system administrator inherits a Debian, Ubuntu, RHEL, Rocky Linux, AlmaLinux, Fedora, SUSE, or other Linux server, the first task should not be updating or hardening it, but understanding exactly what is running on it.

The key points of a Linux server audit in 20 seconds

• Before changing anything, identify the distribution, kernel, packages, users, services, network configuration, and persistence mechanisms.
• Debian and Ubuntu deserve particular attention around APT, dpkg, SSH, and AppArmor.
• On RHEL-based systems, RPM, DNF, SELinux, and firewalld also become important.
• An isolated anomaly does not prove compromise: it needs to be compared with the server’s actual role.

Immediately running apt upgrade or dnf upgrade may seem like the safest approach, but it also changes hundreds of files, timestamps, packages, and logs. If the goal is to understand what happened on an inherited server, it is better to capture a sufficiently detailed snapshot first.

There is also no single command capable of answering the question, “Is this server clean?” An audit involves correlating many different signals and checking whether they all have a reasonable explanation.

Build a Reliable Snapshot of the Server Before Changing It

1. Identify exactly which Linux system you have inherited

The first check is basic, but it determines much of what comes next.

hostnamectl
cat /etc/os-release
uname -a
uname -r
uname -m
uptime
last reboot | head -10
journalctl --list-boots
timedatectl
systemd-detect-virtCode language: PHP (php)

/etc/os-release identifies the distribution and version. uname shows the kernel that is actually running, which does not always match the newest kernel installed on disk.

systemd-detect-virt also deserves attention. Auditing a physical server is different from auditing a virtual machine, an LXC container, or a Docker environment. Inside a container, many kernel-level characteristics belong to the host rather than the environment being inspected.

It is also worth checking disks, mounts, available space, and inodes:

lsblk -f
findmnt
df -hT
df -ih

A filesystem that is only 70% full in terms of storage capacity can still be approaching inode exhaustion because it contains millions of small files.

System time should also be consistent. Logs with incorrect timestamps make any subsequent investigation much harder.

timedatectl

2. Review packages and repositories on Debian and Ubuntu

On Debian and Ubuntu servers, the audit should cover both installed packages and configured repositories.

dpkg-query -W -f='${Package}\t${Version}\n'
apt-cache policy
apt list --upgradable 2>/dev/nullCode language: JavaScript (javascript)

To locate repositories:

grep -RhsE '^(deb |URIs:|Suites:|Components:|Signed-By:)' \
/etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/nullCode language: PHP (php)

Recent Debian and Ubuntu versions can also use .sources files in deb822 format, so checking only /etc/apt/sources.list may miss part of the configuration.

The important question is not simply whether updates are pending. Administrators need to know where packages are coming from.

An abandoned third-party repository, a temporarily added testing branch, an unknown mirror, or a repository belonging to a vendor that is no longer used can all create security and maintenance problems.

For automatic updates:

systemctl status apt-daily.timer apt-daily-upgrade.timer
systemctl list-timers | grep aptCode language: PHP (php)

When unattended-upgrades is present:

dpkg -l unattended-upgrades
grep -R "Unattended-Upgrade" /etc/apt/apt.conf.d/ 2>/dev/null
ls -la /var/log/unattended-upgrades/ 2>/dev/nullCode language: JavaScript (javascript)

On Ubuntu systems using Ubuntu Pro:

pro status

The tools change on RPM-based distributions.

For RHEL, Rocky Linux, AlmaLinux, and Fedora:

dnf repolist
dnf check-update
rpm -qa --last | head -30

For SUSE and openSUSE:

zypper repos
zypper list-updatesCode language: PHP (php)

For Arch Linux:

pacman -Q
pacman -Qu

The principle remains the same regardless of the package manager: determine what is installed, who provides those packages, and how long the system has gone without maintenance.

3. Users: do not stop at looking inside /home

An account can exist without a directory under /home, and a service account can become interactive if someone changes its shell.

A first overview:

getent passwd

To display accounts with potentially interactive shells:

getent passwd | awk -F: '$7 !~ /(nologin|false)$/ {print $1, $3, $6, $7}'Code language: JavaScript (javascript)

One particularly important check is finding accounts with UID 0:

getent passwd | awk -F: '$3 == 0 {print $1, $6, $7}'Code language: JavaScript (javascript)

On most servers, only root should appear.

It is also worth checking /etc/shadow:

sudo awk -F: '$2 == "" {print $1}' /etc/shadowCode language: JavaScript (javascript)

An empty password field requires immediate investigation, although its existence does not automatically mean that SSH permits passwordless access.

Administrative groups also differ between distributions.

Debian and Ubuntu:

getent group sudo

RHEL, Fedora, Rocky Linux, and AlmaLinux:

getent group wheel

The actual privileges, however, are defined in sudoers:

sudo visudo -c
sudo grep -RniE 'NOPASSWD|ALL=\(ALL' \
/etc/sudoers /etc/sudoers.d 2>/dev/nullCode language: JavaScript (javascript)

Finding NOPASSWD is not automatically a security problem. It may be required for automation, configuration management, or deployments. The important point is that every rule should have a clear reason to exist.

4. SSH: inspect the effective configuration

An SSH audit should not consist only of opening:

/etc/ssh/sshd_config

OpenSSH can load additional files and apply different policies through Match blocks.

It is much more useful to ask the daemon for its effective configuration:

sudo sshd -T

To focus on some of the most relevant parameters:

sudo sshd -T | grep -Ei \
'permitrootlogin|passwordauthentication|pubkeyauthentication|maxauthtries|authorizedkeysfile|authorizedkeyscommand|allowusers|allowgroups'Code language: JavaScript (javascript)

Then look for additional configuration:

grep -RniE '^(Include|Match|PermitRootLogin|PasswordAuthentication|AllowUsers|AllowGroups|AuthorizedKeys)' \
/etc/ssh/sshd_config /etc/ssh/sshd_config.d/ 2>/dev/nullCode language: JavaScript (javascript)

Match rules can cause policies to change depending on the user or source address. OpenSSH can evaluate a specific connection:

sudo sshd -T -C user=admin,host=server,addr=192.0.2.10

Those values should be replaced with ones relevant to the actual environment.

SSH keys should not be trusted simply because they are present in an authorized_keys file.

sudo find /root /home -xdev -type f -name authorized_keys -ls 2>/dev/nullCode language: JavaScript (javascript)

Every key should be associated with a specific person, automation process, or system.

It is also worth checking whether SSH obtains keys through an external command:

sudo sshd -T | grep authorizedkeyscommand

A server can therefore have authorized SSH access even when not every key is physically stored under /home.

From Networking to Persistence: Find Out What the Server Is Really Doing

5. See what is listening and which process opened it

One of the checks with the best ratio of effort to useful information is:

sudo ss -lntup

It shows listening TCP and UDP sockets and, with sufficient privileges, their associated processes.

Active connections can then be inspected:

sudo ss -tpn state established

Every service should be related to the server’s intended role.

A PostgreSQL instance listening only on localhost:

127.0.0.1:5432Code language: CSS (css)

has a very different exposure profile from one listening on:

0.0.0.0:5432Code language: CSS (css)

or:

[::]:5432Code language: CSS (css)

However, listening on all interfaces does not necessarily mean that a service is reachable from the Internet. The firewall still needs to be checked.

On systems using nftables:

sudo nft list rulesetCode language: PHP (php)

Ubuntu may use UFW:

sudo ufw status verbose

RHEL, Fedora, Rocky Linux, and AlmaLinux commonly use firewalld:

sudo firewall-cmd --state
sudo firewall-cmd --get-active-zones
sudo firewall-cmd --list-allCode language: JavaScript (javascript)

There are also systems where checking iptables remains useful:

sudo iptables -L -n -v
sudo ip6tables -L -n -v

On many modern Linux systems, iptables operates through an nftables backend, so administrators should understand which implementation their distribution is actually using.

If the server runs in AWS, Azure, Google Cloud, OpenStack, or a similar environment, the network audit does not end inside Linux. Security groups, ACLs, external firewalls, load balancers, and provider-side policies must also be considered.

6. Verify the actual binary behind each PID

Process names can be misleading.

To investigate a specific PID:

PID=1842

sudo readlink -f /proc/$PID/exe
sudo tr '\0' ' ' < /proc/$PID/cmdline
echo
sudo cat /proc/$PID/status
sudo readlink -f /proc/$PID/cwdCode language: PHP (php)

Open file descriptors can also be inspected:

sudo ls -la /proc/$PID/fd 2>/dev/null | head -50Code language: JavaScript (javascript)

A process called nginx should not automatically be considered legitimate if its actual executable is:

/tmp/.cache/nginx

It is also useful to find processes that continue running executables that have already been deleted:

sudo find /proc/[0-9]*/exe -lname '* (deleted)' -ls 2>/dev/nullCode language: JavaScript (javascript)

A (deleted) result does not mean malware.

After a package upgrade, an old process may still be running an executable that has been replaced on disk. Tools such as needrestart, commonly found on Debian and Ubuntu systems, can help identify services still using outdated components after upgrades.

The anomaly begins when nobody can explain why the deleted executable is still running.

7. Audit persistence through systemd

Cron is no longer the only place to look for automatic execution.

On any systemd-based distribution:

systemctl list-unit-files --state=enabled
systemctl list-timers --all
systemctl --failedCode language: PHP (php)

A particularly useful and sometimes overlooked command is:

systemd-delta

It displays local configuration that overrides or extends units provided by installed packages.

For example, it may reveal that /usr/lib/systemd/system/nginx.service has an override under:

/etc/systemd/system/nginx.service.d/

To review local systemd configuration:

sudo find /etc/systemd/system -maxdepth 3 \( -type f -o -type l \) -ls

Individual services can also be inspected:

systemctl cat ssh.service
systemctl cat nginx.serviceCode language: CSS (css)

Depending on the distribution, the SSH daemon may be named ssh.service or sshd.service.

8. Cron, at, and other persistence points

Cron remains essential:

sudo crontab -l -u root

To inspect every account:

for user in $(cut -d: -f1 /etc/passwd); do
    sudo crontab -l -u "$user" 2>/dev/null
doneCode language: JavaScript (javascript)

Then check:

sudo cat /etc/crontab
sudo ls -la /etc/cron.d/
sudo ls -la /etc/cron.hourly/
sudo ls -la /etc/cron.daily/
sudo ls -la /etc/cron.weekly/
sudo ls -la /etc/cron.monthly/

The at queue also matters:

atq

During a deeper investigation, other locations may be worth checking:

sudo ls -la /etc/profile.d/
sudo ls -la /etc/modules-load.d/
sudo ls -la /etc/modprobe.d/
sudo ls -la /etc/udev/rules.d/
sudo cat /etc/rc.local 2>/dev/null
sudo cat /etc/ld.so.preload 2>/dev/nullCode language: JavaScript (javascript)

/etc/ld.so.preload deserves particular attention if it exists and contains an unexpected library, as it can cause libraries to be preloaded into dynamically linked programs.

On Docker or Podman hosts, persistence can also exist outside cron and systemd:

docker ps --no-trunc
docker ps -a --no-trunc

or:

podman ps --all

Each container should be related to its image, mounts, network configuration, and restart policy.

Integrity, Kernel, and Evidence: When to Stop Trusting the Machine

9. Verify packages on Debian and Ubuntu

On Debian and Ubuntu:

sudo dpkg --verify

Another traditional tool is debsums:

sudo debsums -s

If it is not installed:

sudo apt install debsums

However, this last command creates a problem during an investigation: installing software changes the system being audited.

For that reason, debsums is much more useful when it was already installed.

There is another conceptual limitation. Comparing checksums can reveal modified files, but it does not prove that a server is trustworthy. If an attacker gained sufficient control, the information used as a reference may itself have been manipulated.

To find which package owns a file:

dpkg-query -S /usr/bin/ssh

On RHEL, Rocky Linux, AlmaLinux, Fedora, and other RPM-based systems:

sudo rpm -Va

For a specific file:

rpm -qf /usr/bin/ssh

On Arch Linux:

pacman -Qkk

Output from rpm -Va, dpkg --verify, or debsums requires interpretation. Configuration files can be legitimately modified.

10. Review SUID, SGID, and Linux capabilities

Special permissions still matter.

SUID:

sudo find / -xdev -type f -perm -4000 -exec ls -l {} + 2>/dev/nullCode language: JavaScript (javascript)

SGID:

sudo find / -xdev -type f -perm -2000 -exec ls -l {} + 2>/dev/nullCode language: JavaScript (javascript)

A long list does not necessarily mean something is wrong. Legitimate binaries may require these permissions.

The question is which entries are known and which are not.

Linux capabilities can also grant specific privileges without using SUID:

sudo getcap -r / 2>/dev/nullCode language: JavaScript (javascript)

Capabilities such as:

cap_sys_admin
cap_sys_ptrace
cap_dac_override
cap_net_admin
cap_sys_module
cap_sys_rawio

deserve a clear explanation when assigned to unusual executables.

World-writable files can also reveal dangerous configurations:

sudo find / -xdev -type f -perm -0002 -ls 2>/dev/nullCode language: JavaScript (javascript)

And world-writable directories without the sticky bit:

sudo find / -xdev -type d -perm -0002 ! -perm -1000 -ls 2>/dev/nullCode language: JavaScript (javascript)

11. AppArmor on Debian and Ubuntu

A Debian or Ubuntu audit should include AppArmor.

Check its status with:

sudo aa-status

And inspect the service:

systemctl status apparmor

Profiles are normally found under:

ls -la /etc/apparmor.d/

There is an important difference between a profile running in enforce mode and one running in complain mode.

enforce actually applies the restrictions.

complain records actions that would have been blocked but allows them to continue.

Events can be searched in the journal:

sudo journalctl -k | grep -i apparmor

Ubuntu uses AppArmor as one of its main Mandatory Access Control layers. Debian also integrates AppArmor and enables it by default in standard installations.

Having AppArmor enabled does not mean every service is confined. aa-status helps determine how many profiles are loaded and which processes are actually subject to them.

12. SELinux on RHEL, Rocky, AlmaLinux, and Fedora

On Red Hat Enterprise Linux and related distributions, SELinux is the usual mechanism.

getenforce
sestatus

Its main states are:

Enforcing
Permissive
Disabled

Enforcing applies policy.

Permissive records violations without blocking them.

To look for recent events:

sudo ausearch -m AVC,USER_AVC -ts today

If an application “only works when SELinux is disabled,” the problem has not actually been solved. Administrators should determine which operation the policy is blocking and whether that operation should be permitted.

13. Review kernel security parameters

A compact check is:

sysctl \
kernel.randomize_va_space \
kernel.kptr_restrict \
kernel.dmesg_restrict \
kernel.yama.ptrace_scope \
fs.protected_hardlinks \
fs.protected_symlinks \
net.ipv4.ip_forward \
net.ipv4.conf.all.accept_redirects \
net.ipv4.conf.default.accept_redirects \
net.ipv4.conf.all.send_redirects \
net.ipv6.conf.all.accept_redirectsCode language: CSS (css)

A typical server reference might look like this:

Parameter	Common server value
kernel.randomize_va_space	2
fs.protected_hardlinks	1
fs.protected_symlinks	1
kernel.dmesg_restrict	1
net.ipv4.ip_forward	0 if the host does not route traffic
accept_redirects	0 on most servers
send_redirects	0 on most servers

These values should not be copied blindly.

A VPN server, Linux router, Kubernetes node, some Docker hosts, or virtualization platforms may require different settings.

The audit should first establish the role of the node.

14. Kernel modules

Loaded modules can be inspected with:

lsmod
cat /proc/modules

Also check:

cat /proc/cmdline

To determine whether the system allows additional modules to be loaded:

sysctl kernel.modules_disabledCode language: CSS (css)

A value of:

kernel.modules_disabled = 1

prevents additional modules from being loaded until the next reboot.

This can be a useful hardening measure for very static servers, but it is not appropriate for every environment.

It is also useful to see which Linux Security Modules are active:

cat /sys/kernel/security/lsm 2>/dev/nullCode language: JavaScript (javascript)

The best way to identify an unusual kernel module is to compare the system against a known baseline for the same type of server.

Without a baseline, an unfamiliar module could be either a perfectly legitimate driver or something requiring further investigation.

15. Seccomp and NoNewPrivileges

Seccomp applies to individual processes rather than the entire server.

For a specific PID:

PID=1842

grep -E 'NoNewPrivs|Seccomp|Seccomp_filters' /proc/$PID/statusCode language: PHP (php)

The Seccomp field usually reports:

0
1
2

where 0 means disabled, 1 is strict mode, and 2 indicates filter mode.

However, Seccomp: 2 does not mean that the filter is well designed. It only confirms that a filter exists.

On systemd systems, another useful tool is:

systemd-analyze security

For a particular unit:

systemd-analyze security nginx.serviceCode language: CSS (css)

It can evaluate protections such as ProtectSystem, PrivateTmp, namespace restrictions, capabilities, and NoNewPrivileges.

It is not a vulnerability scanner. Its purpose is to show how much isolation the systemd unit provides.

16. Authentication and system logs

On systemd systems:

sudo journalctl -p err..alert --since "7 days ago"Code language: JavaScript (javascript)

For SSH:

sudo journalctl -u ssh -u sshd --since "7 days ago"Code language: JavaScript (javascript)

On Debian and Ubuntu systems with /var/log/auth.log:

sudo grep -iE 'failed password|accepted password|accepted publickey' \
/var/log/auth.log 2>/dev/null | tail -100Code language: JavaScript (javascript)

Recent logins:

last -Fai | head -30

Failed login attempts:

sudo lastb -Fai | head -30

Reboots:

last reboot | head -20
journalctl --list-bootsCode language: PHP (php)

If auditd is available:

systemctl status auditd
sudo auditctl -l
sudo aureport --summary

auditd can provide extremely useful information, but only for activity it was configured to record.

Installing it today cannot reconstruct what happened last week.

There is another limitation: local logs live on the system being investigated. If someone obtained root privileges, they may have deleted or modified them.

That is why centralized logs sent in real time to another host or a SIEM are much more valuable during an incident.

17. Check resources without treating every spike as an IOC

CPU:

ps -eo pid,ppid,user,%cpu,%mem,lstart,cmd --sort=-%cpu | head -30

Memory:

ps -eo pid,ppid,user,%cpu,%mem,lstart,cmd --sort=-%mem | head -30

Disk:

df -hT
df -ih

Directories:

sudo du -xhd1 /var 2>/dev/null | sort -h
sudo du -xhd1 /tmp 2>/dev/null | sort -h
sudo du -xhd1 /opt 2>/dev/null | sort -hCode language: JavaScript (javascript)

I/O, if sysstat is installed:

iostat -xz 1 3

A process consuming a lot of CPU is not automatically a cryptominer. A large /tmp directory does not prove data exfiltration either.

These metrics help identify anomalies that then need to be explained.

18. Not every Linux server uses systemd

Although Debian, Ubuntu, RHEL, Rocky Linux, AlmaLinux, Fedora, and many other distributions use systemd, administrators should not assume it is always present.

Alpine Linux commonly uses OpenRC.

In that case:

rc-status
rc-update show

are more useful than systemctl.

Arch Linux normally uses systemd, but leaves a significant amount of system configuration to the administrator.

Similarly, the presence of UFW, firewalld, AppArmor, SELinux, or any particular security policy should never be assumed without checking.

19. Quick reference for major Linux distributions

Area	Debian / Ubuntu	RHEL / Rocky / Alma / Fedora	SUSE / openSUSE	Arch
Packages	APT + dpkg	DNF + RPM	Zypper + RPM	pacman
Verification	dpkg --verify, debsums	rpm -Va	rpm -Va	pacman -Qkk
Common MAC	AppArmor	SELinux	AppArmor	Depends on configuration
Common firewall	nftables / UFW	firewalld / nftables	firewalld / nftables	Depends on configuration
Services	systemd	systemd	systemd	systemd
Logs	journal + syslog depending on configuration	journal	journal	journal
SSH	OpenSSH	OpenSSH	OpenSSH	OpenSSH

The table is a guideline rather than an absolute rule. Linux allows administrators to replace almost all these components, and a real installation may not follow the distribution defaults.

20. Know when to stop hardening and start incident response

This may be the most important point in the entire audit.

Finding a permissive PermitRootLogin setting is a configuration problem.

Finding an unknown UID 0 account is a different situation.

Discovering a systemd timer that periodically executes a hidden binary from /tmp, an unknown SSH key, an unexpected library in /etc/ld.so.preload, or an unexplained kernel module may indicate something much more serious.

At that point, immediately “cleaning up” the machine can be counterproductive.

Rebooting destroys volatile process and memory state.

Updating replaces files.

Deleting an account destroys information.

Removing a binary can eliminate evidence.

If there is a reasonable suspicion of compromise, the priority should shift to isolation, preservation, and analysis, following the organization’s incident-response procedures.

The trust model also changes if an attacker may have obtained root privileges.

Simply removing whatever looks malicious cannot reliably prove that the rest of the machine is clean.

In many environments, the safer approach will be to rebuild the server from a known-good image, install packages from trusted repositories, restore only verified data, and rotate SSH keys, passwords, API tokens, certificates, and other secrets that may have been exposed.

Build a Baseline After the Audit

Once the server has been assessed and the decision has been made to keep it, its known-good state should be documented.

Component	Baseline worth keeping
Distribution	Version and support lifecycle
Kernel	Expected version
Packages	Inventory and authorized repositories
Users	Valid accounts
SSH	Authorized keys and access policy
sudo	Approved privileges
Services	Enabled units
Network	Expected listening ports
Firewall	Policy and exceptions
Cron/systemd	Known jobs and timers
AppArmor/SELinux	Expected status and policies
Integrity	Checksums from a trusted source
Logs	Location and retention
Backups	Policy and last tested restore

This completely changes the next audit.

The first time, an administrator may have to ask what 42 enabled services are doing.

On the next audit, the question can be much more precise: yesterday there were 41, so what is the new service, who installed it, and why?

That comparison is what turns a collection of Linux commands into an operational security process.

A Linux server should not be considered trustworthy simply because it has been running for 300 days, because top looks normal, or because nothing unusual appears in ss.

Trust starts to become reasonable when every account, package, service, listening port, repository, persistence mechanism, and security exception has an explanation, and there is a known baseline against which future changes can be compared.

Frequently Asked Questions

What should a sysadmin check first when taking over a Linux server?

The distribution, kernel, virtualization environment, disks, system time, repositories, and installed packages should be recorded before making changes. Users, SSH, sudo, services, networking, persistence, and security controls can then be audited.

Which commands are particularly useful when auditing Debian and Ubuntu?

Useful commands include dpkg --verify, apt-cache policy, sshd -T, aa-status, systemctl list-timers, systemd-delta, ss -lntup, and journalctl. debsums can provide an additional integrity check when it is already installed.

What changes when auditing Rocky Linux, AlmaLinux, or RHEL?

The methodology is largely the same, but APT and dpkg are replaced by DNF and RPM, SELinux is commonly used instead of AppArmor, and firewalld is frequently used to manage firewall policy.

Can an audit prove that a Linux server is clean?

Not absolutely. An audit can identify anomalies and establish a baseline, but a privileged compromise may have affected system tools, the kernel, or local logs. When there is sufficient evidence of intrusion, rebuilding from trusted sources may be safer than attempting to clean the existing installation.

Sources:

• Debian Administrator’s Handbook, Debian system administration and security.
• Debian Manpages, documentation for dpkg, debsums, systemd, and AppArmor.
• Ubuntu Server Documentation, Ubuntu Server security, updates, and administration.
• Ubuntu Security Documentation, AppArmor and system-hardening mechanisms.
• Red Hat Enterprise Linux Documentation, SELinux, firewalld, RPM, and system auditing.
• Arch Linux Wiki, package administration, systemd, and security.
• Alpine Linux Documentation, service management with OpenRC.
• OpenSecurity