Linux logs can reveal signs of an intrusion long before an administrator notices its consequences. With journalctl, the command-line tool for querying systemd-journald, it is possible to look for unexpected network connections, software installations, account creation, and attempts to tamper with the logs themselves. None of these signs proves that an attack has occurred on its own, but together they can expose behavior that deserves investigation.

The key points about Linux logs in 30 seconds

Unexpected network connections can reveal communication with unknown systems or activity from processes that should not connect externally.
Package installations outside normal maintenance windows may indicate that someone has added tools to the server.
A new account, particularly one with elevated privileges, can provide an attacker with a way to maintain access.
Missing or unexpectedly altered logs can also be a warning sign because an intruder with sufficient privileges may try to erase evidence.
journalctl does not record everything, so serious investigations need to combine it with other sources of evidence.

Reviewing logs is not about finding a single line that says a server has been compromised. The value comes from identifying changes or behavior that do not fit what is normally expected from that machine.

A server can establish legitimate network connections, install packages during routine updates, or create users as part of normal administration. The concern arises when those events have no known explanation or occur at the same time as other suspicious activity.

That is why journalctl is useful as a starting point for investigating what happened and when.

Network connections that should not be there

One of the first things worth checking is unexpected network activity. An attacker who has gained access to a server may attempt to communicate with another system, download files, or establish a connection to a remote server.

A simple search in the journal can locate messages related to connections:

sudo journalctl --grep="connection|connect|network"Code language: JavaScript (javascript)

Kernel messages can also be searched:

sudo journalctl -k --grep="connection|network"Code language: JavaScript (javascript)

And events associated with blocked or rejected connections:

sudo journalctl --grep="DROP|REJECT|blocked|denied"Code language: JavaScript (javascript)

Several patterns deserve attention during a review: an unknown service listening on a network port, repeated connections to an unfamiliar destination, or outbound traffic generated by a process that normally has no reason to communicate with the internet.

Repeated blocked connection attempts can also provide useful context. They do not necessarily indicate an intrusion, but they can help reconstruct what was happening on the system.

The key is to compare the events with the server’s normal behavior. An unexpected connection on a machine that never needs internet access means something very different from the same connection on a server that routinely communicates with external services.

Software installed without a clear explanation

Software installation activity is another useful source of clues. Once someone gains administrative privileges, they may install tools for network reconnaissance, lateral movement, data exfiltration, or cryptocurrency mining.

Tools such as nmap, netcat, or masscan can appear in these situations, although the presence of any one of them does not prove malicious activity.

Package-manager activity can be searched with:

sudo journalctl --grep="apt|dpkg|dnf|yum|rpm"Code language: JavaScript (javascript)

It is also possible to search for installations performed through sudo:

sudo journalctl _COMM=sudo --grep="install"Code language: JavaScript (javascript)

Several patterns deserve attention: software installed without a known reason, security or networking tools suddenly appearing, packages installed outside the normal maintenance window, or installations that occur shortly after a suspicious login.

It is also worth checking who performed the operation. An installation carried out by someone who does not normally administer the server can provide important context.

The goal is not to treat every new tool as malicious. Administrators install software regularly. What matters is connecting the change with the expected role of the machine and with other events recorded around the same time.

A new account can become another way in

The appearance of an unexpected account is another event that deserves investigation.

An attacker who has gained access may create a new user to maintain a way back into the system. That account could be given a password, an SSH key, or sudo privileges, allowing access to continue even after the original entry point has been fixed.

journalctl can be used to search for user-management events:

sudo journalctl --grep="useradd|adduser|new user"Code language: JavaScript (javascript)

The search can also include modifications and deletions:

sudo journalctl --grep="useradd|usermod|userdel"Code language: JavaScript (javascript)

And operations performed through sudo:

sudo journalctl _COMM=sudo --grep="useradd|adduser|usermod|userdel"Code language: JavaScript (javascript)

A new account is not automatically suspicious. It could be part of a deployment, a new employee’s setup, or an automated task. The useful questions are who created it, when it was created, and what privileges it received.

An account created shortly after an anomalous login deserves more attention than one created during a documented maintenance operation.

When the logs themselves start disappearing

Logs can also become a target. An attacker who knows that their actions are being recorded may attempt to delete, rotate, or modify logs to make a later investigation more difficult.

A query such as this can be used to review activity from systemd-journald:

sudo journalctl -u systemd-journald --since "30 days ago"Code language: JavaScript (javascript)

It is also possible to search for messages associated with rotation, deletion, corruption, or clock changes:

sudo journalctl --since "30 days ago" \
  -g 'Vacuuming|rotate|corrupt|Time has been changed|Suppressed'Code language: JavaScript (javascript)

The absence of logs can itself be useful information. If there is a period when events should normally have been recorded but the journal suddenly contains a gap, it is worth finding out why.

However, journalctl has important limitations. Some systems use volatile storage, meaning certain records may disappear after a reboot. Excessive message rates can also cause journald to suppress events.

Why `journalctl` does not tell the whole story

One of the biggest risks when using the journal as the only source of information is assuming that it contains everything that happened on the server.

On some systems, journald can use volatile storage. In that configuration, some records can disappear after a reboot. To keep journals on disk, the configuration can include:

[Journal]
Storage=persistent

There is also message rate limiting. When a service generates too many events in a short period, journald can suppress some of them. A message such as:

Suppressed 100 messages from sshd.serviceCode language: JavaScript (javascript)

should be treated as relevant during an investigation because it means the journal does not contain every message that was generated.

The behavior can be controlled using RateLimitIntervalSec and RateLimitBurst in journald.conf.

Another limitation is that an attacker who has obtained root privileges may be able to rotate, vacuum, or delete local journal files. For stronger protection, Forward Secure Sealing can help detect changes to historical journal entries, while logs can also be shipped to a separate system so the evidence does not all remain on the potentially compromised server.

Key setup can be initiated with:

sudo journalctl --setup-keys

and sealing can be enabled with:

[Journal]
Seal=yes

Tools such as systemd-journal-remote and rsyslog can also be used to send logs to separate systems and build a centralized logging pipeline.

There is another important limitation: journalctl only shows events that applications, services, the kernel, or other components actually send to the journal. Without additional auditing rules, activities such as certain file accesses, execve calls, or setuid changes may not be visible.

That is why a deeper investigation should combine the journal with tools such as auditd, process inspection, network monitoring, and centralized logging.

Regularly reviewing logs does not prevent an intrusion by itself, but it provides more evidence when something changes on a server. The value of journalctl is its ability to turn scattered system events into a starting point for investigation, provided administrators remember that the journal is a source of evidence rather than a perfect record of everything that happened.

Frequently Asked Questions

What can `journalctl` detect?

It can help identify unexpected network connections, software installations, user creation or modification, and events involving the logging service itself.

Does an unknown network connection mean the server has been attacked?

No. An unexpected connection is a signal that should be investigated and compared with the server’s normal behavior and other available logs.

Can an attacker delete Linux logs?

A user with root privileges may be able to rotate, vacuum, or delete local logs. Keeping copies on a separate logging system and using mechanisms that can detect changes can make it harder to erase evidence without detection.

Is `journalctl` enough to investigate an incident?

No. The journal does not necessarily contain every system event. More complete investigations should combine it with tools such as auditd, process inspection, network monitoring, and centralized logging.