sar, the reporting tool in the sysstat package, quietly records CPU (central processing unit), memory, disk and network activity on a Linux server every few minutes, so administrators can go back and see what the machine was doing when an incident happened, without an agent or a paid service. A Medium post by bektiaw recently described it as a flight recorder that many servers already carry and as a lightweight alternative to platforms such as Datadog. This guide takes that idea and checks it against a real install of sysstat 12.6.1 on Ubuntu 24.04, with notes on where Red Hat Enterprise Linux (RHEL) and its rebuilds behave differently.
The key points of sar and sysstat in 30 seconds
- sysstat’s collector, sadc, writes binary daily files that sar reads back as tables and sadf exports to other formats.
- On Ubuntu 24.04 and on RHEL 8 and 9, collection is driven by systemd timers rather than cron, every 10 minutes by default.
- Disk history needs the “-S DISK” collector option, and TCP history needs “-S SNMP”.
- A one-minute interval plus LC_ALL=C makes short spikes visible and output safe to parse with awk.
It is worth being clear about scope. sar does not page anyone and does not draw dashboards. What it does well is answer, after the fact and at almost no cost, the question every on-call engineer asks the morning after: what happened at 3:12 a.m.?
How sysstat collects data and how to configure it properly
The moving parts
Five pieces do the work. sadc reads kernel counters and appends binary records. sa1 is a small shell wrapper around sadc, and it is what the systemd timer actually runs. sa2 writes a daily text report and prunes or compresses old files. sar turns the binary records into readable reports, and sadf exports the same data for databases, scripts or browsers.
Each day of the month gets its own file: saDD holds the binary data and sarDD the text summary from sa2. One detail changes how the numbers should be read: every sar line is an average over the time since the previous record. With the default 10-minute interval, a 40-second CPU spike that took a site down will look like a modest bump.
| Item | Debian and Ubuntu | RHEL and rebuilds |
|---|---|---|
| Data files | /var/log/sysstat/saDD | /var/log/sa/saDD |
| Config file | /etc/sysstat/sysstat | /etc/sysconfig/sysstat |
| sa1 and sa2 scripts | /usr/lib/sysstat/ | /usr/lib64/sa/ |
| Default retention | HISTORY=7 | HISTORY=28 |
Installing and enabling collection
# Debian and Ubuntu
sudo apt install sysstat
# RHEL, AlmaLinux, Rocky Linux
sudo dnf install sysstat
# Enable the service and both timers, then confirm
sudo systemctl enable --now sysstat
systemctl list-timers 'sysstat*'Code language: PHP (php)Debian and Ubuntu also ship /etc/default/sysstat with ENABLED="false", and most tutorials say to flip it to "true". On a systemd host that variable is only read by the legacy cron path (debian-sa1 and the /etc/cron.daily job), and both scripts exit immediately when they detect systemd. What really decides whether data is collected is the state of sysstat-collect.timer and sysstat-summary.timer. In the Ubuntu 24.04 test, installing the package created the symlinks that enable those units even though the file still said “false”. The tidy way to keep both in sync is sudo dpkg-reconfigure sysstat, which rewrites the variable and enables or disables the units together. RHEL has no such file: systemctl enable --now sysstat is enough, because sysstat.service pulls in both timers through its Also= lines.
Moving to a one-minute interval
The stock collection timer uses OnCalendar=*:00/10 and runs sa1 1 1, which writes a single record per run. That argument pair is correct and does not need changing. To sample every minute, create a drop-in with sudo systemctl edit sysstat-collect.timer:
[Timer]
OnCalendar=
OnCalendar=*:*:00*:*:00 means second zero of every minute; systemd-analyze calendar normalizes it to exactly the same expression as minutely. The empty OnCalendar= line clears the original schedule, since timer entries are additive. Here the union would still be “every minute”, but resetting first avoids surprises with other intervals. systemctl edit reloads the unit files on save, so all that is left is sudo systemctl restart sysstat-collect.timer.
Editing /etc/cron.d/sysstat only helps on hosts without systemd. On Debian and Ubuntu the cron entry is still installed but does nothing under systemd, and Red Hat notes that RHEL 8 and 9 use timers instead of cron.
Retention, compression and optional collectors
HISTORY sets how many days of files are kept. Above 28, sa1 and sa2 switch to saYYYYMMDD names (for example sa20261110) so the 5th of one month does not overwrite the 5th of the previous one. Any script that hard-codes sa05 will break after that change. COMPRESSAFTER compresses files older than the given number of days, but only matters if it is lower than HISTORY. With the shipped defaults (7 and 10 on Ubuntu, 28 and 31 on RHEL), files are deleted before they are ever compressed.
HISTORY=35
COMPRESSAFTER=7
SADC_OPTIONS="-S DISK,SNMP"Code language: JavaScript (javascript)SADC_OPTIONS selects optional activities. -S DISK is still required in sysstat 12.6: a data file written without it made sar -d fail with “Requested activities not available” in the test. Debian, Ubuntu and RHEL all ship it by default. TCP counters for sar -n TCP and sar -n ETCP depend on -S SNMP, which is not enabled out of the box. XALL also adds partitions, filesystems, interrupts and power data, at the cost of larger files. The sadc manual adds a catch: when a day’s file already exists, the activities stored in it win, so a new SADC_OPTIONS value only shows up from the next daily file.
Reading the reports and rebuilding an incident
Without -f, sar reads today’s file. -f /var/log/sysstat/sa04 opens another day, and an interval plus a count (sar 1 5) gives live sampling that does not touch history.
CPU and load
sar -u splits CPU time into %user, %system, %iowait, %steal and %idle, and sar -P ALL breaks it down per core, which exposes a single-threaded process pinning one CPU while the average looks calm. On a virtual machine (VM), %steal is time the virtual CPU spent waiting while the hypervisor served someone else. When it stays high, the bottleneck is on the host, not in the guest.
sar -q reports runq-sz (tasks waiting for a CPU), plist-sz, the 1, 5 and 15-minute load averages and blocked, the number of tasks stuck waiting for input/output (I/O). sysstat 12.6 also exposes kernel Pressure Stall Information through sar -q CPU, sar -q IO and sar -q MEM, showing the share of time tasks were delayed by each resource.
Memory, swap and paging
In sar -r, kbavail is the column to watch: the kernel’s estimate of memory available to new workloads without swapping. A low kbmemfree on its own means little, because Linux uses spare RAM as cache. %commit compares committed memory with RAM plus swap and can exceed 100 % due to overcommit, while kbdirty shows data still waiting to be written to disk.
sar -S shows how much swap is in use, but sar -W is more telling: pages swapped in and out per second are what actually hurt latency. sar -B fills in the rest. pgscank/s counts pages scanned by kswapd in the background, and pgscand/s counts direct scans, which happen when a process needs memory right now and has to wait for reclaim. %vmeff divides reclaimed pages by scanned pages; the manual treats values under 30 % as a sign that virtual memory is struggling.
Disks, network, tasks and kernel tables
sar -d -p lists devices by name (nvme0n1 rather than dev259-0) with tps, await in milliseconds and %util. The manual warns that %util near 100 % only means saturation for devices that serve requests one at a time; RAID arrays and modern SSDs handle requests in parallel, so a rising await is usually the better signal.
For networking, sar -n DEV shows per-interface traffic, -n EDEV errors and drops, -n SOCK socket counts including TIME_WAIT, and -n TCP and -n ETCP active and passive opens and retransmits when SNMP collection is on. sar -w reports processes created and context switches per second. sar -v shows file-nr, the number of file handles open system-wide, which can be compared with /proc/sys/fs/file-max.
Pulling out the incident window
Set the locale before piping sar into awk. Under en_US, sar prints times as “07:42:23 PM”, which adds a field and shifts every column. Under locales such as es_ES or de_DE it uses a decimal comma (“0,15”), which breaks numeric comparisons. LC_ALL=C fixes both, which makes it a safer choice than LC_TIME=C alone.
# Samples where less than 20 % of CPU was idle
LC_ALL=C sar -u -f /var/log/sysstat/sa04 |
awk '$1 ~ /^[0-9]+:/ && $NF ~ /^[0-9.]+$/ && $NF < 20 {print $1, "idle=" $NF, "iowait=" $(NF-2)}'
# Several reports for the same time window in one file
F=/var/log/sysstat/sa04
for o in "-u" "-q" "-r" "-B" "-dp" "-n DEV"; do
echo "== sar $o"
LC_ALL=C sar $o -s 03:00:00 -e 03:30:00 -f "$F"
done > incident-sa04.txtCode language: PHP (php)-s and -e take 24-hour times. By default sar shows timestamps in the time zone of whoever runs it; when analysing a file copied from a server in another region, -t shows them in that server’s original local time instead.
Exporting with sadf
sadf -d prints semicolon-separated records that are easy to turn into CSV (comma-separated values) or load into a database. sadf -j emits JSON (JavaScript Object Notation), and sadf -g renders SVG (Scalable Vector Graphics) charts that open in any browser. Mind the clock: sadf uses UTC unless -t is given.
sadf -d -t /var/log/sysstat/sa04 -- -u -r > cpu-mem.csv
sadf -j /var/log/sysstat/sa04 -- -q > load.json
sadf -g /var/log/sysstat/sa04 -- -d -p > disks.svgCode language: JavaScript (javascript)Symptom, command and what to check
| Symptom | Command | What to check |
|---|---|---|
| CPU is busy | sar -u, sar -P ALL, sar -q | High %user or %system, runq-sz above the CPU count |
| CPU is idle but the box is slow | sar -u, sar -q, sar -d -p | %iowait, blocked, disk await and aqu-sz |
| Memory is tight | sar -r, sar -W, sar -B | Falling kbavail, pswpin/s and pswpout/s, pgscand/s, %vmeff |
| Remote calls time out | sar -n EDEV, sar -n ETCP, sar -n SOCK | rxdrop/s, retrans/s, tcp-tw |
| A VM feels sluggish | sar -u, sar -P ALL | Sustained %steal |
| “Too many open files” | sar -v | file-nr against file-max, plus the per-process limit |
| Sudden load spike | sar -q, sar -w | ldavg-1, proc/s, cswch/s |
That “Too many open files” row needs a caveat. The error usually comes from a per-process limit (ulimit -n, or LimitNOFILE in a systemd unit), which sar cannot see. file-nr only catches system-wide exhaustion, so /proc/<PID>/limits is worth checking too. For live troubleshooting, sar pairs well with vmstat, the first command to run on a slow Linux server, and with these six warning signs for finding the bottleneck.
When sar is not enough
sar is per host, has no alerting and stores its history on the same machine it is watching. If the disk fills up or the instance is terminated, the history goes with it. It also has no view of the application layer: request latency, queue depth and slow queries are outside its reach.
Once a fleet grows past a handful of servers, or someone needs to be woken up when a threshold is crossed, another layer makes sense. Prometheus with node_exporter scrapes similar host metrics into a central store, joins them with application metrics and works with Alertmanager and Grafana. Netdata gives per-second dashboards with very little setup. Zabbix covers inventory, templates and alerting across mixed environments, with or without agents. None of these makes sar redundant: it can stay on as a cheap local backup and a second opinion when the central system was also affected by the incident.
Frequently asked questions
Where does sar store its data?
On Debian and Ubuntu, in /var/log/sysstat/saDD; on RHEL and its rebuilds, in /var/log/sa/saDD. If HISTORY is set above 28 days, files are named saYYYYMMDD instead.
How do I change the sar collection interval to one minute?
On systemd hosts, run systemctl edit sysstat-collect.timer, clear the schedule with an empty OnCalendar= line and add OnCalendar=*:*:00. Editing the cron file only works on systems without systemd.
Why does sar -d or sar -n TCP say the activity is not available?
Those collectors are optional. Disk data needs -S DISK and TCP data needs -S SNMP in SADC_OPTIONS, and the change applies from the next daily file.
Can sar replace Prometheus, Netdata or Zabbix?
Not entirely. sar is good for after-the-fact analysis on each server, but it does not centralize data or send alerts, so it works best as a complement.
Sources:
- bektiaw, “sar: The Flight Recorder Already Running on Your Server”, Medium, November 5, 2026.
- sar manual page (sysstat 12.6, Ubuntu 24.04)
- sadc manual page (sysstat 12.6, Ubuntu 24.04)
- sadf manual page (sysstat 12.6, Ubuntu 24.04)
- sysstat project: features
- sysstat repository on GitHub
- Red Hat: How to use SAR to monitor system performance in RHEL
- sysstat package spec for RHEL 9 (CentOS Stream 9, AlmaLinux Git)