In Linux, every system call carries a cost. While they are essential, each one involves a context switch from user space to kernel space, consuming both CPU cycles and time. Optimizing and reducing unnecessary system calls can make a noticeable difference, especially on servers processing thousands of requests per second.
One little-known but powerful example is the TZ environment variable, which defines a process’s timezone. Setting it correctly can eliminate thousands of redundant system calls, improve application performance, and free up resources in production environments.
The Root of the Problem: time and localtime
In Linux, two functions are frequently used together:
time: returns the number of seconds since January 1, 1970.localtime: converts that number into a human-readable date based on the local timezone.
The issue arises because every time localtime runs, glibc checks whether the /etc/localtime file has changed. To do so, it makes a stat system call. This happens even on servers that never modify their timezone, such as those permanently running on UTC.
The result: each log entry, SQL query, or formatted timestamp can trigger hundreds of extra stat calls, negating the performance benefits of vDSO for time.
How It Was Detected
This behavior was observed in Ubuntu Precise (12.04) and Ubuntu Xenial (16.04), though it affects other distributions as well.
With a simple C program calling time and localtime in a loop, and running it under strace, one can clearly see repeated stat("/etc/localtime") calls.
On production-like workloads, this pattern can generate nearly 15,000 calls in just 30 seconds — roughly 500 calls per second that add no real value.
The Fix: Setting the TZ Variable
The solution is remarkably simple: explicitly tell glibc which timezone file to use via the TZ variable.
For example:
export TZ=:/etc/localtime
Code language: JavaScript (javascript)With this, glibc reads the file once, caches it in memory, and never calls stat again as long as TZ remains unchanged.
Running the same test program with TZ set shows that /etc/localtime is read a single time, eliminating thousands of redundant system calls.
Impact on Production Servers
The performance gains depend on workload:
- For web applications (such as Ruby on Rails) that format dates constantly, the difference can be very significant.
- On systems with little
localtimeusage, the impact is smaller, but it remains a best practice.
In one real-world test, setting TZ reduced stat calls from 14,925 down to just 8 over 30 seconds — saving about 10,000 operations in half a minute.
How to Apply the Change
There are several ways to configure this:
- In the shell (session-only):
export TZ=:/etc/localtime - In a systemd service (persistent):
[Service] Environment="TZ=:/etc/localtime" - Inside application code:
setenv("TZ", ":/etc/localtime", 1);
The key is ensuring TZ remains constant during the process’s lifecycle. If the system timezone changes, simply restart the affected service.
Technical Context: vDSO and glibc
This case highlights the interaction between two core Linux components:
- vDSO (Virtual Dynamic Shared Object): a mechanism allowing certain system calls (like
timeorgettimeofday) to be executed without entering the kernel, reducing latency. - glibc: the standard C library that implements functions such as
localtime. Whiletimebenefits from vDSO,localtimecan reintroduce overhead by repeatedly callingstatunlessTZis set.
On servers where the timezone never changes, this verification becomes wasted effort.
Beyond TZ: The Lesson
This finding underscores a key principle in systems administration: optimizations often hide in small details.
- Tools like
straceandltracereveal hidden patterns of system calls. - Simple configuration tweaks can eliminate thousands of unnecessary operations.
- Even invisible improvements translate into faster, more stable servers.
Conclusion
A simple adjustment — defining the TZ environment variable — can deliver substantial benefits in Linux environments.
In high-load systems, this change removes thousands of redundant calls, optimizes performance, and frees up resources.
It’s a reminder that efficiency doesn’t always require new hardware: often, it comes from understanding how software really works.
Frequently Asked Questions (FAQ)
What value should I use for TZ?
The most common is :/etc/localtime, but you can point to any valid timezone file, such as /usr/share/zoneinfo/UTC.
Does this affect all processes?
Yes, as long as they inherit the TZ variable. Setting it via systemd ensures persistence across services.
What if the system timezone changes?
Processes will continue using the TZ value they inherited. Restart them to apply the new timezone.
Should this always be enabled?
Yes, on production servers with fixed timezones (e.g., UTC). On laptops or desktops that switch timezones often, it may be better to leave automatic checks enabled.
