time touch /tmp/test.dat
real 0m1.03s
user 0m0.00s
sys 0m1.02s
A full second of sys-mode time to create a file in /tmp
. That can become unbearable for ksh scripts that open dozens of files in /tmp
for subshell handling.
strace shows the time at the openat
call:
strace -tttT touch /tmp/test.dat
. . . [clip] . . .
1737560680.004656 close(3) = 0 <0.000013>
1737560680.004750 openat(AT_FDCWD, "/tmp/test.dat", O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666) = 3 <1.370377>
1737560681.375253 dup2(3, 0) = 0 <0.000026>
. . . [clip] . . .
Current CPU during these tests has 70-80% idle. Plenty of memory. /tmp
is only 1% used, though it does have a lot of empty directories beneath it (5,268). Server has been up 96 days. We had this problem on another server which panicked for some reason and rebooted. After the reboot, the problem was gone - access to /tmp
was fast again. So something over time is causing /tmp
access to get slower and slower, and a reboot clears it.
OS version: 5.4.17-2136.322.6.4.el8uek.x86_64
, built by Oracle (this is an Exadata compute node)
/tmp mount: /dev/mapper/VGExaDb-LVDbTmp xfs 45G 45M 45G 1% /tmp
Oracle support threw up their hands (didn't really try, to be honest). Any Unix gurus out there have ideas on what I can look into? What kinds of things can cause this to be so slow?
find /tmp
)? Have you run fsck on that volume? Is the output of anfsck
run on the other system after the reboot available? Can you delete some of these empty directories? Is it an option to use a different filesystem on that block device?