55
votes
Accepted
Why don't reads from /dev/zero count as IO_RBYTES?
They do count as I/O, but not of the type measured by the fields you’re looking at.
In htop, IO_RBYTES and IO_WBYTES show the read_bytes and write_bytes fields from /proc/<pid>/io, and those ...
- 481k
43
votes
Accepted
How to redirect stdout to a file, and stdout+stderr to another one?
Problem is that when you redirect your output, it's not available anymore for the next redirect. You can pipe to tee in a subshell to keep the output for the second redirection:
( cmd | tee -a file2 )...
- 23.2k
31
votes
Accepted
How does memory mapping a file have significant performance increases over the standard I/O system calls?
Memory mapping a file directly avoids copying buffers which happen with read() and write() calls. Calls to read() and write() include a pointer to buffer in process' address space where the data is ...
- 15.8k
22
votes
Accepted
systemd connect to stdin/stdout after service has started
I can think of multiple ways to do this. Of course, each has its own caveats. ...
- 1,949
21
votes
Accepted
Differences Between `/dev/null` and Devices Under `null_blk` Driver
/dev/null is handled by a device driver, but it’s a very simple one.
The major difference between /dev/null and null_blk is that the former is a character device, the latter a block device. The former ...
- 481k
20
votes
How does memory mapping a file have significant performance increases over the standard I/O system calls?
First, in most IO operations the characteristics of the underlying storage hardware dominates the performance. A poorly-configured RAID5 array of twenty-nine S-L-O-W 5400 rpm SATA disks on a slow, ...
- 4,008
17
votes
How to redirect stdout to a file, and stdout+stderr to another one?
With zsh:
cmd >& out+err.log > out.log
In append mode:
cmd >>& out+err.log >> out.log
In zsh, and provided the mult_ios option has not been disabled, when a file descriptor ...
- 585k
16
votes
Differences Between `/dev/null` and Devices Under `null_blk` Driver
/dev/null is a "character device", on which all reads return empty and all writes complete fully with no additional side-effects. These are the semantics since V5 UNIX in 1974, and are ...
- 114k
15
votes
Graceful shutdown with suspend job hanging in syscall
I know this doesn't exactly answer the question but maybe it will help somewhat.
You or someone else who will read this.
I had (or still have) similar problem now. Suspend didn't finish returning to ...
- 161
14
votes
How to Throttle per process I/O to a max limit?
The answer of fche is a very good hint, thanks for that, although it dosn't really solve the problem cause the question was to limit a process to a specific bandwidth.
I would suggest something like ...
- 267
13
votes
Can I watch the progress of a `sync` operation?
Based on the other answers I made this over-bloated one-liner that tracks summary progress as well as device-specific progress:
# Create the command
watchSync() {
watch -n1 'grep -E "(Dirty|...
- 900
13
votes
how to find out iowait per process on linux with top command
If you need to monitor processes in realtime, use iotop instead.
top can show total amount of iowait of all processes in wa parameter:
wa, IO-wait : time waiting for I/O completion
In addition with ...
- 2,583
11
votes
Accepted
Bash IO redirection, open & close 'fd', can someone explain?
Some translations for Bourne-like shells:
system call
shell interface
shells
effect
open("file", O_RDONLY)
exec 3< file
all
Opens on fd 3 at the start for reading only¹
open("file&...
- 585k
10
votes
How can I specify that curl (via command line) overwrites a file if it already exists?
Pass --clobber. This also covers server-provided filename when using -J.
- 201
10
votes
Accepted
Is there any way to kill or end a process in "disk sleep"
If a process is in an uninterruptible sleep state, then no, you cannot kill or otherwise end the process until it exits that state. While in that state, the process has invoked a system call on the ...
- 14.7k
10
votes
Is memory mapped I/O only used internally by OS, not exposed to and used by programmers on top of Linux?
On Linux, MMIO is possible from user-space using mmap on /dev/mem. For example, the X server does
fd = open("/dev/mem", O_RDWR);
if (ioBase == NULL) {
ioBase = (volatile unsigned char *) mmap(0, ...
- 481k
10
votes
Accepted
Difference between posix_fadvise and readahead
Let's look at the code that these syscalls cause to be executed.
Citing mm/readahead.c here:
611 │ ssize_t ksys_readahead(int fd, loff_t offset, size_t count)
612 │ {
613 │ ssize_t ret;
...
- 51.2k
9
votes
Accepted
iostat: what is exactly the concept of merge
A merge happens when two i/o requests can be collapsed into one single longer-length request. For example, a write to block 1234 followed by a write to block 1235 can be merged into a single i/o ...
- 54.7k
9
votes
Accepted
How is /dev/null implemented?
In Linux, /dev/null is implemented by a number of functions, including write_null:
static ssize_t write_null(struct file *file, const char __user *buf,
size_t count, loff_t *ppos)
{
...
- 481k
8
votes
iotop but for particular disk?
This isn't an ideal answer, but this will tell you, every second, what process wrote the most, and how much it wrote, to a given disk (in this case /dev/sda):
dstat -tdD /dev/sda --top-io
You'll see ...
- 237
8
votes
Accepted
Is a device being a block or character device determined purely by hardware?
No, it is determined by the OS.
FreeBSD Architecture Handbook dated 2018-09-23 11:38:04.
9.4 Block Devices (Are Gone)
Other UNIX® systems may support a second type of disk device known as block ...
- 53.5k
8
votes
Accepted
`wc -l` alternative that displays the count, as it receives the lines
awk '{print NR}'
This command prints a new number for each line encountered. If the final line is complete then the last number will agree with what wc -l would say. If the final line is incomplete ...
- 24.4k
8
votes
Why is dd so slow on a NVMe SSD?
NAND flash media like SSDs have a minimum block erase size, which means that if you dd data on them with a smaller block size, the SSD controller has to erase blocks several times to write to ...
- 838
7
votes
What exactly is iodepth in fio?
will that not be a trivial?
Assume direct IO, as required for iodepth= to work.
A sequential job with iodepth=2 will submit two sequential IO requests at a time.
A sequential job with numjobs=2 will ...
- 53.5k
7
votes
Accepted
System lags when doing large R/W operations on external disks
How can I remedy it?
When you write a disk image, use dd with oflag=direct. The O_DIRECT writes will avoid writing the data through the page cache. Note oflag=direct will require a larger block size,...
- 53.5k
7
votes
Why does I/O cause my system to become almost nonresponsive, and how to fix it?
Let's look at some of the things in your question:
As soon as I resume the output, the system begins slowing down, and within a minute it is again almost unusable.
This gives a hint that something ...
- 130k
7
votes
Accepted
Why does io_uring have a layer of indirection for the submission queue?
I reached out to the author of the paper and got some clarity on this.
The reason there's a layer of indirection is because some applications might want to hold submission event data inside their ...
- 211
7
votes
Accepted
update-initramfs: failed for /boot/initrd.img-4.15.0-112-generic with 1
The real error message tucked away in all that output is Write error : cannot write compressed block which, by an overwhelming margin, is found to be the result of insufficient free space available ...
- 523
6
votes
Determining Specific File Responsible for High I/O
You can use sysdig which has a chisel named topfiles_bytes that does exactly that.
Examples:
sysdig -c topfiles_bytes
sysdig -c topfiles_time
List all available chisels:
sysdig -cl
- 61
6
votes
iotop but for particular disk?
Check out fatrace:
For example, cd into the partition you want to (in my case, mount point /hdd) monitor and run
sudo fatrace -c -t
Then, for example a touch /hdd/x will show:
16:11:05.278541 touch(...
- 945
Only top scored, non community-wiki answers of a minimum length are eligible