Timeline for Why Linux read() avoids using full 2 GiB in one call
Current License: CC BY-SA 4.0
16 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 2 at 18:53 | vote | accept | malat | ||
| Apr 30 at 9:24 | comment | added | Paul_Pedant |
@ilkkachu True that read() does not restrict its args for any device, but I don't see tty. pipe, or network needing to consider a 2GB limit. Things that can do transfers that size should consider optimal alignment. I go back to devices that interleaved disk sectors because they were too slow to read consecutive sectors, so would lose a complete revolution per sector. I was messing with dd recently (because I cannot make fallocate work correctly on Mint). dd if=/dev/null of=Q.Tera bs=1T seek=1 fails because it tries to allocate a 1TB input buffer for /dev/null. Ouch!
|
|
| Apr 30 at 6:20 | answer | added | jpa | timeline score: 22 | |
| Apr 30 at 6:02 | comment | added | jpa | Cross-site duplicate: stackoverflow.com/questions/70368651/… | |
| Apr 29 at 20:24 | comment | added | ilkkachu |
@Paul_Pedant, well, yes, but read() can be used on other things too, not just block devices. (And AFAIK,on Linux, it doesn't enforce any alignment for block devices either.)
|
|
| Apr 29 at 20:02 | comment | added | Paul_Pedant | @ilkkachu Agree that read() can deal with any file-pos and length, but a block device is just that: any data that is not precisely block-aligned would have to be cached and trimmed somewhere. The first line of the man page says "attempts to read up to count bytes", so the existing limit is both conformant, and avoids those overheads. | |
| Apr 29 at 16:12 | history | became hot network question | |||
| Apr 29 at 14:19 | history | edited | Kusalananda♦ | CC BY-SA 4.0 |
More flow in title, tiny markup fixes to text
|
| Apr 29 at 12:36 | comment | added | Stéphane Chazelas | See github.com/torvalds/linux/commit/…. "We want to protect lower layers from (the sadly all too common) overflow conditions, but prefer to do so by chopping the requests up, rather than just refusing them outright" | |
| Apr 29 at 12:30 | comment | added | Stéphane Chazelas |
FWIW, it's down to a #define MAX_RW_COUNT (INT_MAX & PAGE_MASK) in the code (in include/linux/fs.h). git log/blame might allow you to get back to the rationale
|
|
| Apr 29 at 10:05 | answer | added | Marcus Müller | timeline score: 9 | |
| Apr 29 at 9:30 | comment | added | ilkkachu |
@Paul_Pedant but read() already needs to be able to read an arbitrary amount of bytes, at an arbitrary file position, so even if 0x7fffffff = 2147483647 is a prime, it shouldn't matter.
|
|
| Apr 29 at 8:42 | comment | added | Paul_Pedant | Transfers for data that is not block-aligned (let alone word-aligned) will have an overhead during transfers to deal with part-blocks. Also, media that are capable of DMA (Direct Memory Access, which bypasses CPU usage) are more likely to be 32-bit limited internally. | |
| Apr 29 at 8:29 | history | edited | Marcus Müller | CC BY-SA 4.0 |
edited title
|
| Apr 29 at 8:22 | answer | added | Martin Kealey | timeline score: 7 | |
| Apr 29 at 8:11 | history | asked | malat | CC BY-SA 4.0 |