Questions tagged [buffer]
This tag is for questions about kernel buffer caches, including pipe buffers. These are used to store the recently accessed files and/or frequently accessed files.
178 questions
3
votes
2
answers
161
views
What explains this very odd behavior of GNU grep interacting with buffering and pipes and how to stop it?
This is best illustrated with an example I feel:
{ printf 'foo\nbar\n' ; sleep 2 ; } | grep -m1 foo
{ printf 'foo\n' ; sleep 2 ; printf 'bar\n' ; sleep 2 ; } | grep -m1 foo
Both of these commands, ...
- 171
4
votes
1
answer
608
views
Can I make a pipe save data to disk instead of blocking until the reader can continue?
Are there any established tools where the pipe spills to disk rather than blocking the upstream process?
As an example, in a traditional pipeline A | B, we get the following behavior when B does not ...
- 151
7
votes
2
answers
2k
views
Configure Linux to regularly sync cached data to disk
The sync command "writes any data buffered in memory out to disk". As far as I understand, data may be buffered in memory for very long time, even if the disks have no activity.
How can I ...
- 597
1
vote
0
answers
65
views
Radix Tree vs Multi-level bitmap
The Linux kernel (at least before using XArrays as far as I'm aware, which to my knowledge are wrappers around Radix Trees anyway) uses radix trees in its address_space structs which every file has. ...
- 11
0
votes
0
answers
34
views
How can I get the _real_ progress of dd, ignoring fast/staging buffers? [duplicate]
I'm using the dd command to write to a USB stick. My command is pretty straightforward:
dd if=myimage.iso of=/dev/sdd bs=1M status=progress
and indeed, I seem to be getting the progress reported:
...
- 10.9k
0
votes
1
answer
402
views
Is my SSD dead? can't mount. I/O errors, 29k+ fsck errors
So I work with a couple of people and we ssh into a Linux server for work. There are several SSDs, and one of the non-boot Samsung 870 SSDs crashed the other day in the morning. This led to an ...
0
votes
2
answers
152
views
Terminating the whole pipeline at once when its last process ends
Given the pipeline
printf '%s\n' 1 2 3 4 5 6 7 8 9 |
while read -r num
do
echo "$num" > /dev/stderr
echo "$num"
done |
while read -r num
do
echo $(( $num * 10 ))
[ &...
- 449
0
votes
1
answer
133
views
Is it possible to set larger buffers for file access on linux
I have a process thats reading the whole filesystem and hashing files. It slows down (by 4x or so) because the reads are causing a lot of seeking. Small chunks of each file are being read by each of 4 ...
- 153
0
votes
1
answer
127
views
How do Kernel use pagecache?
I have a problem with pagecache that I don't understand.
As I understand, the pagecache will serve as disk cache for reading from and writing to disk. But I don't know how the kernel can map 10G ...
20
votes
1
answer
2k
views
Why does unbuffer -p mangle its input?
$ seq 10 | unbuffer -p od -vtc
0000000 1 \n 2 \n 3 \n 4 \n 5 \n 6 \n 7 \n 8 \n
Where did 9 and 10 go?
$ printf '\r' | unbuffer -p od -An -w1 -vtc
\n
Why was \r changed to \...
- 583k
0
votes
1
answer
874
views
Linux: Getting the kernel buffer size for a socket
I have a C application which receives a lot of data over a TCP socket. Is it somehow possible to get the kernel buffer size for that file descriptor / socket? I would like to know how much data is ...
- 213
0
votes
1
answer
1k
views
conditional flush tcp sockets on linux
whenever I write data to a tcp socket, it waits until its filled with enough data ex: 64k (max packet size) then it sends data to the wire.
this misses the data at the right time for the other end.
...
1
vote
1
answer
320
views
Create a file for writing with controlled flushing to disk in large chunks
On Linux I have a process (ffmpeg) that writes very slowly (even slower than 1kb / s sometimes) to disk. Ffmpeg can buffer this to 256kb chunks that get written infrequently but ffmpeg hangs ...
- 153
0
votes
1
answer
896
views
How to clear the /dev/kmsg ring buffer?
Is there a way for me to force clear the /dev/kmsg kernel ring buffer?
- 101
7
votes
1
answer
680
views
Why does tar's handling of stdout and - differ?
After a comment from OP, I discovered that /dev/stdout gives blocks of 10 KiB even after disabling buffering, but - does not. Why is this? I could not find anything regarding this in man tar nor man ...
- 460