Skip to main content
added 421 characters in body
Source Link

AFAIK theThe only way to avoid cachingprevent a file from being cached is to use O_DIRECT to begin with. dd is your friend.
You can use dd with iflag=directIf you don't want to read a file withuse O_DIRECT and pass it to any utility, notice that can read from stdin.
FADV_DONTNEED doesn't actually flush filesthe damage might be done already afterwards (other more important things might be discarded from the Linux pagecachecache).

# fgrep MemFree: /proc/meminfo
4691788 KB
# dd if=massivefile of=/dev/null bs=2M  
# fadv <massivefile
# fgrep MemFree: /proc/meminfo
5097264 KB

where fadv.c is (use stdin to make the code ultra simple):

#include <fcntl.h>
#include <sys/types.h>
#include <unistd.h>

int main(int argc, char **argv)
{ posix_fadvise(0, 0, lseek(0, SEEK_END0, 0SEEK_END), POSIX_FADV_DONTNEED);
  return 0;
}

ButNotice that afterwards there is more free memory wasn't freed (largethan before, meaning data that was in the cache before was thrown out. However if you read the file is >10GB large)with O_DIRECT:

# free  
             total       used       free     shared    buffers     cached  
Mem:       8108404    7953940     154464     402172          0    5910868  
-/+ buffers/cache:    2043072    6065332  
Swap:      1048568          0    1048568
  # fgrep MemFree: /proc/meminfo
  MemFree:         5091464 kB
  # dd if=verylargefile bs=2M of=/dev/null iflag=direct
  6141+1 records in
  6141+1 records out
  12880669515 bytes (13 GB) copied, 72,7641 s, 177 MB/s
  # fgrep MemFree: /proc/meminfo
  MemFree:         5074164 kB

Now there's a little less memory free, meaning other activity used up some memory, but the actual read of this 13GB file didn't wash the cache.

AFAIK the only way to avoid caching a file is to use O_DIRECT to begin with.
You can use dd with iflag=direct to read a file with O_DIRECT and pass it to any utility that can read from stdin.
FADV_DONTNEED doesn't actually flush files from the Linux pagecache.

# dd if=massivefile of=/dev/null bs=2M  
# fadv <massivefile

where fadv.c is:

#include <fcntl.h>
#include <sys/types.h>
#include <unistd.h>

int main(int argc, char **argv)
{ posix_fadvise(0, 0, lseek(0, SEEK_END, 0), POSIX_FADV_DONTNEED);
  return 0;
}

But memory wasn't freed (large file is >10GB large)

# free  
             total       used       free     shared    buffers     cached  
Mem:       8108404    7953940     154464     402172          0    5910868  
-/+ buffers/cache:    2043072    6065332  
Swap:      1048568          0    1048568

The only way to prevent a file from being cached is to use O_DIRECT. dd is your friend.
If you don't want to use O_DIRECT, notice that the damage might be done already afterwards (other more important things might be discarded from the cache).

# fgrep MemFree: /proc/meminfo
4691788 KB
# dd if=massivefile of=/dev/null bs=2M  
# fadv <massivefile
# fgrep MemFree: /proc/meminfo
5097264 KB

where fadv.c is (use stdin to make the code ultra simple):

#include <fcntl.h>
#include <sys/types.h>
#include <unistd.h>

int main(int argc, char **argv)
{ posix_fadvise(0, 0, lseek(0, 0, SEEK_END), POSIX_FADV_DONTNEED);
  return 0;
}

Notice that afterwards there is more free memory than before, meaning data that was in the cache before was thrown out. However if you read the file with O_DIRECT:

  # fgrep MemFree: /proc/meminfo
  MemFree:         5091464 kB
  # dd if=verylargefile bs=2M of=/dev/null iflag=direct
  6141+1 records in
  6141+1 records out
  12880669515 bytes (13 GB) copied, 72,7641 s, 177 MB/s
  # fgrep MemFree: /proc/meminfo
  MemFree:         5074164 kB

Now there's a little less memory free, meaning other activity used up some memory, but the actual read of this 13GB file didn't wash the cache.

Source Link

AFAIK the only way to avoid caching a file is to use O_DIRECT to begin with.
You can use dd with iflag=direct to read a file with O_DIRECT and pass it to any utility that can read from stdin.
FADV_DONTNEED doesn't actually flush files from the Linux pagecache.

# dd if=massivefile of=/dev/null bs=2M  
# fadv <massivefile

where fadv.c is:

#include <fcntl.h>
#include <sys/types.h>
#include <unistd.h>

int main(int argc, char **argv)
{ posix_fadvise(0, 0, lseek(0, SEEK_END, 0), POSIX_FADV_DONTNEED);
  return 0;
}

But memory wasn't freed (large file is >10GB large)

# free  
             total       used       free     shared    buffers     cached  
Mem:       8108404    7953940     154464     402172          0    5910868  
-/+ buffers/cache:    2043072    6065332  
Swap:      1048568          0    1048568