0

I made a search but couldn't find anything similar.

I have huge codebases ( AOSP or Yocto ) stored in ext4 formatted NVME drives ( such as SAMSUNG 980Pro ). These codebases usually consist of hundreds of thousands of files in thousands of subdirectories.

When I search for a file with find , I usually do:

find . -iname "filename.txt"

I don't want to operate on the files, I just want to find the file.

I feel this is not multithreaded ( please correct me if I am wrong ).

Please note that, I know that searching for files are generally IO bound, but my drives are fast, and I believe a tool that can iterate through subdirectories in multiple threads can dramatically speed up finding results.

Please correct me if I am wrong. If I am right, and if there is such tool, please let me know.

Improve this question
3
  • You're mostly going to be io bound until the whole filesystem metadata ends up in cache, even with fast nvmes. That will be hard to separate from the context switch overhead you incur for every single directory that find looks into. Anyways, yeah find is logically structured as sequential thing to enable some of the more advanced control options Commented Dec 10, 2023 at 20:46
  • But, hint, sudo perf top -a during operation will show you how much time the CPU cores spend waiting, searching through ext4 data structures, and exchanging between Userland and Kernel-Land Commented Dec 10, 2023 at 20:47
  • @MarcusMüller thank you but see my below answer. Commented Dec 10, 2023 at 21:12

1 Answer 1

Reset to default
1

I find the tool that I am looking for: https://github.com/sharkdp/fd

Here is my test result on actual system:

stulluk /media/WORK/RDK/521 $  time find . -iname "dsAudio.c"
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-amlogic-tv/gitAUTOINC+08aefcf7c0-r0/git/dsAudio.c
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/.pc/AMLTVT5W-134-Volumelevel_setting_to_0.patch/rpc/srv/dsAudio.c
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/.pc/0001-Add-audio-default-persistent-settings.patch/rpc/srv/dsAudio.c
./build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c
./build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c

real    0m11,925s
user    0m4,753s
sys 0m7,103s
stulluk /media/WORK/RDK/521 $  time find . -iname "dsAudio.c"
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-amlogic-tv/gitAUTOINC+08aefcf7c0-r0/git/dsAudio.c
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/.pc/AMLTVT5W-134-Volumelevel_setting_to_0.patch/rpc/srv/dsAudio.c
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/.pc/0001-Add-audio-default-persistent-settings.patch/rpc/srv/dsAudio.c
./build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c
./build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c

real    0m11,898s
user    0m4,618s
sys 0m7,211s
stulluk /media/WORK/RDK/521 $  time find . -iname "dsAudio.c"
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-amlogic-tv/gitAUTOINC+08aefcf7c0-r0/git/dsAudio.c
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/.pc/AMLTVT5W-134-Volumelevel_setting_to_0.patch/rpc/srv/dsAudio.c
./build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/.pc/0001-Add-audio-default-persistent-settings.patch/rpc/srv/dsAudio.c
./build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c
./build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c

real    0m11,880s
user    0m4,598s
sys 0m7,215s
stulluk /media/WORK/RDK/521 $  time fd dsAudio.c
build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c
build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-amlogic-tv/gitAUTOINC+08aefcf7c0-r0/git/dsAudio.c

real    0m4,495s
user    0m23,689s
sys 0m41,132s
stulluk /media/WORK/RDK/521 $  time fd dsAudio.c
build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c
build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-amlogic-tv/gitAUTOINC+08aefcf7c0-r0/git/dsAudio.c

real    0m4,491s
user    0m24,473s
sys 0m40,966s
stulluk /media/WORK/RDK/521 $  time fd -HI dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/.pc/0001-Add-audio-default-persistent-settings.patch/rpc/srv/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/.pc/AMLTVT5W-134-Volumelevel_setting_to_0.patch/rpc/srv/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-amlogic-tv/gitAUTOINC+08aefcf7c0-r0/git/dsAudio.c
build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c
build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c

real    0m5,446s
user    0m17,727s
sys 0m58,009s
stulluk /media/WORK/RDK/521 $  time fd -HI dsAudio.c
build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c
build/tmp/work/armv7at2hf-neon-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-headers/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/.pc/AMLTVT5W-134-Volumelevel_setting_to_0.patch/rpc/srv/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/.pc/0001-Add-audio-default-persistent-settings.patch/rpc/srv/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/cli/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings/rdkv-2023q2-dunfell+gitrAUTOINC+40fe1c3597-r0/git/rpc/srv/dsAudio.c
build/tmp/work/mesont5w_5.4_lib32_at301-rdkmllib32-linux-gnueabi/lib32-devicesettings-hal-amlogic-tv/gitAUTOINC+08aefcf7c0-r0/git/dsAudio.c

real    0m5,431s
user    0m19,166s
sys 0m56,250s
stulluk /media/WORK/RDK/521 $

Also see: https://github.com/sharkdp/fd#benchmark

Improve this answer
3
  • Nice! That's good to know. Solid speed up, but not by a huge factor. Especially: note how this is circa 3x faster, but "occupied" about 6x as much CPU time. Just out of interest: how many CPU cores do you have? Commented Dec 11, 2023 at 2:42
  • 1
    Sorry for late answer: 8 core 16 threads: Ryzen 5700G. I suspect it is still more I/O bound, but given that I have a very fast NVME drive, I really benefited from fd-find. Huge kudos to the authors. Here is my bash alias for it: alias fd='fd --glob -IH ' Commented Jan 2, 2024 at 11:20
  • Thanks for the answer! Yeah, nice! I think that about maxes out the speed you can get from classically searching directories for files. Commented Jan 2, 2024 at 12:28

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.