Skip to main content
Software Engineering

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

5
  • 3
    Depends on your disk. High end PCIe/NVME storage may have up to four PCI lanes per disk. So in theory reading a single file can be sped up by using up to 4 threads. The parallelism goes even higher with RAID arrays or sharded filesystems Commented Sep 1, 2020 at 6:33
  • 3
    @slebetman: The number of PCIe lanes is totally irrelevant; that's just bandwidth. The relevant measure is the number of command queues. In SATA, that number is 1. All IO's from all threads end up mixed in that one queue, and that includes I/O from the OS too. NVMe allows up to 65536 queues; clearly a quantum jump. Real SSD's today have not nearly that many queues, 8 would not be unusual. Commented Sep 1, 2020 at 14:27
  • @MSalters But command queues does not affect how parallel the actual data transfer can be. You can have more than 4 queues fetching data in parallel from multiple flash chips but all that data would still be serialized into a maximum of 4 lanes to be transferred in parallel. The completion of transfer of the PCI lanes are not necessarily synchronous which allows you to use your CPU time to process the current chunk of completed transfer while other lanes are completing. Commented Sep 1, 2020 at 15:08
  • Neither the number of PCIe lanes or commands queues are relevant. If you read from multiple queues or PCIe lanes, you're dividing bandwidth to multiple readers, but you're not actually going to be reading any faster, and in most cases a single-threaded newline counter would be able to keep up with the full read bandwidth. If you have a really fast, high end SSD array, it might be possible to read faster than a single thread can handle the line processing, but machines with the type of SSD that can saturate the CPU thread for such a simple workload is still fairly uncommon. Commented Sep 1, 2020 at 15:08
  • In either case, programs don't really read from SSD directly, the OS will parallelize the read you multiple PCIe lanes or queues if it has nothing else it needs to read. So these are irrelevant anyway. Commented Sep 1, 2020 at 15:12