Request to add multithreading support

[supply9243]

I don't speak English, these are translated by software:

I've noticed that the software creates only one thread at a time and verifies a single file, then closes the thread.
We need to verify a large number of files, and the disk I/O consumed by verifying a single file is limited.
This approach does not fully utilize the performance of multi-core CPUs and SSDs (some of which can achieve speeds up to 12GB/s).
If you are concerned about affecting the HDD, you can make this an optional setting.

[OV2]

RapidCRC uses multiple threads, but only for calculating different hash types at the same time. You are correct that only one file is calculated at a time.
The current structure makes it difficult to change this, but it is on my todo.

[vlakoff]

You’re absolutely right that verifying multiple files in parallel can improve performance, but it’s also important not to overload the storage device. Even with SSDs, reading too many files at once can reduce throughput instead of increasing it.

A few points that might help with designing a safe multithreaded strategy:

• HDDs vs SSDs can already be distinguished reliably using the Windows DeviceSeekPenaltyProperty.
  HDDs incur a seek penalty, SSDs do not.

• For SSDs, it’s possible to detect whether the device is NVMe, using several Windows APIs:

  • StorageAdapterProperty → BusType == BusTypeNvme (no false positives)
  • Checking the storage driver → stornvme.sys indicates NVMe
  • StorageDeviceProtocolSpecificProperty → protocol type NVMe
    These methods are not perfect, but they allow identifying devices that are certainly NVMe.
    False negatives are acceptable; false positives must be avoided.

• With this information, a safe scheduling strategy could be:

  • HDD: process 1 file at a time
  • SATA SSD: process 2 files in parallel
  • NVMe SSD: process up to 4 files in parallel
    This avoids overloading slower devices while still taking advantage of faster ones.

• Parallel reads from different physical drives should of course be allowed.
  (Handling shared bandwidth on multi‑device controllers is a deeper topic and probably out of scope.)

This kind of adaptive approach would give good performance on modern systems while remaining safe for older HDD‑based setups.

[supply9243]

You’re absolutely right that verifying multiple files in parallel can improve performance, but it’s also important not to overload the storage device. Even with SSDs, reading too many files at once can reduce throughput instead of increasing it.

In practice, this issue mainly affects mechanical hard drives. On SSDs, even when Task Manager shows 100% disk usage, users can still open and interact with other files—just with some slowdown rather than complete blocking.

For example:
When verifying torrent files in qBittorrent, the default number of threads is relatively low, resulting in slow verification speed. However, it provides an option to configure disk I/O threads. By increasing this value significantly, verification runs at 100% disk utilization, yet it does not noticeably interfere with other read/write operations.

Suggestions:

• Check whether it is possible to identify and manage disks individually (e.g., retrieve unique identifiers per drive).

• If supported, implement a UI panel that allows users to:

  • Configure the number of threads per disk
  • Persist these settings (saved by default)
  • Manually reset or clear configurations

• If per-disk management is not feasible:

  • Provide two modes: Automatic and Custom
  • In Custom mode, allow users to temporarily set a thread count

• For resource management:

  • Set CPU, disk I/O, and memory priorities to low by default
    This helps maximize overall hardware utilization while minimizing the impact on other running applications.

[vlakoff]

Honestly, I’m not convinced that adding a dedicated UI for per‑disk thread tuning would be worth the complexity. It would be a lot of work to build and maintain, and only a very small group of power users would ever use it. Most people just want the program to run efficiently without needing to tweak anything.

As for the concurrency numbers I mentioned earlier:

• HDD: process 1 file at a time
• SATA SSD: process 2 files in parallel
• NVMe SSD: process up to 4 files in parallel
  This avoids overloading slower devices while still taking advantage of faster ones.
• Parallel reads from different physical drives should of course be allowed.
  (Handling shared bandwidth on multi‑device controllers is a deeper topic and probably out of scope.)

These figures I posted are indeed way too conservative. HDDs really should stay at 1, but SATA SSDs can probably handle more than 2, and NVMe SSDs are almost limitless — the CPU thread pool becomes the bottleneck long before the drive does, even on something like an AMD Threadripper.

As the values are straightforward for HDDs (1 file) and NVMe SSDs (effectively unlimited files), the only remaining uncertainty is the optimal concurrency for SATA SSDs: 4 files? 8 files? 16 files? And adding a dedicated UI solely for this case doesn’t seem justified.

[vlakoff]

If we had to pick a single “universal” sweet spot for SATA SSDs, I’d say 8 parallel files is probably the best overall choice.

I think that with detection of HDD / SATA SSD / NVMe SSD, with parallel file processing per‑device type, plus parallel reads across different physical drives, we would already cover the vast majority of real‑world performance needs.

But as OV2 mentioned, implementing this would require a significant amount of internal restructuring, and that part is not necessarily simple.