I built this while dealing with large SMB shares over WiFi, where traditional
directory scanning tools were either too slow or caused unnecessary network load.
The main idea is a layered scanning strategy:
- Minimize SMB round-trips
- Use Rayon for controlled parallelism
- Avoid spawning excessive IO-bound tasks
- Keep memory usage predictable
This is not meant to be a fancy UI tool, but a fast and deterministic building
block that can be embedded into other systems.
The project is open-source (Apache 2.0).
I'm especially interested in feedback on:
- The concurrency model
- SMB traversal strategy
- Any edge cases you've seen in similar systems
Thanks for taking a look.
holg•1mo ago
Hi thx for sharing, in general
about concurrency Rayon excels on CPU-heavy tasks, but this is more IO
for traversal avoid any extra round-trips, simple the metadata already given
edge-cases would be infinite loops on symlinks, the Windows 255 char limits
and on WIFI packet loss due to too many reqs
dohuyhoangvn93•1mo ago
Thank you for your feedback.
In the initial CLI test, I tried using Tokio and found that performance didn't improve and the earliest result return speed was quite poor. I skipped the root cause analysis and kept the original rayon design.
I will try redoing the tests and taking more detailed measurements for evaluation.
dohuyhoangvn93•1mo ago
I'm the author of DeepSearch.
I built this while dealing with large SMB shares over WiFi, where traditional directory scanning tools were either too slow or caused unnecessary network load.
The main idea is a layered scanning strategy: - Minimize SMB round-trips - Use Rayon for controlled parallelism - Avoid spawning excessive IO-bound tasks - Keep memory usage predictable
This is not meant to be a fancy UI tool, but a fast and deterministic building block that can be embedded into other systems.
The project is open-source (Apache 2.0). I'm especially interested in feedback on: - The concurrency model - SMB traversal strategy - Any edge cases you've seen in similar systems
Thanks for taking a look.
holg•1mo ago
about concurrency Rayon excels on CPU-heavy tasks, but this is more IO
for traversal avoid any extra round-trips, simple the metadata already given
edge-cases would be infinite loops on symlinks, the Windows 255 char limits and on WIFI packet loss due to too many reqs
dohuyhoangvn93•1mo ago
I will try redoing the tests and taking more detailed measurements for evaluation.
Thank you again.