Feature: JW map Rayon Multithreading

[davidomanovic]

Description

Multithreading in Rayon for faster JW map by parallelizing processes in Rust.

Results

On N2 in cc-pvdz:

1 thread: 14.098538247868419 s
4 threads: 5.772801402956247 s
16 threads: 3.734854843467474 s
48 threads: 3.3065379448235035 s
64 threads: 3.1618218813091516 s

Closes issue #496

[kevinsung]

Just to be sure, this is more performant than your original Python multiprocessing implementation, right?

[davidomanovic]

Just to be sure, this is more performant than your original Python multiprocessing implementation, right?

Yes, even the non-parallelized Rust implementation was faster than the original Python multiprocessing implementation.

[kevinsung]

On N2 in cc-pvdz, 48 threads:

Non-parallel: 13 seconds
Parallel: 3 seconds

Hmm so we only get a ~4x speedup with 48 threads. Do we get the same performance with just 4 threads? Does the performance improvement level off after 4 or so threads?

[davidomanovic]

On N2 in cc-pvdz, 48 threads:

Non-parallel: 13 seconds
Parallel: 3 seconds

Hmm so we only get a ~4x speedup with 48 threads. Do we get the same performance with just 4 threads? Does the performance improvement level off after 4 or so threads?

I tried with different threads:
1 thread: 14.098538247868419 s
4 threads: 5.772801402956247 s
16 threads: 3.734854843467474 s
48 threads: 3.3065379448235035 s
64 threads: 3.1618218813091516 s

I assume this is due to memory bandwidth and serial work that every thread must do, so throwing more threads at it dont help since it anyways just plateus. It actually seems to agree well with Amdahl's law. There is room for tiny improvements (maybe like 20%) with faster hashers, but we are pretty much at the limit here 😄

[kevinsung]

Thank you!

[davidomanovic]

Thank you!

Enjoy! 😄