SSHExecutor fails to create a connection for sublattices and large workflows

[venkatBala]

When multiple workflows have been dispatched, the SSHExecutor fails to create SSH connections. This leaves the node/workflow hanging without ever completing.

[venkatBala]

Can the SSHExecutor made to use an already open ssh connection instead of create a new one each time?

[cjao]

To elaborate, large workflows frequently fail midway with ERROR -- Error reading SSH protocol banner. The current SSH plugin is a working proof of concept but not really suitable for large workflows. The SSH server can easily get overloaded, and at a minimum the plugin should implement some logic to reattempt execution in this scenario.

[cjao]

We should probably stop holding the connection open throughout the entire task but rather poll periodically like we do with Slurm.