fix allreduce usage (meta-pytorch#279)

Summary:
Pull Request resolved: meta-pytorch#279

pass full allreduce options to the pg allreduce to avoid the watchdog abort from getting triggered

Reviewed By: d4l3k

Differential Revision: D84101243

fbshipit-source-id: b2ecf071d9943d75d3f21091805b92948d6e6617

Author: tushar00jain

torchft/manager.py

@@ -423,7 +423,9 @@ def allreduce(
                     torch.accelerator.current_stream(),
                 )
             else:
-                work = self._pg.allreduce([tensor], reduce_op)
+                opts = AllreduceOptions()
+                opts.reduceOp = reduce_op
+                work = self._pg.allreduce([tensor], opts)
 
             # schedule grad normalization as a continuation
             # on the Future