Rank local checkpointing in DCP without collectives (#991)

Summary:

X-link: pytorch/pytorch#147758


Context:
DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon.

Reviewed By: meetv18

Differential Revision: D70112642

Author: saumishr

tests/framework/callbacks/test_dcp_saver.py

@@ -1012,13 +1012,17 @@ class DummyStorageWriter(FileSystemWriter):
     def __init__(self, path: str) -> None:
         super().__init__(path)
 
-    def set_up_storage_writer(self, is_coordinator: bool) -> None:
+    def set_up_storage_writer(
+        self, is_coordinator: bool, *args: Any, **kwargs: Any
+    ) -> None:
         pass
 
 
 class DummyStorageReader(FileSystemReader):
     def __init__(self, path: str) -> None:
         super().__init__(path)
 
-    def set_up_storage_writer(self, is_coordinator: bool) -> None:
+    def set_up_storage_writer(
+        self, is_coordinator: bool, *args: Any, **kwargs: Any
+    ) -> None:
         pass