Add a library to sync conda environments #628
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D78424481 |
andrewjcg
force-pushed
the
export-D78424481
branch
from
0980386 to
41b6a64
Compare
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 23, 2025
Summary:
This adds a crate to do rsync-style syncing of a source conda env to a
destination, as per:
```
$ rsync --archive --update --no-dirtimes --delete --delete-excluded ...
```
As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the
differences in prefixes between source and destination conda envs.
The notable differences with rsync are:
1) In current flows, when users "activate" their conda env locally, prefixes in
the env will be quickly re-written, changing mtimes to be ahead of the
destination (e.g. on MAST, the conda env is unpacked into it's expected
prefix location). This crate uses the `pack-meta/history.jsonl` file to detect
prefix update windows and use that to filter out these sprurious mtime updates.
2) Files that are copied perform in-place prefix replacement (TODO: not finished
yet).
Differential Revision: D78424481
andrewjcg
force-pushed
the
export-D78424481
branch
from
41b6a64 to
c0b3a07
Compare
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 23, 2025
Summary:
This adds a crate to do rsync-style syncing of a source conda env to a
destination, as per:
```
$ rsync --archive --update --no-dirtimes --delete --delete-excluded ...
```
As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the
differences in prefixes between source and destination conda envs.
The notable differences with rsync are:
1) In current flows, when users "activate" their conda env locally, prefixes in
the env will be quickly re-written, changing mtimes to be ahead of the
destination (e.g. on MAST, the conda env is unpacked into it's expected
prefix location). This crate uses the `pack-meta/history.jsonl` file to detect
prefix update windows and use that to filter out these sprurious mtime updates.
2) Files that are copied perform in-place prefix replacement (TODO: not finished
yet).
Differential Revision: D78424481
andrewjcg
force-pushed
the
export-D78424481
branch
from
c0b3a07 to
45cb64d
Compare
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 23, 2025
Summary:
This adds a crate to do rsync-style syncing of a source conda env to a
destination, as per:
```
$ rsync --archive --update --no-dirtimes --delete --delete-excluded ...
```
As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the
differences in prefixes between source and destination conda envs.
The notable differences with rsync are:
1) In current flows, when users "activate" their conda env locally, prefixes in
the env will be quickly re-written, changing mtimes to be ahead of the
destination (e.g. on MAST, the conda env is unpacked into it's expected
prefix location). This crate uses the `pack-meta/history.jsonl` file to detect
prefix update windows and use that to filter out these sprurious mtime updates.
2) Files that are copied perform in-place prefix replacement (TODO: not finished
yet).
Differential Revision: D78424481
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 23, 2025
Summary:
This adds a crate to do rsync-style syncing of a source conda env to a
destination, as per:
```
$ rsync --archive --update --no-dirtimes --delete --delete-excluded ...
```
As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the
differences in prefixes between source and destination conda envs.
The notable differences with rsync are:
1) In current flows, when users "activate" their conda env locally, prefixes in
the env will be quickly re-written, changing mtimes to be ahead of the
destination (e.g. on MAST, the conda env is unpacked into it's expected
prefix location). This crate uses the `pack-meta/history.jsonl` file to detect
prefix update windows and use that to filter out these sprurious mtime updates.
2) Files that are copied perform in-place prefix replacement (TODO: not finished
yet).
Differential Revision: D78424481
andrewjcg
force-pushed
the
export-D78424481
branch
from
45cb64d to
7dfd9cd
Compare
|
This pull request was exported from Phabricator. Differential Revision: D78424481 |
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 23, 2025
Summary: Pull Request resolved: pytorch-labs#628 This adds a crate to do rsync-style syncing of a source conda env to a destination, as per: ``` $ rsync --archive --update --no-dirtimes --delete --delete-excluded ... ``` As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the differences in prefixes between source and destination conda envs. The notable differences with rsync are: 1) In current flows, when users "activate" their conda env locally, prefixes in the env will be quickly re-written, changing mtimes to be ahead of the destination (e.g. on MAST, the conda env is unpacked into it's expected prefix location). This crate uses the `pack-meta/history.jsonl` file to detect prefix update windows and use that to filter out these sprurious mtime updates. 2) Files that are copied perform in-place prefix replacement (TODO: not finished yet). Differential Revision: D78424481
andrewjcg
force-pushed
the
export-D78424481
branch
from
7dfd9cd to
d4f8c86
Compare
|
This pull request was exported from Phabricator. Differential Revision: D78424481 |
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 23, 2025
Summary: Pull Request resolved: pytorch-labs#628 This adds a crate to do rsync-style syncing of a source conda env to a destination, as per: ``` $ rsync --archive --update --no-dirtimes --delete --delete-excluded ... ``` As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the differences in prefixes between source and destination conda envs. The notable differences with rsync are: 1) In current flows, when users "activate" their conda env locally, prefixes in the env will be quickly re-written, changing mtimes to be ahead of the destination (e.g. on MAST, the conda env is unpacked into it's expected prefix location). This crate uses the `pack-meta/history.jsonl` file to detect prefix update windows and use that to filter out these sprurious mtime updates. 2) Files that are copied perform in-place prefix replacement (TODO: not finished yet). Differential Revision: D78424481
andrewjcg
force-pushed
the
export-D78424481
branch
from
d4f8c86 to
12792b8
Compare
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 23, 2025
Summary: Pull Request resolved: pytorch-labs#628 This adds a crate to do rsync-style syncing of a source conda env to a destination, as per: ``` $ rsync --archive --update --no-dirtimes --delete --delete-excluded ... ``` As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the differences in prefixes between source and destination conda envs. The notable differences with rsync are: 1) In current flows, when users "activate" their conda env locally, prefixes in the env will be quickly re-written, changing mtimes to be ahead of the destination (e.g. on MAST, the conda env is unpacked into it's expected prefix location). This crate uses the `pack-meta/history.jsonl` file to detect prefix update windows and use that to filter out these sprurious mtime updates. 2) Files that are copied perform in-place prefix replacement (TODO: not finished yet). Differential Revision: D78424481
andrewjcg
force-pushed
the
export-D78424481
branch
from
12792b8 to
e3e0aea
Compare
|
This pull request was exported from Phabricator. Differential Revision: D78424481 |
1 similar comment
|
This pull request was exported from Phabricator. Differential Revision: D78424481 |
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 23, 2025
Summary: Pull Request resolved: pytorch-labs#628 This adds a crate to do rsync-style syncing of a source conda env to a destination, as per: ``` $ rsync --archive --update --no-dirtimes --delete --delete-excluded ... ``` As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the differences in prefixes between source and destination conda envs. The notable differences with rsync are: 1) In current flows, when users "activate" their conda env locally, prefixes in the env will be quickly re-written, changing mtimes to be ahead of the destination (e.g. on MAST, the conda env is unpacked into it's expected prefix location). This crate uses the `pack-meta/history.jsonl` file to detect prefix update windows and use that to filter out these sprurious mtime updates. 2) Files that are copied perform in-place prefix replacement (TODO: not finished yet). Differential Revision: D78424481
andrewjcg
force-pushed
the
export-D78424481
branch
2 times, most recently
from
93d5b7c to
6636535
Compare
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 25, 2025
Summary:
This adds a crate to do rsync-style syncing of a source conda env to a
destination, as per:
```
$ rsync --archive --update --no-dirtimes --delete --delete-excluded ...
```
As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the
differences in prefixes between source and destination conda envs.
The notable differences with rsync are:
1) In current flows, when users "activate" their conda env locally, prefixes in
the env will be quickly re-written, changing mtimes to be ahead of the
destination (e.g. on MAST, the conda env is unpacked into it's expected
prefix location). This crate uses the `pack-meta/history.jsonl` file to detect
prefix update windows and use that to filter out these sprurious mtime updates.
2) Files that are copied perform in-place prefix replacement (TODO: not finished
yet).
Differential Revision: D78424481
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 25, 2025
Summary:
This adds a crate to do rsync-style syncing of a source conda env to a
destination, as per:
```
$ rsync --archive --update --no-dirtimes --delete --delete-excluded ...
```
As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the
differences in prefixes between source and destination conda envs.
The notable differences with rsync are:
1) In current flows, when users "activate" their conda env locally, prefixes in
the env will be quickly re-written, changing mtimes to be ahead of the
destination (e.g. on MAST, the conda env is unpacked into it's expected
prefix location). This crate uses the `pack-meta/history.jsonl` file to detect
prefix update windows and use that to filter out these sprurious mtime updates.
2) Files that are copied perform in-place prefix replacement (TODO: not finished
yet).
Differential Revision: D78424481
andrewjcg
force-pushed
the
export-D78424481
branch
from
6636535 to
aedee26
Compare
|
This pull request was exported from Phabricator. Differential Revision: D78424481 |
andrewjcg
force-pushed
the
export-D78424481
branch
from
aedee26 to
23ed79e
Compare
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 25, 2025
Summary: Pull Request resolved: pytorch-labs#628 This adds a crate to do rsync-style syncing of a source conda env to a destination, as per: ``` $ rsync --archive --update --no-dirtimes --delete --delete-excluded ... ``` As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the differences in prefixes between source and destination conda envs. The notable differences with rsync are: 1) In current flows, when users "activate" their conda env locally, prefixes in the env will be quickly re-written, changing mtimes to be ahead of the destination (e.g. on MAST, the conda env is unpacked into it's expected prefix location). This crate uses the `pack-meta/history.jsonl` file to detect prefix update windows and use that to filter out these sprurious mtime updates. 2) Files that are copied perform in-place prefix replacement (TODO: not finished yet). Differential Revision: D78424481
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 25, 2025
Summary:
This adds a crate to do rsync-style syncing of a source conda env to a
destination, as per:
```
$ rsync --archive --update --no-dirtimes --delete --delete-excluded ...
```
As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the
differences in prefixes between source and destination conda envs.
The notable differences with rsync are:
1) In current flows, when users "activate" their conda env locally, prefixes in
the env will be quickly re-written, changing mtimes to be ahead of the
destination (e.g. on MAST, the conda env is unpacked into it's expected
prefix location). This crate uses the `pack-meta/history.jsonl` file to detect
prefix update windows and use that to filter out these sprurious mtime updates.
2) Files that are copied perform in-place prefix replacement (TODO: not finished
yet).
Differential Revision: D78424481
andrewjcg
force-pushed
the
export-D78424481
branch
from
23ed79e to
8662143
Compare
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 25, 2025
Summary:
This adds a crate to do rsync-style syncing of a source conda env to a
destination, as per:
```
$ rsync --archive --update --no-dirtimes --delete --delete-excluded ...
```
As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the
differences in prefixes between source and destination conda envs.
The notable differences with rsync are:
1) In current flows, when users "activate" their conda env locally, prefixes in
the env will be quickly re-written, changing mtimes to be ahead of the
destination (e.g. on MAST, the conda env is unpacked into it's expected
prefix location). This crate uses the `pack-meta/history.jsonl` file to detect
prefix update windows and use that to filter out these sprurious mtime updates.
2) Files that are copied perform in-place prefix replacement (TODO: not finished
yet).
Differential Revision: D78424481
andrewjcg
force-pushed
the
export-D78424481
branch
from
8662143 to
c3bcff4
Compare
|
This pull request was exported from Phabricator. Differential Revision: D78424481 |
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 25, 2025
Summary: Pull Request resolved: pytorch-labs#628 This adds a crate to do rsync-style syncing of a source conda env to a destination, as per: ``` $ rsync --archive --update --no-dirtimes --delete --delete-excluded ... ``` As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the differences in prefixes between source and destination conda envs. The notable differences with rsync are: 1) In current flows, when users "activate" their conda env locally, prefixes in the env will be quickly re-written, changing mtimes to be ahead of the destination (e.g. on MAST, the conda env is unpacked into it's expected prefix location). This crate uses the `pack-meta/history.jsonl` file to detect prefix update windows and use that to filter out these sprurious mtime updates. 2) Files that are copied perform in-place prefix replacement (TODO: not finished yet). Differential Revision: D78424481
andrewjcg
force-pushed
the
export-D78424481
branch
from
c3bcff4 to
7db68d7
Compare
|
This pull request was exported from Phabricator. Differential Revision: D78424481 |
andrewjcg
added a commit
to andrewjcg/monarch
that referenced
this pull request
Jul 25, 2025
Summary: Pull Request resolved: pytorch-labs#628 This adds a crate to do rsync-style syncing of a source conda env to a destination, as per: ``` $ rsync --archive --update --no-dirtimes --delete --delete-excluded ... ``` As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the differences in prefixes between source and destination conda envs. The notable differences with rsync are: 1) In current flows, when users "activate" their conda env locally, prefixes in the env will be quickly re-written, changing mtimes to be ahead of the destination (e.g. on MAST, the conda env is unpacked into it's expected prefix location). This crate uses the `pack-meta/history.jsonl` file to detect prefix update windows and use that to filter out these sprurious mtime updates. 2) Files that are copied perform in-place prefix replacement (TODO: not finished yet). Differential Revision: D78424481
andrewjcg
force-pushed
the
export-D78424481
branch
from
7db68d7 to
6c831cb
Compare
Summary: Pull Request resolved: pytorch-labs#628 This adds a crate to do rsync-style syncing of a source conda env to a destination, as per: ``` $ rsync --archive --update --no-dirtimes --delete --delete-excluded ... ``` As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the differences in prefixes between source and destination conda envs. The notable differences with rsync are: 1) In current flows, when users "activate" their conda env locally, prefixes in the env will be quickly re-written, changing mtimes to be ahead of the destination (e.g. on MAST, the conda env is unpacked into it's expected prefix location). This crate uses the `pack-meta/history.jsonl` file to detect prefix update windows and use that to filter out these sprurious mtime updates. 2) Files that are copied perform in-place prefix replacement (TODO: not finished yet). Differential Revision: D78424481
|
This pull request was exported from Phabricator. Differential Revision: D78424481 |
andrewjcg
force-pushed
the
export-D78424481
branch
from
6c831cb to
c342ec6
Compare
highker
pushed a commit
to highker/monarch
that referenced
this pull request
Aug 4, 2025
Summary: Pull Request resolved: pytorch-labs#628 This adds a crate to do rsync-style syncing of a source conda env to a destination, as per: ``` $ rsync --archive --update --no-dirtimes --delete --delete-excluded ... ``` As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the differences in prefixes between source and destination conda envs. The notable differences with rsync are: 1) In current flows, when users "activate" their conda env locally, prefixes in the env will be quickly re-written, changing mtimes to be ahead of the destination (e.g. on MAST, the conda env is unpacked into it's expected prefix location). This crate uses the `pack-meta/history.jsonl` file to detect prefix update windows and use that to filter out these sprurious mtime updates. 2) Files that are copied perform in-place prefix replacement (TODO: not finished yet). Differential Revision: D78424481
highker
pushed a commit
to highker/monarch
that referenced
this pull request
Aug 6, 2025
Summary: Pull Request resolved: pytorch-labs#628 This adds a crate to do rsync-style syncing of a source conda env to a destination, as per: ``` $ rsync --archive --update --no-dirtimes --delete --delete-excluded ... ``` As `rsync` (and this crate) uses mtimes for syncing, it'll work even with the differences in prefixes between source and destination conda envs. The notable differences with rsync are: 1) In current flows, when users "activate" their conda env locally, prefixes in the env will be quickly re-written, changing mtimes to be ahead of the destination (e.g. on MAST, the conda env is unpacked into it's expected prefix location). This crate uses the `pack-meta/history.jsonl` file to detect prefix update windows and use that to filter out these sprurious mtime updates. 2) Files that are copied perform in-place prefix replacement (TODO: not finished yet). Differential Revision: D78424481
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This adds a crate to do rsync-style syncing of a source conda env to a
destination, as per:
As
rsync(and this crate) uses mtimes for syncing, it'll work even with thedifferences in prefixes between source and destination conda envs.
The notable differences with rsync are:
the env will be quickly re-written, changing mtimes to be ahead of the
destination (e.g. on MAST, the conda env is unpacked into it's expected
prefix location). This crate uses the
pack-meta/history.jsonlfile to detectprefix update windows and use that to filter out these sprurious mtime updates.
yet).
Differential Revision: D78424481