From f9ffd46744ed51ffa4969cf92ffd6f2b0c068c31 Mon Sep 17 00:00:00 2001 From: Devon Govett Date: Tue, 11 Jul 2017 09:09:11 -0700 Subject: [PATCH 1/5] Update 0000-yarn-knit.md --- accepted/0000-yarn-knit.md | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/accepted/0000-yarn-knit.md b/accepted/0000-yarn-knit.md index 3d77cef..10b3415 100644 --- a/accepted/0000-yarn-knit.md +++ b/accepted/0000-yarn-knit.md @@ -8,7 +8,7 @@ This is a proposal to improve upon `yarn link` so that developers can more accur # Motivation -`yarn link` (and `npm link`) before it have several problems when working on code bases of non-trivial sizes, especially with multiple apps. The current `link` command doesn't isolate `node_modules` between apps (especially problematic with the advent of Electron), it doesn't allow for working on multiple versions of a library, and it produces a `node_modules` hierarchy that is not faithful to the one produced after the library is published. +`yarn link` (and `npm link` before it) have several problems when working on code bases of non-trivial sizes, especially with multiple apps. The current `link` command doesn't isolate `node_modules` between apps (especially problematic with the advent of Electron), it doesn't allow for working on multiple versions of a library, and it produces a `node_modules` hierarchy that is not faithful to the one produced after the library is published. # Detailed design @@ -52,6 +52,24 @@ This behaves like `yarn add dep` except that it looks at the versions of `dep` t `yarn knit dep` then runs most of the same installation steps that `yarn add dep` would. It creates `app/node_modules/dep` and creates symlinks for each of the symlinks under `~/.yarn-knit/dep-X.Y.Z`. Then it installs the dependencies of `dep` as usual by fetching them from npm. Finally it runs postinstall scripts. +### Running "yarn install --knit" inside of `app` + +This behaves like normal `yarn install` but automatically links any dependencies that have been set up locally with `yarn knit`. This is very useful when you have an application with many locally developed dependencies so that you don't have to manually run `yarn link dep` for each one inside your app. + +# Comparison with other Yarn features + +There are already several ways of linking dependencies in Yarn. This section is a comparison of them, and explains why `yarn knit` is still valuable to add. + +* `yarn link` is similar to `yarn knit` but links directories instead of files. This means that any `node_modules` inside linked dependencies are also included, breaking the flattened and deduped tree that yarn normally provides. Many other differences have already been discussed in this RFC. + +* `link://` dependencies are similar to `yarn link` but actually closer to what `yarn knit` would provide. `link://` dependencies also cause the linked module's dependencies to be installed locally, so as long as there is no `node_modules` folder inside the linked dependency, it would work identically to `yarn knit dep`. The existing implementation of `link://` dependencies could be used to implement `yarn knit dep`. + +* `file://` dependencies cause files to be copied instead of linked. This is not as useful because it means you must re-install every time a change is made to the dependency. + +* `yarn pack` + `yarn install dep.tgz` is similar to `file://` dependencies. The pack + install process must be re-run every time a change is made. It does correctly dedupe dependencies, however, as `node_modules` are excluded by `yarn pack`. + +* [Workspaces](https://github.com/yarnpkg/rfcs/pull/66) are similar but solve a different problem. Where workspaces are great for a tree of related modules (e.g. in a monorepo), `yarn knit` is for linking together modules in separate trees, e.g. things that might be shared between multiple workspaces. + # How We Teach This This proposal is mostly additive and affects only how people work on libraries that they are using in their apps. We would want to document the `knit` command in the "CLI Commands" section of the docs and perhaps add a new section to "The Yarn Workflow". @@ -62,7 +80,7 @@ This proposal is mostly additive and affects only how people work on libraries t One issue with this proposal is that it's not clear what to put in the lockfile after running `yarn link dep` since we don't have an npm URL for the dep yet -- it hasn't been published to npm. -Another issue is that if you change package.json in `dep`, namely changing a dependency or modifying the `files` entry, you have to run `cd dep; yarn knit; cd app; yarn knit dep`. +Another issue is that if you change package.json in `dep`, namely changing a dependency or modifying the `files` entry, you have to run `cd dep; yarn knit; cd app; yarn knit dep`. Same if you add a file at the top level of the dependency since each file is linked individually. Also, if you update the code in dep and bump its version, say from 1.0.0 to 1.1.0, the symlinks in ~/.yarn-knit/dep-1.0.0 will still point to the code in your working directory, which now contains 1.1.0 code. @@ -70,8 +88,6 @@ The symlinks might break but I think that's mostly OK since at that point you're If you want to truly pin the versions of knitted packages then you'd need to have a different working directory for each version. (Git worktrees are great for this use case actually. Worktrees let you check out a repo once and then magically create semi-clones of it in separate directories, with the constraint that the worktrees need to be on different branches, which is totally OK in this scenario. The worktrees all share the same Git repo though, so if you commit in one worktree you can cherry pick that commit within another worktree.) -# Alternatives - -Another similar approach is to run `yarn pack`, which creates a tarball of the library's contents as if it were downloaded from npm, and install the tarball in the app used to test the library. This has the benefits of reducing divergence between development and production -- `library/node_modules` is not shared across apps, the developer can work on multiple versions of the library, and the `node_modules` hierarchy is faithfully represented. The downside is that everytime the developer edits a file, they need to re-pack and reinstall the library. +Finally, another issue is with the way the node require resolution algorithm works. Dependencies of symlinked modules are resolved relative to the realpath, not the symlink. This means that you'll still get duplicates if both modules depend on a third dependency, or errors if that dependency is not installed in either place. This is solved by the recently added runtime option for node `--preserve-symlinks`, which skips getting the realpath when resolving modules. Something similar would need to be added to browserify/webpack to solve this there as well. I recently opened a [PR for browserify](https://github.com/substack/node-browserify/pull/1742) to support the same option. # Unresolved questions From 771f0d75f842265ccc11f076102c1548983fda04 Mon Sep 17 00:00:00 2001 From: Devon Govett Date: Tue, 11 Jul 2017 09:44:19 -0700 Subject: [PATCH 2/5] Update 0000-yarn-knit.md --- accepted/0000-yarn-knit.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/accepted/0000-yarn-knit.md b/accepted/0000-yarn-knit.md index 10b3415..b2e21a3 100644 --- a/accepted/0000-yarn-knit.md +++ b/accepted/0000-yarn-knit.md @@ -60,7 +60,7 @@ This behaves like normal `yarn install` but automatically links any dependencies There are already several ways of linking dependencies in Yarn. This section is a comparison of them, and explains why `yarn knit` is still valuable to add. -* `yarn link` is similar to `yarn knit` but links directories instead of files. This means that any `node_modules` inside linked dependencies are also included, breaking the flattened and deduped tree that yarn normally provides. Many other differences have already been discussed in this RFC. +* `yarn link` is similar to `yarn knit` but links directories instead of files. This means that any `node_modules` inside linked dependencies are also included, breaking the flattened and deduped tree that yarn normally provides. Many other differences have already been discussed in this RFC. Ideally, the behavior of `yarn link` would be what is proposed in this RFC, but for backwards compatibility and to match the behavior of `npm link`, `yarn link` would stay around as is. * `link://` dependencies are similar to `yarn link` but actually closer to what `yarn knit` would provide. `link://` dependencies also cause the linked module's dependencies to be installed locally, so as long as there is no `node_modules` folder inside the linked dependency, it would work identically to `yarn knit dep`. The existing implementation of `link://` dependencies could be used to implement `yarn knit dep`. From 50712e27d2d242bf466224186fcbb87b9bbab0ba Mon Sep 17 00:00:00 2001 From: Devon Govett Date: Tue, 11 Jul 2017 11:39:54 -0700 Subject: [PATCH 3/5] Update 0000-yarn-knit.md --- accepted/0000-yarn-knit.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/accepted/0000-yarn-knit.md b/accepted/0000-yarn-knit.md index b2e21a3..e3eedbe 100644 --- a/accepted/0000-yarn-knit.md +++ b/accepted/0000-yarn-knit.md @@ -68,7 +68,7 @@ There are already several ways of linking dependencies in Yarn. This section is * `yarn pack` + `yarn install dep.tgz` is similar to `file://` dependencies. The pack + install process must be re-run every time a change is made. It does correctly dedupe dependencies, however, as `node_modules` are excluded by `yarn pack`. -* [Workspaces](https://github.com/yarnpkg/rfcs/pull/66) are similar but solve a different problem. Where workspaces are great for a tree of related modules (e.g. in a monorepo), `yarn knit` is for linking together modules in separate trees, e.g. things that might be shared between multiple workspaces. +* [Workspaces](https://github.com/yarnpkg/rfcs/pull/66) are similar but solve a different problem. Where workspaces are great for a tree of related modules (e.g. a monorepo), `yarn knit` is for linking together modules in separate trees, e.g. things that might be shared between multiple workspaces. # How We Teach This @@ -78,13 +78,13 @@ This proposal is mostly additive and affects only how people work on libraries t # Drawbacks -One issue with this proposal is that it's not clear what to put in the lockfile after running `yarn link dep` since we don't have an npm URL for the dep yet -- it hasn't been published to npm. +One issue with this proposal is that it's not clear what to put in the lockfile after running `yarn knit dep` since we don't have an npm URL for the dep yet -- it hasn't been published to npm. -Another issue is that if you change package.json in `dep`, namely changing a dependency or modifying the `files` entry, you have to run `cd dep; yarn knit; cd app; yarn knit dep`. Same if you add a file at the top level of the dependency since each file is linked individually. +Another issue is that if you change package.json in `dep`, namely changing a dependency or modifying the `files` entry, you have to run `cd dep; yarn knit; cd app; yarn knit dep`. Same if you add a file at the top level of the dependency since each file is linked individually. This isn't so bad as those changes are probably more rare than saving changes to existing files. Also, if you update the code in dep and bump its version, say from 1.0.0 to 1.1.0, the symlinks in ~/.yarn-knit/dep-1.0.0 will still point to the code in your working directory, which now contains 1.1.0 code. -The symlinks might break but I think that's mostly OK since at that point you're done working on dep and have published it to npm and it's easy to go run yarn add dep in app and not use the symlinks anymore. +The symlinks might break but I think that's mostly OK since at that point you're done working on dep and have published it to npm and it's easy to go run `yarn add dep` in app and not use the symlinks anymore. If you want to truly pin the versions of knitted packages then you'd need to have a different working directory for each version. (Git worktrees are great for this use case actually. Worktrees let you check out a repo once and then magically create semi-clones of it in separate directories, with the constraint that the worktrees need to be on different branches, which is totally OK in this scenario. The worktrees all share the same Git repo though, so if you commit in one worktree you can cherry pick that commit within another worktree.) From fc9f91b02c060e778723c74279ebdb8fdd72a5c1 Mon Sep 17 00:00:00 2001 From: Devon Govett Date: Thu, 3 Aug 2017 21:06:36 -0700 Subject: [PATCH 4/5] Update 0000-yarn-knit.md --- accepted/0000-yarn-knit.md | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/accepted/0000-yarn-knit.md b/accepted/0000-yarn-knit.md index e3eedbe..159008e 100644 --- a/accepted/0000-yarn-knit.md +++ b/accepted/0000-yarn-knit.md @@ -36,57 +36,57 @@ This is a problem when you are developing & testing `dep-1` with `old-app` and ` Currently `yarn link` symlinks the entire package directory, which brings along its `node_modules` subdirectory with it. With dependency deduping and flattening, bringing in `dep/node_modules` wholesale usually produces a different `node_modules` hierarchy than running `yarn install` in `app` and installing everything from npm. This isn't a problem most of the time but it does go against Yarn's spirit of consistency and the lockfile. -## A practical proposal -- knitting +## A practical proposal -This is a proposal that solves all of the problems above and isn't too hard to implement or understand. I'm going to call it `yarn knit` to distinguish it from `yarn link`. Conceptually, we find all the files we'd normally publish to npm, pack them up using symlinks instead of copies of the files, publish the pack to a local registry (just a directory), and then when installing we look up packages in the local registry directory instead of npm. +This is a proposal that solves all of the problems above and isn't too hard to implement or understand. Conceptually, we find all the files we'd normally publish to npm, pack them up using symlinks instead of copies of the files, publish the pack to a local registry (just a directory), and then when installing we look up packages in the local registry directory instead of npm. -### Running "yarn knit" inside of `dep` +### Running "yarn link --pack" inside of `dep` -This is the step that simulates publishing `dep`. Running `yarn knit` in `dep` finds all the files that "yarn publish" would pack up and upload to npm. Crucially, this excludes `node_modules`, and would follow the same algorithm as "yarn publish" such as reading package.json's `files` field. +This is the step that simulates publishing `dep`. Running `yarn link --pack` in `dep` finds all the files that "yarn publish" would pack up and upload to npm. Crucially, this excludes `node_modules`, and would follow the same algorithm as "yarn publish" such as reading package.json's `files` field. -Then it simulates publishing `dep`: it creates a directory named `dep-X.Y.Z` (where `X.Y.Z` is the version of `dep` in its package.json) inside of a global directory like `~/.yarn-knit`. A symlink is created for each file or directory that `yarn publish` would normally have packed up. This step shares some conceptual similarities with publishing to a registry, except it uses symlinks and it's local on your computer. +Then it simulates publishing `dep`: it creates a directory named `dep-X.Y.Z` (where `X.Y.Z` is the version of `dep` in its package.json) inside of a global directory like `~/.yarn-link`. A symlink is created for each file or directory that `yarn publish` would normally have packed up. This step shares some conceptual similarities with publishing to a registry, except it uses symlinks and it's local on your computer. -### Running "yarn knit dep" inside of `app` +### Running "yarn add dep --link" inside of `app` -This behaves like `yarn add dep` except that it looks at the versions of `dep` that are in the global `~/.yarn-knit` folder and takes the latest one. (You also could run "yarn link dep@X.Y.Z" if you wanted a more specific version, like "yarn add".) +This behaves like normal `yarn add dep` except that it looks at the versions of `dep` that are in the global `~/.yarn-link` folder and takes the latest one rather than installing from a normal registry. (You also could run "yarn add dep@X.Y.Z --link" if you wanted a more specific version, as usual.) -`yarn knit dep` then runs most of the same installation steps that `yarn add dep` would. It creates `app/node_modules/dep` and creates symlinks for each of the symlinks under `~/.yarn-knit/dep-X.Y.Z`. Then it installs the dependencies of `dep` as usual by fetching them from npm. Finally it runs postinstall scripts. +It then runs most of the same installation steps that `yarn add dep` would. It symlinks `app/node_modules/dep` to `~/.yarn-link/dep-X.Y.Z` as `yarn link dep` does. Then it installs the dependencies of `dep` as usual by fetching them from npm. Finally it runs postinstall scripts. -### Running "yarn install --knit" inside of `app` +### Running "yarn install --link" inside of `app` -This behaves like normal `yarn install` but automatically links any dependencies that have been set up locally with `yarn knit`. This is very useful when you have an application with many locally developed dependencies so that you don't have to manually run `yarn link dep` for each one inside your app. +This behaves like normal `yarn install` but automatically links any dependencies that have been set up locally with `yarn link`. This is very useful when you have an application with many locally developed dependencies so that you don't have to manually run `yarn link dep` for each one inside your app. # Comparison with other Yarn features -There are already several ways of linking dependencies in Yarn. This section is a comparison of them, and explains why `yarn knit` is still valuable to add. +There are already several ways of linking dependencies in Yarn. This section is a comparison of them, and explains why these new options are still valuable to add. -* `yarn link` is similar to `yarn knit` but links directories instead of files. This means that any `node_modules` inside linked dependencies are also included, breaking the flattened and deduped tree that yarn normally provides. Many other differences have already been discussed in this RFC. Ideally, the behavior of `yarn link` would be what is proposed in this RFC, but for backwards compatibility and to match the behavior of `npm link`, `yarn link` would stay around as is. +* `yarn link` without the `--pack` option is similar to `yarn link --pack` but links directories instead of files. This means that any `node_modules` inside linked dependencies are also included, breaking the flattened and deduped tree that yarn normally provides. Many other differences have already been discussed in this RFC. Ideally, the behavior of `yarn link` would be what the `--pack` option provides, but for backwards compatibility and to match the behavior of `npm link`, `yarn link` would stay around as is. -* `link://` dependencies are similar to `yarn link` but actually closer to what `yarn knit` would provide. `link://` dependencies also cause the linked module's dependencies to be installed locally, so as long as there is no `node_modules` folder inside the linked dependency, it would work identically to `yarn knit dep`. The existing implementation of `link://` dependencies could be used to implement `yarn knit dep`. +* `link://` dependencies are similar to `yarn link dep` but actually closer to what `yarn add dep --link` would provide. `link://` dependencies also cause the linked module's dependencies to be installed locally, so as long as there is no `node_modules` folder inside the linked dependency, it would work identically to `yarn add dep --link`. The existing implementation of `link://` dependencies could be used to implement the `--link` option. * `file://` dependencies cause files to be copied instead of linked. This is not as useful because it means you must re-install every time a change is made to the dependency. * `yarn pack` + `yarn install dep.tgz` is similar to `file://` dependencies. The pack + install process must be re-run every time a change is made. It does correctly dedupe dependencies, however, as `node_modules` are excluded by `yarn pack`. -* [Workspaces](https://github.com/yarnpkg/rfcs/pull/66) are similar but solve a different problem. Where workspaces are great for a tree of related modules (e.g. a monorepo), `yarn knit` is for linking together modules in separate trees, e.g. things that might be shared between multiple workspaces. +* [Workspaces](https://github.com/yarnpkg/rfcs/pull/66) are similar but solve a different problem. Where workspaces are great for a tree of related modules (e.g. a monorepo), these options are for linking together modules in separate trees, e.g. things that might be shared between multiple workspaces. # How We Teach This -This proposal is mostly additive and affects only how people work on libraries that they are using in their apps. We would want to document the `knit` command in the "CLI Commands" section of the docs and perhaps add a new section to "The Yarn Workflow". +This proposal is mostly additive and affects only how people work on libraries that they are using in their apps. We would want to document the new options in the "CLI Commands" section of the docs and perhaps add a new section to "The Yarn Workflow". `yarn link` would stay around, so people migrating from the npm client wouldn't have to learn anything new at first. # Drawbacks -One issue with this proposal is that it's not clear what to put in the lockfile after running `yarn knit dep` since we don't have an npm URL for the dep yet -- it hasn't been published to npm. +One issue with this proposal is that it's not clear what to put in the lockfile after running `yarn add dep --link` since we don't have an npm URL for the dep yet -- it hasn't been published to npm. -Another issue is that if you change package.json in `dep`, namely changing a dependency or modifying the `files` entry, you have to run `cd dep; yarn knit; cd app; yarn knit dep`. Same if you add a file at the top level of the dependency since each file is linked individually. This isn't so bad as those changes are probably more rare than saving changes to existing files. +Another issue is that if you change package.json in `dep`, namely changing a dependency or modifying the `files` entry, you have to run `cd dep; yarn link --pack`. Same if you add a file at the top level of the dependency since each file is linked individually. This isn't so bad as those changes are probably more rare than saving changes to existing files. -Also, if you update the code in dep and bump its version, say from 1.0.0 to 1.1.0, the symlinks in ~/.yarn-knit/dep-1.0.0 will still point to the code in your working directory, which now contains 1.1.0 code. +Also, if you update the code in dep and bump its version, say from 1.0.0 to 1.1.0, the symlinks in ~/.yarn-link/dep-1.0.0 will still point to the code in your working directory, which now contains 1.1.0 code. The symlinks might break but I think that's mostly OK since at that point you're done working on dep and have published it to npm and it's easy to go run `yarn add dep` in app and not use the symlinks anymore. -If you want to truly pin the versions of knitted packages then you'd need to have a different working directory for each version. (Git worktrees are great for this use case actually. Worktrees let you check out a repo once and then magically create semi-clones of it in separate directories, with the constraint that the worktrees need to be on different branches, which is totally OK in this scenario. The worktrees all share the same Git repo though, so if you commit in one worktree you can cherry pick that commit within another worktree.) +If you want to truly pin the versions of linked packages then you'd need to have a different working directory for each version. (Git worktrees are great for this use case actually. Worktrees let you check out a repo once and then magically create semi-clones of it in separate directories, with the constraint that the worktrees need to be on different branches, which is totally OK in this scenario. The worktrees all share the same Git repo though, so if you commit in one worktree you can cherry pick that commit within another worktree.) Finally, another issue is with the way the node require resolution algorithm works. Dependencies of symlinked modules are resolved relative to the realpath, not the symlink. This means that you'll still get duplicates if both modules depend on a third dependency, or errors if that dependency is not installed in either place. This is solved by the recently added runtime option for node `--preserve-symlinks`, which skips getting the realpath when resolving modules. Something similar would need to be added to browserify/webpack to solve this there as well. I recently opened a [PR for browserify](https://github.com/substack/node-browserify/pull/1742) to support the same option. From 79a041172beedd6533a3a2f6d7f8bc50fb3fa870 Mon Sep 17 00:00:00 2001 From: Devon Govett Date: Thu, 3 Aug 2017 21:08:35 -0700 Subject: [PATCH 5/5] Rename proposal --- accepted/{0000-yarn-knit.md => 0000-link-options.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename accepted/{0000-yarn-knit.md => 0000-link-options.md} (100%) diff --git a/accepted/0000-yarn-knit.md b/accepted/0000-link-options.md similarity index 100% rename from accepted/0000-yarn-knit.md rename to accepted/0000-link-options.md