Implement a winsymlinks mode that prefers native symlinks, falling back to the deep copy mode

[dscho]

It surprises new MSYS2 users no end that ln -s does not create symbolic links at all, but deep copies (and with an exit code indicating success!). This did not at all match the expectations of those users who were familiar with Unix' concept of symbolic links and thought that they could rely on MSYS2 providing those, too, or fail with a non-zero exit code.

Historical reasons are at play here: When MSYS2 was started (or was that already the behavior of MSys? I forget...), symbolic links were not supported on Windows, at least not really: you had to have administrator privileges to create them (but not to delete them... 🤷) in Windows Vista, and before that, Windows simply had no idea about symbolic links.

So what about Cygwin? Well, Cygwin had something like support for symbolic links, using .lnk files for the emulation. The only problem? You had to stay within Cygwin's walled garden to make use of them. All non-Cygwin applications would react with a less or more unpleasant "huh?!?" when encountering those "symbolic links".

That's why MSYS2 chose to deep-copy by default. At least that way ./configure would still work for those projects that required symbolic links. This was instrumental in getting MSYS2's package ecosystem off the ground.

Even when a Windows 10 update introduced support for creating symbolic links without elevation as long as Windows was run in Developer Mode, the created symbolic links are not completely what Unix/Linux/macOS users may be used to, as Windows discerns between directory symlinks and file symlinks.

Be that as it may, now that we've dropped Windows 7 and Windows 8 support, it may be a good time to start switching the default to creating actual symbolic links by default.

Since we still support Windows 8.1 (and a couple of Windows 10 versions that do not allow creating symbolic links in non-elevated operations, even in Developer Mode), we cannot simply switch to a mode where the MSYS2 runtime creates symbolic links when asked for, but we have to have a mode where the MSYS2 runtime first checks whether that is possible with the Windows version on which it is running, and if not, falls back to the deep-copy.

This PR does precisely that: implement that mode, but does not yet flip the default away from deepcopy. The reason is that I want this to be tested by volunteers (myself included) first, and once it is deemed robust and stable enough, flip the default to nativeordeepcopy.

This addresses #113.

[jcrben]

Well, Cygwin had something like support for symbolic links, using .lnk files for the emulation.

Cygwin not only had, it still has symlinks like that - except the docs suggest that the *.lnk are not the default. The cygwin docs say this about their default symlinks https://cygwin.com/cygwin-ug-net/using.html#pathnames-symlinks

special reparse points shared with WSL (on NTFS on Windows 10 1607 or later)
plain files with the system attribute, containing a magic cookie followed by the path to which the link points.

The "shortcut" style links are something different.

The new Developer Mode feature is OK, but it's got a big warning on it when you enable it and enterprises therefore don't want to enable it:

Turning on developer mode, including installing and running apps from outside the Microsoft Store, could expose your device and personal data to security risks or harm your device.

https://stackoverflow.com/questions/74613686/what-does-the-warning-in-developer-mode-mean

I set export MSYS=winsymlinks:sys as I believe this restores the cygwin default special reparse point symlinks, which is fine by me. I also found https://carltonf.github.io/post/symbolic-links-on-windows sort of helpful in dumbing it down

[dscho]

MSYS=winsymlinks:sys

This mode is incompatible with regular Win32 programs; They won't understand those symlinks and misinterpret them all the time. That is an okay stance to take for Cygwin, which wants you to stay within its ecosystem. It is not an okay stance for MSYS2 which wants to integrate with native Win32 programs as much as possible.

The new Developer Mode feature is OK, but it's got a big warning on it when you enable it and enterprises therefore don't want to enable it:

Turning on developer mode, including installing and running apps from outside the Microsoft Store, could expose your device and personal data to security risks or harm your device.

Yes, developer mode comes with a lot of liberties that developing code requires. Nevertheless, it is the mode in which Windows users can create symbolic links in modern Windows without requiring an elevated process (which would increase the security risk a lot more than Developer Mode).

In any case, the mode I introduce here, and which I propose to promote to eventually be the default, retains backwards-compatibility in a fashion, by falling back to MSYS2's current behavior if symbolic links cannot be created.

[jeremyd2019]

I am thinking, would it make sense to try to generalize this? maybe something like winsymlinks:native:deepcopy or something? There's a sort of ambivalence in me: I don't want to just add another symlink type, but I also don't want to increase the combinatorial complexity of symlinks.

[lazka]

I am thinking, would it make sense to try to generalize this?

Can you think of any use case where a different config/order would be preferred by users?

[jeremyd2019]

Not necessarily, it's just now we have a lot of seemingly-poorly-defined fallback cases:

• native (or nativestrict if target isspecial) falls back to default
• default is actually wsl with a fallback to sysfile
• deepcopy falls back to sysfile if target isspecial

I guess now that I write it out, that's not too bad, everything that falls back eventually falls back to sysfile. Perhaps deepcopy should fall back to default also now for consistency (the wsl stuff obviously didn't exist when deepcopy was written). Moving deepcopy up into the block where native/strict happens would make it more obvious that it's a fallback case too.

[jeremyd2019]

Note that if target isspecial res will be -2, so will still fall back to default. This is probably reasonable, since deepcopy would just fall back to sysfile in that case. default will let it try wsl before falling back to sysfile. I'm thinking it might be best to make deepcopy fall back to default instead of sysfile anyway.

[jeremyd2019]

making deepcopy fall back to wsl instead of sysfile breaks one git test which tries to commit a symlink to /dev/null. The result of doing so is almost certainly not helpful (I bet it commits the sysfile, not a symlink), but when it's a wsl symlink it fails with Function not implemented

[dscho]

when it's a wsl symlink it fails with Function not implemented

That's my fault, as I have not managed to spend time on upstreaming Git for Windows' symlink support: git/git@master...dscho:git:support-symlinks-on-windows

[jeremyd2019]

I had a quick look at those changes, and I didn't see any handling of wsl symlinks in there, so I don't think that would help.

[dscho]

Huh. I misremembered. A git grep IO_REPARSE_TAG_LX_SYMLINK indeed comes up empty...

[dscho]

Perhaps deepcopy should fall back to default also now for consistency (the wsl stuff obviously didn't exist when deepcopy was written).

My rationale was: deepcopy exists only as a Win32-compatible fall-back if symlink support does not exist, and symlink support predates WSL (and both require Developer Mode to be enabled IIRC).

But then, this entire PR is about moving off of deepcopy in the first place, preparing for a time when MSYS2 drops support for Windows versions where symlinks can only be created when running elevated.

FWIW @jeremyd2019 I really like the refactoring you did, moving the deepcopy code into its own function, I'd just suggest to do it as a preparatory, separate commit instead of squashing it in.

[jcrben]

symlink support predates WSL (and both require Developer Mode to be enabled IIRC)

Not to derail into something off-topic - not sure if you're saying WSL2 requires Developer Mode - it doesn't. Probably easier to get WSL2 to a level that meets enterprise security than Developer Mode, altho that's no simple thing either.

There's been a trend for tools to just use junctions instead since they don't require admin privileges - you can see a bit of context at kubernetes-sigs/krew#843 - obv junctions just link a folder, but in many cases that can suffice

[dscho]

There's been a trend for tools to just use junctions instead since they don't require admin privileges

@jcrben It had been a trend, but thankfully it's no longer all that common. See e.g. this insightful comment for an explanation. Here's an excerpt:

Feature	Unix	Windows symlink	Windows junction
Target is relative to source directory	yes	yes	no
Target is relative to current computer	yes [1]	yes	no
Target is absolute	no	yes [2]	yes [3]
Link is resolved by	client	client	server

[1] Symlinks to absolute paths are resolved by the client and thus relative to the client computer. E.g. if /usr is mounted to some central network drive, the symlink /usr/local/etc -> /etc/local refers to /etc/local on the client.
[2] Via UNC path, e.g. \\server\share\path.
[3] Via local path, e.g. C:\path or \\?\Volume{...}\path.

tl;dr Windows junctions are nowhere close to being a valid symlink replacement.

[jcrben]

@dscho well, in some ways using junctions is becoming more common - fnm switched to junctions in 2021, volta just did in 2024, jbangdev did it in 2024 at my urging, kubectl plugin manager krew may also do it. avoiding a complex and scary step of enabling Developer Mode is pretty nice. wish we could get symlinks on Windows without enabling Developer Mode. See also: Why do you have to be an admin to create a symlink in Windows?

[sskras]

@jcrben writes:

wish we could get symlinks on Windows without enabling Developer Mode. See also: Why do you have to be an admin to create a symlink in Windows?

Exactly that. You don't need the Developer Mode or become elevated every time you create a symlink. Assigning SeCreateSymbolicLinkPrivilege to your user once is enough. That's what I did in the past on my w10, and I am running fine since.

[dscho]

in some ways using junctions is becoming more common

@jcrben Yes, and that's true even when mentioning better-known projects (I never heard about your examples, but IIRC pnpm is an example where junctions are used).

The important thing to keep in mind? The usage resembles Unix mount points, much more than symbolic links. I do encourage you to study the insightful comment about this that I linked earlier.

Assigning SeCreateSymbolicLinkPrivilege to your user once is enough. That's what I did in the past on my w10, and I am running fine since.

@sskras That's enticing, but leaves the reader hanging for a clear description how to do that. In other words, your comment is too terse, saves too much on time at the cost of being less helpful than it could be. Please do fill in the blanks.

[jeremyd2019]

https://packages.msys2.org/packages/mingw-w64-x86_64-editrights
https://cygwin.com/cgit/cygwin-apps/editrights/tree/editrights.README

editrights -u YourUser -a SeCreateSymbolicLinkPrivilege

? (untested)

[sskras]

@dscho commented 3 days ago

Assigning SeCreateSymbolicLinkPrivilege to your user once is enough. That's what I did in the past on my w10, and I am running fine since.

@sskras That's enticing, but leaves the reader hanging for a clear description how to do that. [...] Please do fill in the blanks.

OK. There are different ways/tools to configure this:

1, using secpol.msc GUI.
2, using secedit.exe CLI.
3, using ntrights.exe old CLI (it was present on the old Windows Server Resource Kit only, and it harder to find).
4, using 3rd party tools, eg:

• https://github.com/pandell/NTRightsNet (C#)
• https://github.com/jcasale/UserRights (C#)
• https://github.com/petemoore/ntr (Golang)
• https://github.com/zippy1981/ntrights (C++ PoC, requires hardcoding)

I didn't think earlier a Windows developer would need guidance on that, but now visualized usage of the 1st tool:

0. Open "Run" dialog, type secpol.msc.
1. Press OK ("Local Security Policy" opens).
2. Navigate to "Local Policies", expand it.
3. Navigate to "User Rights Assignment".
4. Double-click "Create symbolic links" ("Create symbolic links Properties" dialog opens).
5. Click "Add User or Group..." ("Select Users or Groups" dialog opens).
6. Type in the needed username.
7. Press OK twice.
8. Check if the username already occurs on the line from step 4.
9. Maybe run gpupdate after that (I already forgot / didn't test that).

---

@jeremyd2019 commented 3 days ago

https://packages.msys2.org/packages/mingw-w64-x86_64-editrights https://cygwin.com/cgit/cygwin-apps/editrights/tree/editrights.README

editrights -u YourUser -a SeCreateSymbolicLinkPrivilege

? (untested)

Thanks, I didn't know about this tool! Might be a nice addition to the list above.

[dscho]

Honestly, I was thinking more about an automatable solution that requires nothing but PowerShell, something like:

# Export the current security settings to a file
$SecEditExportFile = "$env:TEMP\secedit.inf"
secedit /export /cfg $SecEditExportFile /areas USER_RIGHTS

# Modify the security settings file to include the privilege for the user
$User = "DOMAIN\User"
$Privilege = "SeCreateSymbolicLinkPrivilege"

# Read the exported settings file
$lines = Get-Content $SecEditExportFile

# Add the privilege for the user
$index = $lines.IndexOf("[Privilege Rights]")
$privilegeLine = $lines[$index..($lines.Length - 1)] | Where-Object { $_ -match "^$Privilege" }

if ($privilegeLine) {
    $existingValues = $privilegeLine -replace "$Privilege = ", ""
    $newValues = "$existingValues,$User"
    $lines[$lines.IndexOf($privilegeLine)] = "$Privilege = $newValues"
} else {
    $lines += "$Privilege = $User"
}

# Write the modified settings back to the file
$lines | Set-Content $SecEditExportFile

# Import the modified security settings:
secedit /configure /db secedit.sdb /cfg $SecEditExportFile /areas USER_RIGHTS

# Refresh the policy:
gpupdate /force

Since this here is a public bug tracker, one must assume that people might find this and would be delighted by guidance how to achieve the goal they set out to accomplish when trying to find more information.

I didn't think earlier a Windows developer would need guidance on that, but now visualized usage of the 1st tool:

Stating that you didn't think earlier a Windows developer would need guidance on that may very well come over as pretty smug and arrogant, and is therefore potentially something you'd like to avoid.

[sskras]

@jeremyd2019, editrights that you found seems to work just fine:

Thank you.

[sskras]

@dscho wrote:

Honestly, I was thinking more about an automatable solution that requires nothing but PowerShell, something like:

  ...
secedit /export /cfg $SecEditExportFile /areas USER_RIGHTS
  ...
secedit /configure /db secedit.sdb /cfg $SecEditExportFile /areas USER_RIGHTS
  ...

It still relies on running secedit a couple of times internally. Could be done quickly using sh + AWK too, eg:

Easily extendable to add the new SeCreateSymbolicLinkPrivilege = ... line in case it was missing, but unsure if that's needed.

IIUC this would mean that the current install of the supported Windows versions (w10, w11) is misconfigured or pretty broken, eg. having corrupt Security Database. But I am not against that.

And... how is it better than using editrights ?

Since this here is a public bug tracker, one must assume that people might find this and would be delighted by guidance how to achieve the goal they set out to accomplish when trying to find more information.

I see your point. For starters, as an ordinary user of MSYS2 I hope to find such info on the installation page, eg: https://www.msys2.org/#installation

Crossing fingers so the ways get discussed properly and then get better publicity. Cheers :)

I didn't think earlier a Windows developer would need guidance on that, but now visualized usage of the 1st tool:

Stating that you didn't think earlier a Windows developer would need guidance on that may very well come over as pretty smug and arrogant, and is therefore potentially something you'd like to avoid.

That's sad. What drove me is not the pride, but the ambiguity of the ways to solve this + my aim at keeping the Signal-to-Noise ratio high. I hope I proved that by illustrating two of these ways.

[culyun]

Came here looking for answers, read the thread, got good info.

The conversation is fine.
Not smug, just direct.

Probably the main concern is making this work out of the box in the corporate environment.
Coorporate IT tends to be very defensive about exceptions that bubble up in GUI settings like "Developer Mode".

Automatically enabling NTFS symlinks with an opt-out (like Git-Bash) should be the chief goal here.
This is merely codifying the principles of no harm and implicit permission granted through SeCreateSymbolicLinkPrivilege.