[windows] add Embeddable Python 3.10.1 to build.ps1 #83488

charles-zablit · 2025-08-01T16:13:58Z

This patch adds Embeddable Python 3.10.1 to build.ps1.

This is part of 2 other PRs which aim to package Python with the Swift toolchain installer on Windows:

[lldb][windows] delay loading liblldb.dll and manually add Python's DLL directory llvm-project#11571
[windows] add an option to install Embeddable Python 3.10.1 in the toolchain installer swift-installer-scripts#447

This is a follow-up to this discussion on the forums.

rdar://157773554

charles-zablit · 2025-08-01T16:14:44Z

@swift-ci please test

charles-zablit · 2025-08-01T16:15:29Z

utils/build.ps1

    SHA256 = "353815f59a7f64cdaca1c0307ee13558a0512f6db064e92fe833784f08539c7a";
    Dependencies = @();
  };
-  "unittest2" = @{


unittest2 is not compatible with python3.10+ since it's a back port of unittest. There is no need for it anymore.

Lets leave that in place until we have switched to 3.10 by default.

The only references to unittest2 in the whole toolchain will be removed in swiftlang/llvm-project#11084. I think it's safe to remove it, as even the 3.9.10 version of Python comes with unittest.

Ah, nice! In that case, we should split this change out and merge it whenever swiftlang/llvm-project#11084 gets merged.

[windows] remove unittest2 from build.ps1 #83519 was merged 👍

compnerd · 2025-08-01T16:15:31Z

utils/build.ps1

+      URL = "https://www.python.org/ftp/python/3.10.1/python-3.10.1-embed-arm64.zip";
+      SHA256 = "1f9e215fe4e8f22a8e8fba1859efb1426437044fb3103ce85794630e3b511bc2";
    };
  }


Lets leave the definitions for 3.9, allowing to optionally upgrade to 3.10 while we iterate to get this working. That will allow you to make progress and get the changes merged earlier rather than trying to keep up to date with the changes in this script.

Fixed, thanks

Is there a particular reason that we need both the regular and the embedded URLs? Could we not get away with just the embedded?

Yes, embeddable python does not come with pip. We could download and run get-pip.py, but I don't think we would gain a lot of CI time in doing so. I have not tried it out.

The reasoning for the version selection here is not CI times but rather to match what llvm.org distributes. I think that we don't need to inject pip unless we are building the installer for distribution (i.e. nightlies/releases).

compnerd · 2025-08-01T16:16:13Z

utils/build.ps1

+  Copy-Item -Force `
+    -Path "$(Get-EmbeddedPythonPath $HostPlatform)\*" `
+    -Destination "$($HostPlatform.ToolchainInstallRoot)\usr\bin" `
+    -Recurse


No, we don't want to embed the Python release. This was what we discussed on the forums - this should be an additional separate MSI that you are building and optionally installing.

Embed was not the right choice of words. I want to put the files in a place where wix can find them for the follow up patch here: swiftlang/swift-installer-scripts#447

I don't think that we should merge this into the toolchain. I think that we should be doing a separate installation: S:\Program Files\Swift\Python-3.10.1 for the image (the WiX authoring should be able to pick it up from there).

In the latest revision, the python files are now extracted to S:\Program Files\Swift\Python. Do we really need the version number? Doing so would mean passing a new parameter to Wix to specify the files path right?

I would prefer that we keep the version number. Upgrading python is something that would be rare, so I don't think that it is a big deal. We already have the information in build.ps1 where we can pass that down.

Added the Python version in the latest revision 👍

compnerd · 2025-08-04T15:24:54Z

utils/build.ps1

+      URL = "https://www.python.org/ftp/python/3.10.1/python-3.10.1-embed-arm64.zip";
+      SHA256 = "1f9e215fe4e8f22a8e8fba1859efb1426437044fb3103ce85794630e3b511bc2";
    };
  }


Is there a particular reason that we need both the regular and the embedded URLs? Could we not get away with just the embedded?

compnerd · 2025-08-04T15:26:03Z

utils/build.ps1

    SHA256 = "353815f59a7f64cdaca1c0307ee13558a0512f6db064e92fe833784f08539c7a";
    Dependencies = @();
  };
-  "unittest2" = @{


Ah, nice! In that case, we should split this change out and merge it whenever swiftlang/llvm-project#11084 gets merged.

compnerd · 2025-08-04T15:26:36Z

utils/build.ps1

  return [IO.Path]::Combine("$BinaryCache\", "Python$($Platform.Architecture.CMakeName)-$PythonVersion")
 }

+function Get-EmbeddedPythonPath([Hashtable] $Platform) {


Again, is there a bigger picture around the embedded vs non-embedded python that the extraction needs special logic?

The only reason is pip missing in embeddedable python. We build with the regular Python, but we bundle the embeddable python in the installer.

There-in lies the trick! We can inject pip into the packaging :)

I don't think injecting pip would work well because of this SO comment, especially because the Python documentation explicitly states:

Using pip to manage dependencies as for a regular Python installation is not supported with this distribution [embeddable Python]

Given that, and the extra steps it would take to manually install pip with the embeddable python, I think it's simpler to:

Build with the regular Python

Test with the regular Python (easier to install dependencies)

Package the embeddable Python.

I think that we should just use a boolean state on the known dictionary rather than extract this. Overall, I'd like to see the python distribution to be complete enough for use in scripting (i.e. has pip).

I think that we should just use a boolean state on the known dictionary rather than extract this.

I'm not sure I understand what that boolean state would look like: would it be something like hasPip?

Overall, I'd like to see the python distribution to be complete enough for use in scripting (i.e. has pip).

I agree that it would be more convenient, however in that case, why not package a complete Python install? pip is not supported by embeddable Python: https://github.com/swiftlang/swift/pull/83488/files#r2254566330.

I see the Embeddable Python in the installer feature as a way for users to get a functional lldb whether they have the correct Python version installed or not. This is the main source of confusion for users: llvm/llvm-project#74073 (comment)

I think it's reasonable to expect power users who want to have scripting features to install Python themselves and have proper pip support rather than to risk using pip in an unsupported way with Embeddable Python.

compnerd · 2025-08-04T15:27:42Z

utils/build.ps1

-    "$($HostPlatform.ToolchainInstallRoot)\usr\lib\swift\windows\_InternalSwiftScan.lib" `
-    "$($HostPlatform.ToolchainInstallRoot)\usr\lib"
+    -Path "$($HostPlatform.ToolchainInstallRoot)\usr\lib\swift\windows\_InternalSwiftScan.lib" `
+    -Destination "$($HostPlatform.ToolchainInstallRoot)\usr\lib"


The named parameters are a nice cleanup; we could split those out and merge them as a cleanup before even getting the python 3.10.1 upgrade completed.

Opened #83517 to split the changes 👍

charles-zablit · 2025-10-06T15:53:17Z

@swift-ci please smoke test

compnerd · 2025-10-06T15:55:50Z

utils/build.ps1

+      if ($null -eq $Python) {
+        Write-Output "Python $PythonVersion does not have an embeddable version."
+        return
+      }


I think that we should handle this similarly to Install-Python.

Sorry I'm not sure I understand: the 2 functions are identical to me apart from the if ($null -eq $Python) { check.

compnerd · 2025-10-06T15:57:07Z

utils/build.ps1

  return [IO.Path]::Combine("$BinaryCache\", "Python$($Platform.Architecture.CMakeName)-$PythonVersion")
 }

+function Get-EmbeddedPythonPath([Hashtable] $Platform) {


I think that we should just use a boolean state on the known dictionary rather than extract this. Overall, I'd like to see the python distribution to be complete enough for use in scripting (i.e. has pip).

charles-zablit requested a review from compnerd as a code owner August 1, 2025 16:13

charles-zablit self-assigned this Aug 1, 2025

charles-zablit requested a review from adrian-prantl August 1, 2025 16:14

charles-zablit added Windows Platform: Windows lldb labels Aug 1, 2025

charles-zablit commented Aug 1, 2025

View reviewed changes

compnerd requested changes Aug 1, 2025

View reviewed changes

charles-zablit mentioned this pull request Aug 4, 2025

[windows] add an option to install Embeddable Python 3.10.1 in the toolchain installer swiftlang/swift-installer-scripts#447

Open

charles-zablit force-pushed the charles-zablit/windows/upgrade-python-to-3.10 branch 2 times, most recently from bc91500 to bd4476a Compare August 4, 2025 15:12

compnerd reviewed Aug 4, 2025

View reviewed changes

charles-zablit force-pushed the charles-zablit/windows/upgrade-python-to-3.10 branch 3 times, most recently from 1fc553e to 7d3fdec Compare August 8, 2025 15:44

charles-zablit force-pushed the charles-zablit/windows/upgrade-python-to-3.10 branch 2 times, most recently from 800ed49 to baf0b97 Compare September 8, 2025 15:14

charles-zablit mentioned this pull request Oct 6, 2025

[lldb][windows] delay loading liblldb.dll and manually add Python's DLL directory swiftlang/llvm-project#11571

Closed

charles-zablit force-pushed the charles-zablit/windows/upgrade-python-to-3.10 branch from baf0b97 to 2b42dd6 Compare October 6, 2025 15:50

charles-zablit requested a review from compnerd October 6, 2025 15:53

compnerd reviewed Oct 6, 2025

View reviewed changes

charles-zablit changed the title ~~[windows] upgrade to Python 3.10.1 embeddable~~ [windows] add Embeddable Python 3.10.1 to build.ps1 Oct 6, 2025

[windows] add Embeddable Python to build.ps1

a83efe5

charles-zablit force-pushed the charles-zablit/windows/upgrade-python-to-3.10 branch from 2b42dd6 to a83efe5 Compare October 8, 2025 15:46

[windows] add Embeddable Python 3.10.1 to build.ps1 #83488

Are you sure you want to change the base?

[windows] add Embeddable Python 3.10.1 to build.ps1 #83488

Conversation

charles-zablit commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

charles-zablit commented Aug 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charles-zablit Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charles-zablit commented Oct 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charles-zablit commented Aug 1, 2025 •

edited

Loading

charles-zablit Aug 5, 2025 •

edited

Loading