Skip to content

[windows] upgrade to Python 3.10.1 embeddable #83488

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+62 −11
Open

[windows] upgrade to Python 3.10.1 embeddable #83488

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 62 additions & 11 deletions utils/build.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,24 @@ $KnownPythons = @{
URL = "https://www.nuget.org/api/v2/package/pythonarm64/3.9.10";
SHA256 = "429ada77e7f30e4bd8ff22953a1f35f98b2728e84c9b1d006712561785641f69";
};
};
"3.10.1" = @{
AMD64 = @{
URL = "https://www.nuget.org/api/v2/package/python/3.10.1";
SHA256 = "987a0e446d68900f58297bc47dc7a235ee4640a49dace58bc9f573797d3a8b33";
};
AMD64_Embedded = @{
URL = "https://www.python.org/ftp/python/3.10.1/python-3.10.1-embed-amd64.zip";
SHA256 = "502670dcdff0083847abf6a33f30be666594e7e5201cd6fccd4a523b577403de";
};
ARM64 = @{
URL = "https://www.nuget.org/api/v2/package/pythonarm64/3.10.1";
SHA256 = "16becfccedf1269ff0b8695a13c64fac2102a524d66cecf69a8f9229a43b10d3";
};
ARM64_Embedded = @{
URL = "https://www.python.org/ftp/python/3.10.1/python-3.10.1-embed-arm64.zip";
SHA256 = "1f9e215fe4e8f22a8e8fba1859efb1426437044fb3103ce85794630e3b511bc2";
};
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets leave the definitions for 3.9, allowing to optionally upgrade to 3.10 while we iterate to get this working. That will allow you to make progress and get the changes merged earlier rather than trying to keep up to date with the changes in this script.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, thanks

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a particular reason that we need both the regular and the embedded URLs? Could we not get away with just the embedded?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, embeddable python does not come with pip. We could download and run get-pip.py, but I don't think we would gain a lot of CI time in doing so. I have not tried it out.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reasoning for the version selection here is not CI times but rather to match what llvm.org distributes. I think that we don't need to inject pip unless we are building the installer for distribution (i.e. nightlies/releases).

}

Expand All @@ -351,11 +369,6 @@ $PythonModules = @{
SHA256 = "353815f59a7f64cdaca1c0307ee13558a0512f6db064e92fe833784f08539c7a";
Dependencies = @();
};
"unittest2" = @{
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unittest2 is not compatible with python3.10+ since it's a back port of unittest. There is no need for it anymore.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets leave that in place until we have switched to 3.10 by default.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only references to unittest2 in the whole toolchain will be removed in swiftlang/llvm-project#11084. I think it's safe to remove it, as even the 3.9.10 version of Python comes with unittest.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, nice! In that case, we should split this change out and merge it whenever swiftlang/llvm-project#11084 gets merged.

Version = "1.1.0";
SHA256 = "22882a0e418c284e1f718a822b3b022944d53d2d908e1690b319a9d3eb2c0579";
Dependencies = @("argparse", "six", "traceback2", "linecache2");
};
"argparse" = @{
Version = "1.4.0";
SHA256 = "c31647edb69fd3d465a847ea3157d37bed1f95f19760b11a47aa91c04b666314";
Expand Down Expand Up @@ -602,6 +615,10 @@ function Get-PythonPath([Hashtable] $Platform) {
return [IO.Path]::Combine("$BinaryCache\", "Python$($Platform.Architecture.CMakeName)-$PythonVersion")
}

function Get-EmbeddedPythonPath([Hashtable] $Platform) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, is there a bigger picture around the embedded vs non-embedded python that the extraction needs special logic?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only reason is pip missing in embeddedable python. We build with the regular Python, but we bundle the embeddable python in the installer.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There-in lies the trick! We can inject pip into the packaging :)

return [IO.Path]::Combine("$BinaryCache\", "EmbeddedPython$($Platform.Architecture.CMakeName)-$PythonVersion")
}

function Get-PythonExecutable {
return [IO.Path]::Combine((Get-PythonPath $BuildPlatform), "tools", "python.exe")
}
Expand All @@ -610,6 +627,10 @@ function Get-PythonScriptsPath {
return [IO.Path]::Combine((Get-PythonPath $BuildPlatform), "tools", "Scripts")
}

function Get-EmbeddedPythonInstallDir() {
return [IO.Path]::Combine("$ImageRoot\", "Program Files", "Swift", "Python")
}

function Get-Syft {
return $KnownSyft[$SyftVersion][$BuildArchName]
}
Expand Down Expand Up @@ -1097,11 +1118,33 @@ function Get-Dependencies {
return $KnownPythons[$PythonVersion].$ArchName
}

function Get-KnownEmbeddedPython([string] $ArchName) {
if (-not $KnownPythons.ContainsKey($PythonVersion)) {
throw "Unknown python version: $PythonVersion"
}
if (-not $KnownPythons[$PythonVersion].ContainsKey("${ArchName}_Embedded")) {
return $null
}
return $KnownPythons[$PythonVersion]["${ArchName}_Embedded"]
}

function Install-Python([string] $ArchName) {
$Python = Get-KnownPython $ArchName
DownloadAndVerify $Python.URL "$BinaryCache\Python$ArchName-$PythonVersion.zip" $Python.SHA256
if (-not $ToBatch) {
Expand-ZipFile Python$ArchName-$PythonVersion.zip "$BinaryCache" Python$ArchName-$PythonVersion
Expand-ZipFile "Python$ArchName-$PythonVersion.zip" "$BinaryCache" "Python$ArchName-$PythonVersion"
}
}

function Install-EmbeddedPython([string] $ArchName) {
$Python = Get-KnownEmbeddedPython $ArchName
if ($Python -eq $null) {
Write-Output "Python $PythonVersion does not have an embeddable version."
return
}
DownloadAndVerify $Python.URL "$BinaryCache\EmbeddedPython$ArchName-$PythonVersion.zip" $Python.SHA256
if (-not $ToBatch) {
Expand-ZipFile "EmbeddedPython$ArchName-$PythonVersion.zip" "$BinaryCache" "EmbeddedPython$ArchName-$PythonVersion"
}
}

Expand Down Expand Up @@ -1149,11 +1192,11 @@ function Get-Dependencies {
if ($Test -contains "lldb") {
Install-PythonModule "psutil" # Required for testing LLDB
$env:Path = "$(Get-PythonScriptsPath);$env:Path" # For unit.exe
Install-PythonModule "unittest2" # Required for testing LLDB
}
}

Install-Python $HostArchName
Install-EmbeddedPython $HostArchName
if ($IsCrossCompiling) {
Install-Python $BuildArchName
}
Expand Down Expand Up @@ -3450,16 +3493,24 @@ function Install-HostToolchain() {

# Restructure _InternalSwiftScan (keep the original one for the installer)
Copy-Item -Force `
"$($HostPlatform.ToolchainInstallRoot)\usr\lib\swift\_InternalSwiftScan" `
"$($HostPlatform.ToolchainInstallRoot)\usr\include"
-Path "$($HostPlatform.ToolchainInstallRoot)\usr\lib\swift\_InternalSwiftScan" `
-Destination "$($HostPlatform.ToolchainInstallRoot)\usr\include"
Copy-Item -Force `
"$($HostPlatform.ToolchainInstallRoot)\usr\lib\swift\windows\_InternalSwiftScan.lib" `
"$($HostPlatform.ToolchainInstallRoot)\usr\lib"
-Path "$($HostPlatform.ToolchainInstallRoot)\usr\lib\swift\windows\_InternalSwiftScan.lib" `
-Destination "$($HostPlatform.ToolchainInstallRoot)\usr\lib"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The named parameters are a nice cleanup; we could split those out and merge them as a cleanup before even getting the python 3.10.1 upgrade completed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened #83517 to split the changes 👍


# Switch to swift-driver
$SwiftDriver = ([IO.Path]::Combine((Get-ProjectBinaryCache $HostPlatform Driver), "bin", "swift-driver.exe"))
Copy-Item -Force $SwiftDriver "$($HostPlatform.ToolchainInstallRoot)\usr\bin\swift.exe"
Copy-Item -Force $SwiftDriver "$($HostPlatform.ToolchainInstallRoot)\usr\bin\swiftc.exe"

# Copy embeddable Python
New-Item -Type Directory -Path "$(Get-EmbeddedPythonInstallDir)" -ErrorAction Ignore | Out-Null
Write-Output "$(Get-EmbeddedPythonInstallDir)"
Copy-Item -Force `
-Path "$(Get-EmbeddedPythonPath $HostPlatform)\*" `
-Destination "$(Get-EmbeddedPythonInstallDir)" `
-Recurse
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we don't want to embed the Python release. This was what we discussed on the forums - this should be an additional separate MSI that you are building and optionally installing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Embed was not the right choice of words. I want to put the files in a place where wix can find them for the follow up patch here: swiftlang/swift-installer-scripts#447

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that we should merge this into the toolchain. I think that we should be doing a separate installation: S:\Program Files\Swift\Python-3.10.1 for the image (the WiX authoring should be able to pick it up from there).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the latest revision, the python files are now extracted to S:\Program Files\Swift\Python. Do we really need the version number? Doing so would mean passing a new parameter to Wix to specify the files path right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer that we keep the version number. Upgrading python is something that would be rare, so I don't think that it is a big deal. We already have the information in build.ps1 where we can pass that down.

}

function Build-Inspect([Hashtable] $Platform) {
Expand Down