[windows] add Python 3.10.1 to the installer #447
Conversation
charles-zablit
requested review from
shahmishal,
compnerd and
etcwilde
as code owners
| @@ -0,0 +1,5 @@ | |||
| <Project Sdk="WixToolset.Sdk/4.0.5"> | |||
| <PropertyGroup> | |||
| <OutputName>python</OutputName> | |||
There was a problem hiding this comment.
since this is going be part of the toolchain, we will need to have one per variant. here is my last change for how to make the authoring work for all variants: #444
i can help make that happen if needed.
There was a problem hiding this comment.
Do we really need 1 build per variant? I think it should be possible to share the Python installation across the different toolchain variants. We should lay them out such that they are isolated:
[INSTALLROOT]\Swift\Python-3.10.1\...
If we layout python this way, it would be isolated from any system installations and should be possible to share across different variants of the toolchain.
There was a problem hiding this comment.
[INSTALLROOT]\Swift\Python-3.10.1\...
Just to make sure I understand, would the Embeddable Python actually live in [INSTALLROOT]\Swift\Python-3.10.1\...? As in, once installed, python.exe will be at [INSTALLROOT]\Swift\Python-3.10.1\python.exe?
In that case, how would lldb resolve the path to the embeddable python?
My initial plan was to extract the Python files into T:\Program Files\Swift\Python like you suggested here and then install them in the bin directory alongside lldb.exe.
There was a problem hiding this comment.
The installation information should be queryable and you can extract that from the registry. This is basically what tcrun does as well.
platforms/Windows/python/python.wxs
Outdated
| Version="$(NonSemVerProductVersion)" | ||
| Scope="$(PackageScope)"> | ||
|
|
||
| <Media Id="1" Cabinet="$(VariantCabinetName)" EmbedCab="$(ArePackageCabsEmbedded)" /> |
There was a problem hiding this comment.
this is not going to build without these variables defined. i would recommend moving this file to a wxi, and defining these variables in a wxs file that imports it. see https://github.com/swiftlang/swift-installer-scripts/blob/main/platforms/Windows/bld/asserts/bld.asserts.wxs
There was a problem hiding this comment.
Fixed, thanks! However, I think the file structure in swift-installer-scripts\platforms\Windows\python is not correct: the subdirectory should not be named asserts. Do you have a suggestion of what this should be named? Or should I just adopt a flat directory structure?
There was a problem hiding this comment.
I think that cpy.wxs (as this is CPython) is fine as a name.
There was a problem hiding this comment.
Right now, the file structure is:
python/
├─ asserts/
│ ├─ python.wxs
│ ├─ python.wixproject
python.wxi
I will rename the python files to cpy but what should the asserts folder be named?
There was a problem hiding this comment.
@mhegazy regarding the variant name, I would really like to remove asserts for Python as it's not an asserts build, just a regular Python distribution. Would the following file structure be OK?
python/
├─ python.wxs
├─ python.wixproject
├─ python.wxi
platforms/Windows/python/python.wxs
Outdated
| </Component> | ||
|
|
||
| <Component Directory="toolchain_$(VariantName)_usr_bin"> | ||
| <File Source="$(PythonRoot)\python.exe" /> |
There was a problem hiding this comment.
do we really need the executable? i thought that all lldb needed is the dll and the built in modules?
There was a problem hiding this comment.
Yes, we should include thee executable. Ultimately, the issue with all this is that LLDB is meant to be usable as a scriptable debugger, including for post-mortem scenarios. In such a case, you want to be able to execute a script, which is going to require the interpreter.
There was a problem hiding this comment.
do we really need the executable? i thought that all lldb needed is the dll and the built in modules?
On Windows, following this discussion, we want to bundle an executable Python with the toolchain installer.
| @@ -9,6 +9,7 @@ | |||
| Scope="$(PackageScope)"> | |||
|
|
|||
| <?define PlatformRoot = "$(ImageRoot)\Platforms\Windows.platform"?> | |||
| <?define PythonRoot = "$(ImageRoot)\python"?> | |||
There was a problem hiding this comment.
this should be dedined in python msi project and not here.
| @@ -19,6 +19,7 @@ | |||
| <ProjectReference Include="..\bld\asserts\bld.asserts.wixproj" BindName="bld.asserts" /> | |||
| <ProjectReference Include="..\cli\asserts\cli.asserts.wixproj" BindName="cli.asserts" /> | |||
| <ProjectReference Include="..\dbg\asserts\dbg.asserts.wixproj" BindName="dbg.asserts" /> | |||
| <ProjectReference Include="..\python\python.wixproj" BindName="python" /> | |||
There was a problem hiding this comment.
Lets shuffle this down after all ide MSI. I wonder if it makes sense to rename this to match the 3-letter naming.
There was a problem hiding this comment.
I moved it down below ide. Should it be at the very bottom?
Regarding the 3-letter renaming, I'm not opposed to it, however I think it would be good to differentiate between embedded Python and regular Python if we ever decide to bundle/chain the full Python msi with the toolchain installer.
If we are sure we are never going to chain the full Python msi, then we could name this py3?
There was a problem hiding this comment.
I am certainly curious about why the differentiation matters.
Ultimately, the difference between the embedded and full python is something that I think that we should somewhat blur. The actual difference is the lack of an installer (not a problem), a slightly smaller standard library (is that truly a concern?), and the missing pip which we can inject. So, difference-wise it is pretty small.
They also do not author MSIs, which means that the chaining would be more complicated than if they provided a MSM/MSI.
platforms/Windows/bundle/theme.xml
Outdated
| <Checkbox Name="OptionsInstallAndroidSDKAMD64" X="210" Y="363" Width="-11" Height="17" TabStop="yes" FontId="3" EnableCondition="OptionsInstallAndroidPlatform">#(loc.Sdk_ProductName_Android_amd64)</Checkbox> | ||
| <Checkbox Name="OptionsInstallAndroidSDKARM" X="210" Y="381" Width="-11" Height="17" TabStop="yes" FontId="3" EnableCondition="OptionsInstallAndroidPlatform">#(loc.Sdk_ProductName_Android_armv7)</Checkbox> | ||
| <Checkbox Name="OptionsInstallAndroidSDKX86" X="210" Y="399" Width="-11" Height="17" TabStop="yes" FontId="3" EnableCondition="OptionsInstallAndroidPlatform">#(loc.Sdk_ProductName_Android_x86)</Checkbox> | ||
| <Checkbox Name="OptionsInstallEmbeddedPython" X="192" Y="165" Width="-11" Height="17" TabStop="yes" FontId="3">#(loc.EmbeddedPython_ProductName)</Checkbox> |
There was a problem hiding this comment.
I think that this should come at the very end as it is not really part of the toolchain, it is a third party dependency we are installing.
| @@ -9,6 +9,7 @@ | |||
| Scope="$(PackageScope)"> | |||
|
|
|||
| <?define PlatformRoot = "$(ImageRoot)\Platforms\Windows.platform"?> | |||
| <?define PythonRoot = "$(ImageRoot)\python"?> | |||
| @@ -0,0 +1,5 @@ | |||
| <Project Sdk="WixToolset.Sdk/4.0.5"> | |||
| <PropertyGroup> | |||
| <OutputName>python</OutputName> | |||
There was a problem hiding this comment.
Do we really need 1 build per variant? I think it should be possible to share the Python installation across the different toolchain variants. We should lay them out such that they are isolated:
[INSTALLROOT]\Swift\Python-3.10.1\...
If we layout python this way, it would be isolated from any system installations and should be possible to share across different variants of the toolchain.
platforms/Windows/python/python.wxs
Outdated
| Name="$(VariantProductName)" | ||
| UpgradeCode="$(VariantUpgradeCode)" | ||
| Version="$(NonSemVerProductVersion)" | ||
| Scope="$(PackageScope)"> |
There was a problem hiding this comment.
How do we version this component? Should we version it as per the Python version or the Swift toolchain version?
platforms/Windows/python/python.wxs
Outdated
| </Component> | ||
|
|
||
| <Component Directory="toolchain_$(VariantName)_usr_bin"> | ||
| <File Source="$(PythonRoot)\python.exe" /> |
There was a problem hiding this comment.
Yes, we should include thee executable. Ultimately, the issue with all this is that LLDB is meant to be usable as a scriptable debugger, including for post-mortem scenarios. In such a case, you want to be able to execute a script, which is going to require the interpreter.
charles-zablit
force-pushed
the
charles-zablit/windows/embed-python-in-the-installer
branch
from
d806397 to
91159fc
Compare
charles-zablit
changed the title
| <?define PythonRoot = "$(ImageRoot)\Python"?> | ||
|
|
||
| <?include ../python.wxi ?> | ||
| </Wix> |
| <?define VariantName = asserts ?> | ||
| <?define VariantUpgradeCode = $(PythonUpgradeCode)?> | ||
| <?define VariantProductName = !(loc.EmbeddedPython_ProductName)?> | ||
| <?define VariantCabinetName = python.asserts.cab?> |
There was a problem hiding this comment.
I agree with you, the cabinet name should not have .asserts in the name. We do not build the python interpreter with assertions.
There was a problem hiding this comment.
we install the embedded-python binaries to a variant-specific folder. so this means that each variant will need a copy.
the right approach would be to install python to a non-variant-specific folder, e.g. it is own folder. and add it to the path so that lldb would find it. this way we only have one msi. bonus if we change lldb to use an env var or other input to know where the embedded one is instead of using path for the dll and PYTHONPATH for the libraries.
platforms/Windows/python/python.wxi
Outdated
|
|
||
| <ComponentGroup Id="EmbeddedPython"> | ||
| <Component Directory="toolchain_$(VariantName)_usr_bin"> | ||
| <File Source="$(PythonRoot)\libcrypto-1_1-arm64.dll" /> |
There was a problem hiding this comment.
Is this file included on both ARM64 and AMD64?
There was a problem hiding this comment.
It's not, I've added the ArchSuffix define to address this. Thanks!
platforms/Windows/python/python.wxi
Outdated
| </Component> | ||
|
|
||
| <Component Directory="toolchain_$(VariantName)_usr_bin"> | ||
| <File Source="$(PythonRoot)\libssl-1_1-arm64.dll" /> |
There was a problem hiding this comment.
It's not, I've added the ArchSuffix define to address this. Thanks!
platforms/Windows/python/python.wxi
Outdated
| <Component Directory="toolchain_$(VariantName)_usr_bin"> | ||
| <File Source="$(PythonRoot)\vcruntime140.dll" /> | ||
| </Component> | ||
|
|
||
| <Component Directory="toolchain_$(VariantName)_usr_bin"> | ||
| <File Source="$(PythonRoot)\vcruntime140_1.dll" /> | ||
| </Component> |
There was a problem hiding this comment.
This brings up the question - do we want to bundle an additional copy of the VCRuntime? This is an interesting question because the toolchain itself also does depend on the runtime. There is a copy in the system. And now in python. The system one however will get security updates, this will not. Since part of the argument for this upgrade is "security" - we shouldn't be bundling them IMO.
platforms/Windows/python/python.wxi
Outdated
| <WixVariable Id="SideBySidePackageUpgradeCode" Value="$(VariantUpgradeCode)" /> | ||
| <FeatureGroupRef Id="SideBySideUpgradeStrategy" /> | ||
|
|
||
| <ComponentGroup Id="EmbeddedPython"> |
There was a problem hiding this comment.
Can the directory attribute be moved to the ComponentGroup instead of the Component? That would simplify the authoring here.
platforms/Windows/python/python.wxi
Outdated
| </Component> | ||
| </ComponentGroup> | ||
|
|
||
| <Feature Id="EmbeddedPython" AllowAbsent="yes" Title="$(VariantProductName)"> |
There was a problem hiding this comment.
I don't think that we should allow this component to be absent. We would install nothing then.
platforms/Windows/python/python.wxi
Outdated
|
|
||
| <Component Directory="toolchain_$(VariantName)_usr_bin"> | ||
| <File Source="$(PythonRoot)\LICENSE.txt" /> | ||
| </Component> |
There was a problem hiding this comment.
Why is the LICENSE file in /usr/bin? Shouldn't that be under /usr/share/licenses?
There was a problem hiding this comment.
I did not know about usr/share/licenses. Created the directory and added the license there 👍
| @@ -24,6 +24,7 @@ | |||
| <RtlUpgradeCode>{BEA8C6DC-F73E-445B-9486-2333D1CF2886}</RtlUpgradeCode> | |||
| <AndroidPlatformUpgradeCode>{313B9C1F-D5B5-4FED-B7E0-138F1EE6B26A}</AndroidPlatformUpgradeCode> | |||
| <WindowsPlatformUpgradeCode>{01AFF1CF-A025-41B6-BCBC-728D794353FD}</WindowsPlatformUpgradeCode> | |||
| <PythonUpgradeCode>{5FC42BA9-ABF5-4CCD-B93B-BDFED936BA37}</PythonUpgradeCode> | |||
There was a problem hiding this comment.
Name should have variant to allow us to add another one later on. PythonAssertsUpgradeCode
| @@ -20,6 +20,7 @@ | |||
| <ProjectReference Include="..\cli\asserts\cli.asserts.wixproj" BindName="cli.asserts" /> | |||
| <ProjectReference Include="..\dbg\asserts\dbg.asserts.wixproj" BindName="dbg.asserts" /> | |||
| <ProjectReference Include="..\ide\asserts\ide.asserts.wixproj" BindName="ide.asserts" /> | |||
| <ProjectReference Include="..\python\asserts\python.wixproj" BindName="python" /> | |||
There was a problem hiding this comment.
| <ProjectReference Include="..\python\asserts\python.wixproj" BindName="python" /> | |
| <ProjectReference Include="..\python\asserts\python.asserts.wixproj" BindName="python.asserts" /> |
| @@ -0,0 +1,5 @@ | |||
| <Project Sdk="WixToolset.Sdk/4.0.5"> | |||
| <PropertyGroup> | |||
| <OutputName>python</OutputName> | |||
There was a problem hiding this comment.
The msi name should have the variant in it to separate it when we build multiple for each variant
| <OutputName>python</OutputName> | |
| <OutputName>python.asserts</OutputName> |
| @@ -8,6 +8,7 @@ | |||
| <String Id="CliAsserts_ProductName" Value="Swift Command Line Tools (Asserts)" /> | |||
| <String Id="Dbg_ProductName" Value="Swift Debugging Tools" /> | |||
| <String Id="DbgAsserts_ProductName" Value="Swift Debugging Tools (Asserts)" /> | |||
| <String Id="EmbeddedPython_ProductName" Value="Embedded Python 3.10.1" /> | |||
There was a problem hiding this comment.
Since this is variant specific msi, we should have the name reflect that.
| <String Id="EmbeddedPython_ProductName" Value="Embedded Python 3.10.1" /> | |
| <String Id="EmbeddedPythonAsserts_ProductName" Value="Embedded Python 3.10.1 (Asserts)" /> |
| <?define VariantCabinetName = python.asserts.cab?> | ||
| <?define ToolchainVersionedVariantDirectory = ToolchainVersionedAsserts ?> | ||
| <?define VariantEnvironmentComponentGUID = 30629e0c-b376-47bc-bedf-fefb7d4ca61d?> | ||
| <?if $(ProductArchitecture) = "arm64" ?> |
There was a problem hiding this comment.
this is not variant specific. i would recommend moving it to the wxi to avoid it being duplicated for diffrent variants.
This patch adds an option to install the embeddable version of Python directly from the toolchain installer on Windows.
The installer will pick up the python files from
T:\Program Files\Swift\Pythonand install them in thebindirectory alongsidelldb.exe.This is in continuation of the work in swiftlang/swift#83488.