Merge PEP 660 implementation (#3488)

Author: abravalheri

bootstrap.egg-info/entry_points.txt

@@ -2,6 +2,7 @@
 egg_info = setuptools.command.egg_info:egg_info
 build_py = setuptools.command.build_py:build_py
 sdist = setuptools.command.sdist:sdist
+editable_wheel = setuptools.command.editable_wheel:editable_wheel
 
 [distutils.setup_keywords]
 include_package_data = setuptools.dist:assert_bool

changelog.d/3265.breaking.rst

@@ -0,0 +1,15 @@
+Added implementation for *editable install* hooks (PEP 660).
+
+By default the users will experience a *lenient* behavior  which prioritises
+the ability of the users of changing the distributed packages (e.g. adding new
+files or removing old ones).
+But they can also opt into a *strict* mode, which will try to replicate as much
+as possible the behavior of the package as if it would be normally installed by
+end users. The *strict* editable installation is not able to detect if files
+are added or removed from the project (a new installation is required).
+
+.. important::
+   The *editable* aspect of the *editable install* supported this implementation
+   is restricted to the Python modules contained in the distributed package.
+   Changes in binary extensions (e.g. C/C++), entry-point definitions,
+   dependencies, metadata, datafiles, etc may require a new installation.

changelog.d/3380.change.rst

@@ -0,0 +1,10 @@
+Improved the handling of the ``config_settings`` parameter in both PEP 517 and
+PEP 660 interfaces:
+
+- It is possible now to pass both ``--global-option`` and ``--build-option``.
+  As discussed in #1928, arbitrary arguments passed via ``--global-option``
+  should be placed before the name of the setuptools' internal command, while
+  ``--build-option`` should come after.
+
+- Users can pass ``editable-mode=strict`` to select a strict behaviour for the
+  editable installation.

changelog.d/3380.deprecation.rst

@@ -0,0 +1,7 @@
+Passing some types of parameters via ``--global-option`` to setuptools PEP 517/PEP 660 backend
+is now considered deprecated. The user can pass the same arbitrary parameter
+via ``--build-option`` (``--global-option`` is now reserved for flags like
+``--verbose`` or ``--quiet``).
+
+Both ``--build-option`` and ``--global-option`` are supported as a **transitional** effort (a.k.a. "escape hatch").
+In the future a proper list of allowed ``config_settings`` may be created.

changelog.d/3392.change.rst

@@ -0,0 +1,5 @@
+Exposed ``get_output_mapping()`` from ``build_py`` and ``build_ext``
+subcommands. This interface is reserved for the use of ``setuptools``
+Extensions and third part packages are explicitly disallowed to calling it.
+However, any implementation overwriting ``build_py`` or ``build_ext`` are
+required to honour this interface.

changelog.d/3412.change.rst

@@ -0,0 +1,3 @@
+Added ability of collecting source files from custom build sub-commands to
+``sdist``. This allows plugins and customization scripts to automatically
+add required source files in the source distribution.

changelog.d/3414.change.rst

@@ -0,0 +1,4 @@
+Users can *temporarily* specify an environment variable
+``SETUPTOOLS_ENABLE_FEATURE=legacy-editable`` as a escape hatch for the
+:pep:`660` behavior. This setting is **transitional** and may be removed in the
+future.

changelog.d/3414.doc.rst

@@ -0,0 +1,2 @@
+Updated :doc:`Development Mode </userguide/development_mode>` to reflect on the
+implementation of :pep:`660`.

changelog.d/3484.change.rst

@@ -0,0 +1,6 @@
+Added *transient* ``compat`` mode to editable installs.
+This more will be temporarily available (to facilitate the transition period)
+for those that want to emulate the behavior of the ``develop`` command
+(in terms of what is added to ``sys.path``).
+This mode is provided "as is", with limited support, and will be removed in
+future versions of ``setuptools``.

docs/userguide/development_mode.rst

@@ -1,34 +1,260 @@
-Development Mode
-================
-
-Under normal circumstances, the ``setuptools`` assume that you are going to
-build a distribution of your project, not use it in its "raw" or "unbuilt"
-form.  However, if you were to use the ``setuptools`` to build a distribution,
-you would have to rebuild and reinstall your project every time you made a
-change to it during development.
-
-Another problem that sometimes comes is that you may
-need to do development on two related projects at the same time.  You may need
-to put both projects' packages in the same directory to run them, but need to
-keep them separate for revision control purposes.  How can you do this?
-
-Setuptools allows you to deploy your projects for use in a common directory or
-staging area, but without copying any files.  Thus, you can edit each project's
-code in its checkout directory, and only need to run build commands when you
-change files that need to be compiled or the provided metadata and setuptools configuration.
-
-You can perform a ``pip`` installation passing the ``-e/--editable``
-flag (e.g., ``pip install -e .``). It works very similarly to
-``pip install .``, except that it doesn't actually install anything.
-Instead, it creates a special ``.egg-link`` file in the target directory
-(usually ``site-packages``) that links to your project's source code.
-It may also update an existing ``easy-install.pth`` file
-to include your project's source code, thereby making
-it available on ``sys.path`` for all programs using that Python installation.
-
-You can deploy the same project to multiple staging areas, e.g., if you have
-multiple projects on the same machine that are sharing the same project you're
-doing development work.
+Development Mode (a.k.a. "Editable Installs")
+=============================================
+
+When creating a Python project, developers usually want to implement and test
+changes iteratively, before cutting a release and preparing a distribution archive.
+
+In normal circumstances this can be quite cumbersome and require the developers
+to manipulate the ``PATHONPATH`` environment variable or to continuous re-build
+and re-install the project.
+
+To facilitate iterative exploration and experimentation, setuptools allows
+users to instruct the Python interpreter and its import machinery to load the
+code under development directly from the project folder without having to
+copy the files to a different location in the disk.
+This means that changes in the Python source code can immediately take place
+without requiring a new installation.
+
+You can enter this "development mode" by performing an :doc:`editable installation
+<pip:topics/local-project-installs>` inside of a :term:`virtual environment`,
+using :doc:`pip's <pip:cli/pip_install>` ``-e/--editable`` flag, as shown bellow:
+
+.. code-block:: bash
+
+   $ cd your-python-project
+   $ python -m venv .venv
+   # Activate your environemt with:
+   #      `source .venv/bin/activate` on Unix/macOS
+   # or   `.venv\Scripts\activate` on Windows
+
+   $ pip install --editable .
+
+   # Now you have access to your package
+   # as if it was installed in .venv
+   $ python -c "import your_python_project"
+
+
+An "editable installation" works very similarly to a regular install with
+``pip install .``, except that it only installs your package dependencies,
+metadata and wrappers for :ref:`console and GUI scripts <console-scripts>`.
+Under the hood, setuptools will try to create a special :mod:`.pth file <site>`
+in the target directory (usually ``site-packages``) that extends the
+``PYTHONPATH`` or install a custom :doc:`import hook <python:reference/import>`.
 
 When you're done with a given development task, you can simply uninstall your
 package (as you would normally do with ``pip uninstall <package name>``).
+
+Please note that, by default an editable install will expose at least all the
+files that would be available in a regular installation. However, depending on
+the file and directory organization in your project, it might also expose
+as a side effect files that would not be normally available.
+This is allowed so you can iteratively create new Python modules.
+Please have a look on the following section if you are looking for a different behaviour.
+
+.. admonition:: Virtual Environments
+
+   You can think about virtual environments as "isolated Python runtime deployments"
+   that allow users to install different sets of libraries and tools without
+   messing with the global behaviour of the system.
+
+   They are a safe way of testing new projects and can be created easily
+   with the :mod:`venv` module from the standard library.
+
+   Please note however that depending on your operating system or distribution,
+   ``venv`` might not come installed by default with Python. For those cases,
+   you might need to use the OS package manager to install it.
+   For example, in Debian/Ubuntu-based systems you can obtain it via:
+
+   .. code-block:: bash
+
+       sudo apt install python3-venv
+
+   Alternatively, you can also try installing :pypi:`virtualenᴠ`.
+   More information is available on the Python Packaging User Guide on
+   :doc:`PyPUG:guides/installing-using-pip-and-virtual-environments`.
+
+.. note::
+    .. versionchanged:: v64.0.0
+       Editable installation hooks implemented according to :pep:`660`.
+       Support for :pep:`namespace packages <420>` is still **EXPERIMENTAL**.
+
+
+"Strict" editable installs
+--------------------------
+
+When thinking about editable installations, users might have the following
+expectations:
+
+1. It should allow developers to add new files (or split/rename existing ones)
+   and have them automatically exposed.
+2. It should behave as close as possible to a regular installation and help
+   users to detect problems (e.g. new files not being included in the distribution).
+
+Unfortunately these expectations are in conflict with each other.
+To solve this problem ``setuptools`` allows developers to choose a more
+*"strict"* mode for the editable installation. This can be done by passing
+a special *configuration setting* via :pypi:`pip`, as indicated bellow:
+
+.. code-block:: bash
+
+    pip install -e . --config-settings editable_mode=strict
+
+In this mode, new files **won't** be exposed and the editable installs will
+try to mimic as much as possible the behavior of a regular install.
+Under the hood, ``setuptools`` will create a tree of file links in an auxiliary
+directory (``$your_project_dir/build``) and add it to ``PYTHONPATH`` via a
+:mod:`.pth file <site>`. (Please be careful to not delete this repository
+by mistake otherwise your files may stop being accessible).
+
+.. warning::
+   Strict editable installs require auxiliary files to be placed in a
+   ``build/__editable__.*`` directory (relative to your project root).
+
+   Please be careful to not remove this directory while testing your project,
+   otherwise your editable installation may be compromised.
+
+   You can remove the ``build/__editable__.*`` directory after uninstalling.
+
+
+.. note::
+    .. versionadded:: v64.0.0
+       Added new *strict* mode for editable installations.
+       The exact details of how this mode is implemented may vary.
+
+
+Limitations
+-----------
+
+- The *editable* term is used to refer only to Python modules
+  inside the package directories. Non-Python files, external (data) files,
+  executable script files, binary extensions, headers and metadata may be
+  exposed as a *snapshot* of the version they were at the moment of the
+  installation.
+- Adding new dependencies, entry-points or changing your project's metadata
+  require a fresh "editable" re-installation.
+- Console scripts and GUI scripts **MUST** be specified via :doc:`entry-points
+  </userguide/entry_point>` to work properly.
+- *Strict* editable installs require the file system to support
+  either :wiki:`symbolic <symbolic link>` or :wiki:`hard links <hard link>`.
+  This installation mode might also generate auxiliary files under the project directory.
+- There is *no guarantee* that the editable installation will be performed
+  using a specific technique. Depending on each project, ``setuptools`` may
+  select a different approach to ensure the package is importable at runtime.
+- There is *no guarantee* that files outside the top-level package directory
+  will be accessible after an editable install.
+- There is *no guarantee* that attributes like ``__path__`` or ``__file__``
+  will correspond to the exact location of the original files (e.g.,
+  ``setuptools`` might employ file links to perform the editable installation).
+  Users are encouraged to use tools like :mod:`importlib.resources` or
+  :mod:`importlib.metadata` when trying to access package files directly.
+- Editable installations may not work with
+  :doc:`namespaces created with pkgutil or pkg_resouces
+  <PyPUG:guides/packaging-namespace-packages>`.
+  Please use :pep:`420`-style implicit namespaces [#namespaces]_.
+- Support for :pep:`420`-style implicit namespace packages for
+  projects structured using :ref:`flat-layout` is still **experimental**.
+  If you experience problems, you can try converting your package structure
+  to the :ref:`src-layout`.
+
+.. attention::
+   Editable installs are **not a perfect replacement for regular installs**
+   in a test environment. When in doubt, please test your projects as
+   installed via a regular wheel. There are tools in the Python ecosystem,
+   like :pypi:`tox` or :pypi:`nox`, that can help you with that
+   (when used with appropriate configuration).
+
+
+Legacy Behavior
+---------------
+
+If your project is not compatible with the new "editable installs" or you wish
+to replicate the legacy behavior, for the time being you can also perform the
+installation in the ``compat`` mode:
+
+.. code-block:: bash
+
+    pip install -e . --config-settings editable_mode=compat
+
+This installation mode will try to emulate how ``python setup.py develop``
+works (still within the context of :pep:`660`).
+
+.. warning::
+   The ``compat`` mode is *transitional* and will be removed in
+   future versions of ``setuptools``, it exists only to help during the
+   migration period.
+   Also note that support for this mode is limited:
+   it is safe to assume that the ``compat`` mode is offered "as is", and
+   improvements are unlikely to be implemented.
+   Users are encouraged to try out the new editable installation techniques
+   and make the necessary adaptations.
+
+If the ``compat`` mode does not work for you, you can also disable the
+:pep:`editable install <660>` hooks in ``setuptools`` by setting an environment
+variable:
+
+.. code-block::
+
+   SETUPTOOLS_USE_FEATURE="legacy-editable"
+
+This *may* cause the installer (e.g. ``pip``) to effectively run the "legacy"
+installation command: ``python setup.py develop`` [#installer]_.
+
+
+How editable installations work?
+--------------------------------
+
+*Advanced topic*
+
+There are many techniques that can be used to expose packages under development
+in such a way that they are available as if they were installed.
+Depending on the project file structure and the selected mode, ``setuptools``
+will choose one of these approaches for the editable installation [#criteria]_.
+
+A non-exhaustive list of implementation mechanisms is presented below.
+More information is available on the text of :pep:`PEP 660 <660#what-to-put-in-the-wheel>`.
+
+- A static ``.pth`` file [#static_pth]_ can be added to one of the directories
+  listed in :func:`site.getsitepackages` or :func:`site.getusersitepackages` to
+  extend :obj:`sys.path`.
+- A directory containing a *farm of file links* that mimic the
+  project structure and point to the original files can be employed.
+  This directory can then be added to :obj:`sys.path` using a static ``.pth`` file.
+- A dynamic ``.pth`` file [#dynamic_pth]_ can also be used to install an
+  "import :term:`finder`" (:obj:`~importlib.abc.MetaPathFinder` or
+  :obj:`~importlib.abc.PathEntryFinder`) that will hook into Python's
+  :doc:`import system <python:reference/import>` machinery.
+
+.. attention::
+   ``Setuptools`` offers **no guarantee** of which technique will be used to
+   perform an editable installation. This will vary from project to project
+   and may change depending on the specific version of ``setuptools`` being
+   used.
+
+
+----
+
+.. rubric:: Notes
+
+.. [#namespaces]
+   You *may* be able to use *strict* editable installations with namespace
+   packages created with ``pkgutil`` or ``pkg_namespaces``, however this is not
+   officially supported.
+
+.. [#installer]
+   For this workaround to work, the installer tool needs to support legacy
+   editable installations. (Future versions of ``pip``, for example, may drop
+   support for this feature).
+
+.. [#criteria]
+   ``setuptools`` strives to find a balance between allowing the user to see
+   the effects of project files being edited while still trying to keep the
+   editable installation as similar as possible to a regular installation.
+
+.. [#static_pth]
+   i.e., a ``.pth`` file where each line correspond to a path that should be
+   added to :obj:`sys.path`. See :mod:`Site-specific configuration hook <site>`.
+
+.. [#dynamic_pth]
+   i.e., a ``.pth`` file that starts where each line starts with an ``import``
+   statement and executes arbitrary Python code. See :mod:`Site-specific
+   configuration hook <site>`.