Skip to content

Support Direct URL editable requirements #13495

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions docs/html/cli/pip_install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@ for the name and project version (this is in theory slightly less reliable
than using the ``egg_info`` command, but avoids downloading and processing
unnecessary numbers of files).

Any URL may use the ``#egg=name`` syntax (see :doc:`../topics/vcs-support`) to
explicitly state the project name.
The :pep:`508` requirement syntax can be used to explicitly state the project
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hate that I'm adding more references to individual PEPs in the user documentation, but I legitimately don't know where to find a good user guide for Direct URL requirements. Linking to the specification is not exactly user friendly. I defaulted to linking the PEP as this document already does that.

Suggestions would be welcome.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

name (see :doc:`../topics/vcs-support`).

Satisfying Requirements
-----------------------
Expand Down Expand Up @@ -367,21 +367,21 @@ Examples

.. code-block:: shell

python -m pip install -e 'git+https://git.repo/some_pkg.git#egg=SomePackage' # from git
python -m pip install -e 'hg+https://hg.repo/some_pkg.git#egg=SomePackage' # from mercurial
python -m pip install -e 'svn+svn://svn.repo/some_pkg/trunk/#egg=SomePackage' # from svn
python -m pip install -e 'git+https://git.repo/some_pkg.git@feature#egg=SomePackage' # from 'feature' branch
python -m pip install -e 'git+https://git.repo/some_repo.git#egg=subdir&subdirectory=subdir_path' # install a python package from a repo subdirectory
python -m pip install -e 'SomePackage @ git+https://git.repo/some_pkg.git' # from git
python -m pip install -e 'SomePackage @ hg+https://hg.repo/some_pkg.git' # from mercurial
python -m pip install -e 'SomePakcage @ svn+svn://svn.repo/some_pkg/trunk/' # from svn
python -m pip install -e 'SomePackage @ git+https://git.repo/some_pkg.git@feature' # from 'feature' branch
python -m pip install -e 'subdir @ git+https://git.repo/some_repo.git#subdirectory=subdir_path' # install a python package from a repo subdirectory

.. tab:: Windows

.. code-block:: shell

py -m pip install -e "git+https://git.repo/some_pkg.git#egg=SomePackage" # from git
py -m pip install -e "hg+https://hg.repo/some_pkg.git#egg=SomePackage" # from mercurial
py -m pip install -e "svn+svn://svn.repo/some_pkg/trunk/#egg=SomePackage" # from svn
py -m pip install -e "git+https://git.repo/some_pkg.git@feature#egg=SomePackage" # from 'feature' branch
py -m pip install -e "git+https://git.repo/some_repo.git#egg=subdir&subdirectory=subdir_path" # install a python package from a repo subdirectory
py -m pip install -e "SomePackage @ git+https://git.repo/some_pkg.git" # from git
py -m pip install -e "SomePackage @ hg+https://hg.repo/some_pkg.git" # from mercurial
py -m pip install -e "SomePackage @ svn+svn://svn.repo/some_pkg/trunk/" # from svn
py -m pip install -e "SomePackage @ git+https://git.repo/some_pkg.git@feature" # from 'feature' branch
py -m pip install -e "subdir @ git+https://git.repo/some_repo.git#subdirectory=subdir_path" # install a python package from a repo subdirectory

#. Install a package with extras, i.e., optional dependencies
(:ref:`specification <pypug:dependency-specifiers>`).
Expand Down
38 changes: 30 additions & 8 deletions docs/html/topics/vcs-support.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,25 @@ control system being used). It is used through URL prefixes:
- Subversion -- `svn+`
- Bazaar -- `bzr+`

The general form of a VCS requirement is `ProjectName @ VCS_URL`, e.g.

```none
MyProject @ git+https://git.example.com/MyProject
MyProject[extra] @ git+https:/git.example.com/MyProject
```

This is the Direct URL ({pep}`508`) requirement syntax. It is also permissible
to remove `MyProject @` portion is removed and provide a bare VCS URL.

```none
git+https://git.example.com/MyProject
```

This is a pip specific extension. This form can be used as long as pip does
not need to know the project name in advance. pip is generally able to infer
the project name except in the case of {ref}`editable-vcs-installs`. In
addition, extras cannot be requested using a bare VCS URL.

## Supported VCS

### Git
Expand Down Expand Up @@ -81,8 +100,8 @@ MyProject @ svn+ssh://[email protected]/MyProject
You can also give specific revisions to an SVN URL, like so:

```none
-e svn+http://svn.example.com/svn/MyProject/trunk@2019#egg=MyProject
-e svn+http://svn.example.com/svn/MyProject/trunk@{20080101}#egg=MyProject
-e MyProject @ svn+http://svn.example.com/svn/MyProject/trunk@2019
-e MyProject @ svn+http://svn.example.com/svn/MyProject/trunk@{20080101}
```

Note that you need to use [Editable VCS installs](#editable-vcs-installs) for
Expand Down Expand Up @@ -115,6 +134,9 @@ MyProject @ bzr+http://bzr.example.com/MyProject/[email protected]
VCS projects can be installed in {ref}`editable mode <editable-installs>` (using
the {ref}`--editable <install_--editable>` option) or not.

In editable mode, the project name must be provided upfront using the Direct URL
(`MyProject @ URL`) form so pip can determine the VCS clone location.

- The default clone location (for editable installs) is:

- `<venv path>/src/SomeProject` in virtual environments
Expand All @@ -133,15 +155,15 @@ take on the VCS requirement (not the commit itself).
## URL fragments

pip looks at the `subdirectory` fragments of VCS URLs for specifying the path to the
Python package, when it is not in the root of the VCS directory. eg: `pkg_dir`.
Python package, when it is not in the root of the VCS directory.

pip also looks at the `egg` fragment specifying the "project name". In practice the
`egg` fragment is only required to help pip determine the VCS clone location in editable
mode. In all other circumstances, the `egg` fragment is not necessary and its use is
discouraged.
```{note}
pip also supports an `egg` fragment to specify the "project name". This is a legacy
feature and its use is discouraged in favour of the Direct URL ({pep}`508`) form.

The `egg` fragment **should** be a bare {ref}`project name <pypug:name-normalization>`.
Anything else is not guaranteed to work.
```

````{admonition} Example
If your repository layout is:
Expand All @@ -164,6 +186,6 @@ $ pip install "pkg @ vcs+protocol://repo_url/#subdirectory=pkg_dir"
or:

```{pip-cli}
$ pip install -e "vcs+protocol://repo_url/#egg=pkg&subdirectory=pkg_dir"
$ pip install -e "pkg @ vcs+protocol://repo_url/#subdirectory=pkg_dir"
```
````
2 changes: 1 addition & 1 deletion docs/html/user_guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,7 @@ In practice, there are 4 common uses of Requirements files:
``sometag``. You'd reference it in your requirements file with a line like
so::

git+https://myvcs.com/some_dependency@sometag#egg=SomeDependency
SomeDependency @ git+https://myvcs.com/some_dependency@sometag

If ``SomeDependency`` was previously a top-level requirement in your
requirements file, then **replace** that line with the new line. If
Expand Down
1 change: 1 addition & 0 deletions news/13495.feature.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add support installing an editable requirement written as a Direct URL.
56 changes: 41 additions & 15 deletions src/pip/_internal/req/constructors.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,17 +86,25 @@ def _set_requirement_extras(req: Requirement, new_extras: set[str]) -> Requireme
return get_requirement(f"{pre}{extras}{post}")


def parse_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
"""Parses an editable requirement into:
- a requirement name
- an URL
- extras
- editable options
Accepted requirements:
svn+http://blahblah@rev#egg=Foobar[baz]&subdirectory=version_subdir
.[some_extra]
"""
def _parse_direct_url_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
try:
req = Requirement(editable_req)
except InvalidRequirement:
pass
else:
if req.url:
# Join the marker back into the name part. This will be parsed out
# later into a Requirement again.
if req.marker:
name = f"{req.name} ; {req.marker}"
else:
name = req.name
return (name, req.url, req.extras)

raise ValueError


def _parse_pip_syntax_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
url = editable_req

# If a file path is specified with extras, strip off the extras.
Expand All @@ -122,23 +130,41 @@ def parse_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
url = f"{version_control}+{url}"
break

return Link(url).egg_fragment, url, set()


def parse_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
"""Parses an editable requirement into:
- a requirement name
- an URL
- extras
Accepted requirements:
- svn+http://blahblah@rev#egg=Foobar[baz]&subdirectory=version_subdir
- local_path[some_extra]
- Foobar[extra] @ svn+http://blahblah@rev#subdirectory=subdir ; markers
"""
try:
package_name, url, extras = _parse_direct_url_editable(editable_req)
except ValueError:
package_name, url, extras = _parse_pip_syntax_editable(editable_req)

link = Link(url)

if not link.is_vcs:
if not link.is_vcs and not link.url.startswith("file:"):
backends = ", ".join(vcs.all_schemes)
raise InstallationError(
f"{editable_req} is not a valid editable requirement. "
f"It should either be a path to a local project or a VCS URL "
f"(beginning with {backends})."
)

package_name = link.egg_fragment
if not package_name:
# The project name can be inferred from local file URIs easily.
if not package_name and not link.url.startswith("file:"):
raise InstallationError(
f"Could not detect requirement name for '{editable_req}', "
"please specify one with #egg=your_package_name"
"please specify one with your_package_name @ URL"
)
return package_name, url, set()
return package_name, url, extras


def check_first_requirement_in_file(filename: str) -> None:
Expand Down
6 changes: 5 additions & 1 deletion src/pip/_internal/resolution/resolvelib/candidates.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,12 @@ def make_install_req_from_editable(
link: Link, template: InstallRequirement
) -> InstallRequirement:
assert template.editable, "template not editable"
if template.name:
req_string = f"{template.name} @ {link.url}"
else:
req_string = link.url
ireq = install_req_from_editable(
link.url,
req_string,
user_supplied=template.user_supplied,
comes_from=template.comes_from,
use_pep517=template.use_pep517,
Expand Down
50 changes: 50 additions & 0 deletions tests/functional/test_install_reqs.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
PipTestEnvironment,
ResolverVariant,
TestData,
_create_test_package,
_create_test_package_with_subdirectory,
create_basic_sdist_for_package,
create_basic_wheel_for_package,
Expand Down Expand Up @@ -941,3 +942,52 @@ def test_nonpep517_setuptools_import_failure(script: PipTestEnvironment) -> None
exc_message = "ImportError: this 'setuptools' was intentionally poisoned"
assert nice_message in result.stderr
assert exc_message in result.stderr


class TestEditableDirectURL:
def test_install_local_project(
self, script: PipTestEnvironment, data: TestData, common_wheels: Path
) -> None:
uri = (data.src / "simplewheel-2.0").as_uri()
script.pip(
"install", "--no-index", "-e", f"simplewheel @ {uri}", "-f", common_wheels
)
script.assert_installed(simplewheel="2.0")

def test_install_local_project_with_extra(
self, script: PipTestEnvironment, data: TestData, common_wheels: Path
) -> None:
uri = (data.src / "requires_simple_extra").as_uri()
script.pip(
"install",
"--no-index",
"-e",
f"requires-simple-extra[extra] @ {uri}",
"-f",
common_wheels,
"-f",
data.packages,
)
script.assert_installed(requires_simple_extra="0.1")
script.assert_installed(simple="1.0")

def test_install_local_git_repo(
self, script: PipTestEnvironment, common_wheels: Path
) -> None:
repo_path = _create_test_package(script.scratch_path, "simple")
url = "git+" + repo_path.as_uri()
script.pip(
"install", "--no-index", "-e", f"simple @ {url}", "-f", common_wheels
)
script.assert_installed(simple="0.1")

@pytest.mark.network
def test_install_remote_git_repo_with_extra(
self, script: PipTestEnvironment, data: TestData, common_wheels: Path
) -> None:
req = "pip-test-package[extra] @ git+https://github.com/pypa/pip-test-package"
script.pip(
"install", "--no-index", "-e", req, "-f", common_wheels, "-f", data.packages
)
script.assert_installed(pip_test_package="0.1.1")
script.assert_installed(simple="3.0")
19 changes: 19 additions & 0 deletions tests/unit/test_req.py
Original file line number Diff line number Diff line change
Expand Up @@ -853,6 +853,25 @@ def test_install_req_extend_extras(
assert extended.permit_editable_wheels == req.permit_editable_wheels


@pytest.mark.parametrize(
"req_str, expected",
[
(
'foo[extra] @ svn+http://foo ; os_name == "nt"',
('foo ; os_name == "nt"', "svn+http://foo", {"extra"}),
),
(
"foo @ svn+http://foo",
("foo", "svn+http://foo", set()),
),
],
)
def test_parse_editable_pep508(
req_str: str, expected: tuple[str, str, set[str]]
) -> None:
assert parse_editable(req_str) == expected


@mock.patch("pip._internal.req.req_install.os.path.abspath")
@mock.patch("pip._internal.req.req_install.os.path.exists")
@mock.patch("pip._internal.req.req_install.os.path.isdir")
Expand Down
Loading