Optimize dependency iteration

[killerdevildog]

…cies

PROBLEM IDENTIFIED:
Before this optimization, dependencies were being parsed twice during candidate resolution:

1. During _check_metadata_consistency() for validation (line 233 in original code)
2. During iter_dependencies() for actual dependency resolution (line 258)

This caused significant performance issues because:

• dist.iter_provided_extras() was called multiple times
• dist.iter_dependencies() was called multiple times
• Parsing requirements from package metadata is computationally expensive
• The TODO comment at line 230 specifically noted this performance problem

SOLUTION IMPLEMENTED:
Added caching mechanism with two new instance variables:

• _cached_dependencies: stores list[Requirement] after parsing once
• _cached_extras: stores list[NormalizedName] after parsing once

HOW THE CACHING WORKS:

1. Cache variables are initialized as None in init()
2. During _prepare() -> _check_metadata_consistency(), dependencies are parsed and cached during validation
3. During iter_dependencies(), the cached results are reused via _get_cached_dependencies()
4. Cache is populated lazily - only when first accessed
5. Subsequent calls to iter_dependencies() use cached data (no re-parsing)
6. Each candidate instance has its own cache (thread-safe)

ADDITIONAL OPTIMIZATIONS:

• Also optimized ExtrasCandidate.iter_dependencies() to cache iter_provided_extras() results
• Ensures consistency between validation and dependency resolution phases

TESTING PERFORMED:

1. Created comprehensive test script (test_performance_optimization.py)
2. Used mock objects to verify iter_provided_extras() and iter_dependencies() are called at most once
3. Verified pip install --dry-run works correctly with caching
4. Test results showed 0 additional calls to parsing methods during multiple iter_dependencies() invocations
5. Functional testing confirmed dependency resolution still works correctly

PERFORMANCE IMPACT:

• Eliminates duplicate parsing during metadata consistency checks
• Reduces CPU time for packages with complex dependency trees
• Especially beneficial for packages with many dependencies
• Memory overhead is minimal (only stores parsed results, not raw metadata)

Resolves TODO comment about performance in candidates.py line 230

[notatallshaw]

Thanks for your PR to pip. Please be aware all maintainers are volunteers so it may take a moment for someone to review.

Let me know if you need any help fixing the the linting and pre-commit errors, you should be able to locally run them following this guide: https://pip.pypa.io/en/stable/development/getting-started/#running-linters

Also, are there real world scenarios that inspired you to fix this? Or is this more of an academic exercise?

[ichard26]

@killerdevildog while I understand there was a comment mentioning this inefficiency, could you please provide a demonstration of the speed up achieved by this optimization? The example can be a bit contrived, but I do want to see actual numbers before adding further complexity. Thank you!

[killerdevildog]

@ichard26 if you run the provided test, it shows the improvement, this is the results on my Ubuntu 24.04 installation
the test file can be deleted before merge, just there to show the performance increase.

=== Dependency Caching Performance Test Results ===
Number of iter_dependencies() calls: 50
Old approach (no caching): 0.0812 seconds
New approach (with caching): 0.0017 seconds
Time saved: 0.0795 seconds
Speedup: 48.32x
Performance improvement: 97.9%
=======================================================
All tests passed!

Another test with 10k iterations show it as well

=== Dependency Caching Performance Test Results ===
Number of iter_dependencies() calls: 10000
Old approach (no caching): 16.2887 seconds
New approach (with caching): 0.0044 seconds
Time saved: 16.2843 seconds
Speedup: 3729.65x
Performance improvement: 100.0%
=======================================================
All tests passed!

[notatallshaw]

@killerdevildog thanks for the bench marking, bench mark tests should not be included in the final PR, you can remove them from the branch and we can use the git history to look at them if we need another copy.

I will put this PR on my list to review, as I am always excited to see speed ups in resolution, please be aware it might be at least a few weeks before I get chance to spend time on it.

[notatallshaw]

Let's not discuss this any further, I'm going to mark the preceding comments as off topic and I'm going to take everyone on good faith here, that @timmc is trying to prevent OSS projects from wasting time and @killerdevildog is genuinely trying their best to contribute to the project.

@killerdevildog it would be of some help if you could merge in main, or rebase to latest main, and if tests are still failing check why.

[ofek]

I'm not a maintainer technically but I would strongly encourage you to remove the new meta-path logic (and associated test file change), instead focusing this PR exclusively on optimization. It's bad practice in general to not separate features into their own changes.

[ofek]

This needlessly adds to the diff, I would revert this part.