[pypi] Support pulling dependencies using direct url specifiers without using an index

[mering]

🐞 bug report

Affected Rule

Module extension:

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(...)

Is this a regression?

Not sure.

Description

The repository rule for pip dependencies doesn't use repository_ctx.download(). This breaks usages of --experimental_downloader_config for example to use a transparent dependency mirror.

🔬 Minimal Reproduction

https://github.com/bazelbuild/rules_python/blob/b6fc2a080d4884510dea02cc77b4e0b8fd3a0ccd/python/private/pypi/whl_library.bzl#L245-L256

https://github.com/bazelbuild/rules_python/blob/b6fc2a080d4884510dea02cc77b4e0b8fd3a0ccd/python/private/pypi/whl_installer/wheel_installer.py#L168-L190

🔥 Exception or Error

Stack trace prints to the code linked above.

🌍 Your Environment

Operating System:

Linux

Output of bazel version:

7.0.2

Rules_python version:

0.37.2

Anything else relevant?

[aignas]

Please see the documentation here: https://rules-python.readthedocs.io/en/latest/pypi-dependencies.html#bazel-downloader-and-multi-platform-wheel-hub-repository.

This currently has a limitation that it needs hashes for all of the files in the requirements.txt. The stabilization of this feature is tracked by #260.

[mering]

@aignas How can I use experimental_index_url in combination with dependencies without an index like the following:

proto-breaking-change-detector @ https://github.com/googleapis/proto-breaking-change-detector/archive/refs/tags/v2.4.0.tar.gz
tensorflow_graphics @ git+https://github.com/tensorflow/graphics@ca63cf788978db0bde2db4ce488e102c455da2e7

[aignas]

The support for direct references has not been added yet, I'll reopen this issue and rename it to scope it to supporting it, but I am not sure when I can work on it.

proto-breaking-change-detector @ https://github.com/googleapis/proto-breaking-change-detector/archive/refs/tags/v2.4.0.tar.gz could be easy enough. Outline of the changes:

• Update the https://github.com/bazelbuild/rules_python/blob/main/python/private/pypi/index_sources.bzl file to support extracting URLs from the requirements file
• Change the code that calls the index to not call it for requirement lines that refer to URLs
• Pipe the url and the filename to the whl_library respective attributes.

For the second one it may be harder - I am not sure how to handle git repos yet, however, with bazel 7.4 the extension can pass a labels within the extension more easily, so one could look into how to do that.

For now the workaround would be to build the wheels from sdists outside bazel and host them in some PyPI compatible artifact mirror and then use experimental_index_url flag to enable the usage of the bazel-downloader.