pip_parse is burdensome for downstream projects

[pauldraper]

Typically, to install an upstream project, it looks something like

http_repository(
  name = "example",
  # ...
)

load("@example//:workspace.bzl", "example_repositories")
example_repositories()

However, suppose example uses pip dependencies.

The install must be at minimum:

http_repository(
  name = "example",
  # ...
)

# create pip_parse repo
load("@example://workspace_part_1.bzl", "example_repositories_part_1")
example_repositories_part_1()

# load requirements.bzl from pip_parse repo and call install_deps()
load("@example//:workspace_part_2.bzl", "example_repositories_part_2")
example_repositories_part_2()

This looks weird, and further burdens already lengthy WORKSPACE files.

---

A solution is to expose generating requirements.bzl. Then the upstream project can commit that to the source tree (probably the same step as generating the requirements lock file today). And then downstream projects can initialize repos with only one step.

This allows the repository creation to happen in one step.

[hrfuller]

Thanks for the report @pauldraper. I'm open to the solution you proposed of generating and committing a requirements.bzl file. However I'd like to understand if a solution like this macro could work to hide the creation of a dependency on rules_python?

[pauldraper]

@hrfuller that definitely could.

Whatever you used to generate _RULE_DEPS and the rest of that is exactly what I'm looking for.

(If you did that by hand, then automating that is what I'm looking for.)

[hrfuller]

_RULE_DEPS was not generated to the best of my knowledge. What I was imagining to tidy up the WORKSPACE file is something like:

http_archive(
   name = "rules_python",
   ...
)

load("@rules_python//:macros.bzl", "create_pip_parse_repo")
create_pip_parse_repo(
   name = "pip_parse_custom",
   extra_pip_args = ["--index-url", ...]
   requirements_lock = "//path/to:requirements_lock.txt"
)

Would that work? I think its a better solution than committing a generated requirements.bzl file, which would have to be manually regenerated and commited anytime you want to change the python requirements available from requirements.bzl

[pauldraper]

You need a second step there.

You haven't actually created any @pip_boto3 etc repos.

You'll need another load() and those can't existing in Starlark function bodies.

I think its a better solution than committing a generated requirements.bzl file, which would have to be manually regenerated and commited anytime you want to change the python requirements available from requirements.bzl

I understand....but that is how lock files normally work.

requirements.bzl is a lockfile, with URLs, digests, etc.

[alexeagle]

Yes and checking in the generated .bzl is how it works in Go as well, https://github.com/bazelbuild/bazel-gazelle#update-repos

[alexeagle]

Also the two-step execution makes this a lot harder to model as a Module Extension in bzlmod. I had a similar setup for JS and had a lot of trouble which I documented on bazelbuild/bazel#14445
I'm pretty sure that's resolvable, but I didn't actually get back to that to confirm yet.

Let's add a mode here where the generator can be run as a tool that writes to a .bzl file in the workspace.

[hrfuller]

You'll need another load() and those can't existing in Starlark function bodies.
Ah yeah good point. That changes things.