Changing pip's build system to use `build`

[pradyunsg]

I'd like to trigger the discussion for how we'd get to pip using build internally, to orchestrate the pyproject.toml-based builds. The first obvious question is: Are the build maintainers interested in pip moving to doing something like this?

Assuming the answer is yes (if it's no, then please go ahead and close this issue), the things that need to be figured out for this are:

1. Environment management: pip will likely need to create it's own subclass, that determines how pip will get installed into the environment. We might actually move to using an import hook for making pip available in the build environment?

2. Fork-bomb protection: build currently uses pip to install packages in the build environments, which means it is protected from a recursive build graph by pip's implementation of that protection (search for "build tracker" in pip's codebase). We'd need some way to recreate these protections when we switch to using build internally to pip.

3. The importlib-metadata dependency: pip currently does not vendor importlib_metadata and I'm wary of adding more dependencies to pip's dependency graph. This dependency is only there for a utility function, that pip won't actually use -- can we move that to be an optional dependency as an extra?

[pradyunsg]

I'm not sure if there's more stuff here, but these three seemed like a good starting point for the discussion. :)

[layday]

A PR which genericised the isolated env class and made importlib-metadata optional for the exact purpose of making build re-usable by pip can be found here: #361. I'd need to know about how fork-bomb protection operates before I can comment on that one. What does pip need from build to be able to track in-progress builds?

[pradyunsg]

What does pip need from build to be able to track in-progress builds?

I'm looking at the code now (as part of pypa/pip#10989) and I don't think there's anything necessary for this.

IIUC, the build tracking logic lives outside of the actual build invocation, so we might not need anything from build on that front. (there's only for one place where we call tracker.track in pip's codebase, and that's a few layers above the build system invocations).

[henryiii]

can we move that to be an optional dependency as an extra?

Pip would vendor build, rather than installing it, so what build declares shouldn't matter to pip, right? We'd just need to make sure it works without that dependency. Since we don't have "negative" dependencies, I'm thinking about prosing that virtualenv be added as a required dependency, but we support having it uninstalled. Build is faster with virtualenv over venv, but venv is useful for bootstrapping (and maybe inside pip?).

This will be something we can work on after the next release (#468).

[uranusjr]

Or we could simply patch the vendored build in pip to never use virtualenv (but only venv), if build can make that simple.

[gaborbernat]

@pradyunsg is this still a plan, anything else to discuss?