gh-47798: Add a subprocess.run_pipeline() API

Doc/library/subprocess.rst

@@ -261,6 +261,229 @@ underlying :class:`Popen` interface can be used directly.
         *stdout* and *stderr* attributes added
 
 
+.. function:: run_pipeline(*commands, stdin=None, input=None, \
+                           stdout=None, stderr=None, capture_output=False, \
+                           timeout=None, check=False, encoding=None, \
+                           errors=None, text=None, env=None, \
+                           **other_popen_kwargs)
+
+   Run a pipeline of commands connected via pipes, similar to shell pipelines.
+   Wait for all commands to complete, then return a :class:`CompletedPipeline`
+   instance.
+
+   Each positional argument should be a command (a sequence of strings) to
+   execute. The standard output of each command is connected to the standard
+   input of the next command in the pipeline.
+
+   This function requires at least two commands. For a single command, use
+   :func:`run` instead.
+
+   If *capture_output* is true, the standard output of the final command and
+   the standard error of all commands will be captured. All processes in the
+   pipeline share a single stderr pipe, so their error output will be
+   interleaved. The *stdout* and *stderr* arguments may not be supplied at
+   the same time as *capture_output*.
+
+   A *timeout* may be specified in seconds. If the timeout expires, all
+   child processes will be killed and waited for, and then a
+   :exc:`TimeoutExpired` exception will be raised.
+
+   The *input* argument is passed to the first command's stdin. If used, it
+   must be a byte sequence, or a string if *encoding* or *errors* is specified
+   or *text* is true.
+
+   If *check* is true, and any process in the pipeline exits with a non-zero
+   exit code, a :exc:`PipelineError` exception will be raised. This behavior
+   is similar to the shell's ``pipefail`` option.
+
+   If *encoding* or *errors* are specified, or *text* is true, file objects
+   are opened in text mode using the specified encoding and errors.
+
+   .. note::
+
+      When using ``text=True`` with ``capture_output=True`` or ``stderr=PIPE``,
+      be aware that stderr output from multiple processes may be interleaved
+      in ways that produce incomplete multi-byte character sequences. For
+      reliable text decoding of stderr, consider capturing in binary mode
+      and decoding manually with appropriate error handling, or use
+      ``errors='replace'`` or ``errors='backslashreplace'``.
+
+   .. note::
+
+      Passing ``stderr=STDOUT`` redirects each child process's stderr to its
+      own stdout file descriptor. For non-final processes in the pipeline
+      this means stderr is merged into the next process's stdin, which is
+      rarely what callers want. Only the final process's stderr ends up at
+      the pipeline's stdout destination. To merge stderr-and-stdout output
+      from the whole pipeline, use ``capture_output=True`` (or pass an
+      explicit file descriptor as *stderr*) instead of ``STDOUT``.
+
+   .. note::
+
+      When stderr is captured (via ``capture_output=True`` or
+      ``stderr=PIPE``), every command in the pipeline inherits a copy of the
+      same stderr write file descriptor. If any child spawns a daemon or
+      grandchild that keeps stderr open after the child exits, the parent's
+      read on the stderr pipe will not see EOF and :func:`run_pipeline` will
+      hang. This matches shell ``2>&1 | other`` behavior. If this is a
+      concern, either do not capture stderr, manage each command with an
+      individual :class:`Popen` so each has its own stderr pipe, or ensure
+      grandchildren fully detach (closing inherited fds) before
+      daemonizing.
+
+   If *stdin* is specified, it is connected to the first command's standard
+   input. If *stdout* is specified, it is connected to the last command's
+   standard output. When *stdout* is :data:`PIPE`, the output is available
+   in the returned :class:`CompletedPipeline`'s :attr:`~CompletedPipeline.stdout`
+   attribute. Other keyword arguments are passed to each :class:`Popen` call;
+   in particular *pass_fds* is forwarded to *every* command in the pipeline,
+   so any inheritable file descriptor passed via *pass_fds* is visible to all
+   children (unlike a shell, which can give each command its own fd set).
+   ``close_fds=False`` is rejected because inherited copies of the
+   inter-process pipe ends in sibling children would prevent EOF from being
+   signaled and cause deadlocks. ``shell=True`` and ``executable=`` are also
+   rejected: the pipeline itself replaces the shell, and per-stage shell
+   interpretation would re-introduce the quoting and injection surface this
+   function exists to avoid.
+
+   Examples::
+
+      >>> import subprocess
+      >>> # Equivalent to: echo "hello world" | tr a-z A-Z
+      >>> result = subprocess.run_pipeline(
+      ...     ['echo', 'hello world'],
+      ...     ['tr', 'a-z', 'A-Z'],
+      ...     capture_output=True, text=True
+      ... )
+      >>> result.stdout
+      'HELLO WORLD\n'
+      >>> result.returncodes
+      [0, 0]
+
+      >>> # Pipeline with three commands
+      >>> result = subprocess.run_pipeline(
+      ...     ['echo', 'one\ntwo\nthree'],
+      ...     ['sort'],
+      ...     ['head', '-n', '2'],
+      ...     capture_output=True, text=True
+      ... )
+      >>> result.stdout
+      'one\nthree\n'
+
+      >>> # Using input parameter
+      >>> result = subprocess.run_pipeline(
+      ...     ['cat'],
+      ...     ['wc', '-l'],
+      ...     input='line1\nline2\nline3\n',
+      ...     capture_output=True, text=True
+      ... )
+      >>> result.stdout.strip()
+      '3'
+
+      >>> # Error handling with check=True
+      >>> subprocess.run_pipeline(
+      ...     ['echo', 'hello'],
+      ...     ['false'],  # exits with status 1
+      ...     check=True
+      ... )
+      Traceback (most recent call last):
+        ...
+      subprocess.PipelineError: Pipeline failed: command 1 ['false'] returned 1
+
+   .. versionadded:: next
+
+
+.. class:: CompletedPipeline
+
+   The return value from :func:`run_pipeline`, representing a pipeline of
+   processes that have finished.
+
+   .. attribute:: commands
+
+      The list of commands used to launch the pipeline.
+
+   .. attribute:: returncodes
+
+      List of exit status codes for each command in the pipeline. Typically,
+      an exit status of 0 indicates that the command ran successfully.
+
+      A negative value ``-N`` indicates that the command was terminated by
+      signal ``N`` (POSIX only).
+
+   .. attribute:: returncode
+
+      Exit status of the final command in the pipeline. This is a convenience
+      property equivalent to ``returncodes[-1]``.
+
+      Note that this matches shell behavior *without* ``pipefail``: a zero
+      ``returncode`` does not imply that earlier commands in the pipeline
+      succeeded. To check that all commands succeeded, use
+      :meth:`check_returncodes` or pass ``check=True`` to
+      :func:`run_pipeline`.
+
+   .. attribute:: stdout
+
+      Captured stdout from the final command in the pipeline. A bytes sequence,
+      or a string if :func:`run_pipeline` was called with an encoding, errors,
+      or ``text=True``. ``None`` if stdout was not captured.
+
+   .. attribute:: stderr
+
+      Captured stderr from all commands in the pipeline, combined. A bytes
+      sequence, or a string if :func:`run_pipeline` was called with an
+      encoding, errors, or ``text=True``. ``None`` if stderr was not captured.
+
+   .. method:: check_returncodes()
+
+      If any command's :attr:`returncode` is non-zero, raise a
+      :exc:`PipelineError`.
+
+   .. versionadded:: next
+
+
+.. exception:: PipelineError
+
+   Subclass of :exc:`SubprocessError`, raised when a pipeline run by
+   :func:`run_pipeline` (with ``check=True``) contains one or more commands
+   that returned a non-zero exit status. This is similar to the shell's
+   ``pipefail`` behavior.
+
+   :exc:`PipelineError` is a *sibling* of :exc:`CalledProcessError`, not a
+   subclass: a pipeline carries N commands and N return codes, so there is
+   no single ``returncode`` or ``cmd`` value an existing
+   ``except CalledProcessError:`` handler could be shown without being
+   misleading.  To handle both single-command and pipeline failures with
+   one ``except`` clause, catch :exc:`SubprocessError` (which is also the
+   common base of :exc:`TimeoutExpired`).  Retry helpers and decorators
+   that match on :exc:`CalledProcessError` will not catch pipeline
+   failures by default.
+
+   .. attribute:: commands
+
+      List of commands that were used in the pipeline.
+
+   .. attribute:: returncodes
+
+      List of exit status codes for each command in the pipeline.
+
+   .. attribute:: stdout
+
+      Output of the final command if it was captured. Otherwise, ``None``.
+
+   .. attribute:: stderr
+
+      Combined stderr output of all commands if it was captured.
+      Otherwise, ``None``.
+
+   .. attribute:: failed
+
+      List of ``(index, command, returncode)`` tuples for each command
+      that returned a non-zero exit status. The *index* is the position
+      of the command in the pipeline (0-based).
+
+   .. versionadded:: next
+
+
 .. _frequently-used-arguments:
 
 Frequently Used Arguments
@@ -1393,24 +1616,32 @@ Replacing shell pipeline
 
 becomes::
 
-   p1 = Popen(["dmesg"], stdout=PIPE)
-   p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
-   p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
-   output = p2.communicate()[0]
+   result = run_pipeline(["dmesg"], ["grep", "hda"],
+                         capture_output=True, check=True)
+   output = result.stdout
 
-The ``p1.stdout.close()`` call after starting the p2 is important in order for
-p1 to receive a SIGPIPE if p2 exits before p1.
+:func:`run_pipeline` connects the stages, closes the parent's copies of the
+intermediate pipe ends, waits for every process, and (with ``check=True``)
+raises :exc:`PipelineError` if *any* stage fails -- equivalent to the
+shell's ``set -o pipefail`` without needing a shell.
 
-Alternatively, for trusted input, the shell's own pipeline support may still
-be used directly:
+If you need to read the final stage's output incrementally rather than
+waiting for the whole pipeline to finish (for example, streaming a large
+decompressed file), chain :class:`Popen` instances directly::
 
-.. code-block:: bash
-
-   output=$(dmesg | grep hda)
-
-becomes::
-
-   output = check_output("dmesg | grep hda", shell=True)
+   p1 = Popen(["dmesg"], stdout=PIPE)
+   p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
+   p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
+   for line in p2.stdout:
+       ...
+   p2.wait()
+   p1.wait()
+   if p1.returncode or p2.returncode:
+       ...  # handle failure in either stage
+
+The ``p1.stdout.close()`` call after starting p2 is important in order for
+p1 to receive a SIGPIPE if p2 exits before p1.  Each process must be
+waited on and its return code checked to detect failure in any stage.
 
 
 Replacing :func:`os.system`

Doc/whatsnew/3.15.rst

@@ -1121,6 +1121,17 @@ ssl
 subprocess
 ----------
 
+* Added :func:`subprocess.run_pipeline` for running a sequence of commands
+  connected stdout-to-stdin, similar to a shell ``a | b | c`` pipeline,
+  without invoking a shell.  It returns a
+  :class:`~subprocess.CompletedPipeline` carrying the return codes of every
+  stage; passing ``check=True`` raises :exc:`~subprocess.PipelineError` if
+  any stage fails (pipefail semantics).  This replaces the manual
+  :class:`~subprocess.Popen`-chaining recipe and the
+  ``bash -c "set -o pipefail; ..."`` workaround previously needed to detect
+  failures in non-final stages.
+  (Contributed by Gregory P. Smith in :gh:`47798`.)
+
 * :meth:`subprocess.Popen.wait`: when ``timeout`` is not ``None`` and the
   platform supports it, an efficient event-driven mechanism is used to wait for
   process termination: