Running Shell Commands from Python with subprocess

Maria
Python
20 Sep, 2026

Sooner or later a Python script needs to call another program: run git to find the current commit, invoke ffmpeg to convert a video, call pg_dump for a backup, or kick off a build tool. Python's answer is the subprocess module. It replaced older tools like os.system and os.popen, and it gives you full control over arguments, input, output, exit codes, environment, and timeouts.

The module has a reputation for being fiddly, mostly because it exposes a lot of options. In practice you need one function, subprocess.run(), for nearly everything, plus Popen for the few cases where you need to interact with a process while it's running.

This guide covers running commands and capturing their output, handling failures properly, timeouts, passing input, environment variables and working directories, why shell=True is risky and what to do instead, streaming output, and building pipelines. The examples use Python 3.13.

The Basics: subprocess.run()

import subprocess

result = subprocess.run(["git", "--version"], capture_output=True, text=True)
print(result)
print(result.returncode)
print(result.stdout.strip())

CompletedProcess(args=['git', '--version'], returncode=0, stdout='git version 2.49.0\n', stderr='')
0
git version 2.49.0

Three things are worth understanding right away.

The command is a list. Each element is one argument, passed to the program exactly as written. There's no shell involved, so spaces, quotes, and special characters in an argument need no escaping. This is the safe default and the one you should use almost always.

capture_output=True collects the program's standard output and standard error instead of letting them print to your terminal. Without it, result.stdout and result.stderr are None.

text=True decodes the output into str using your locale's encoding. Without it you get bytes. If you need a specific encoding, pass encoding="utf-8" instead (which also implies text mode), and consider errors="replace" for programs that might emit invalid bytes. The bytes and Unicode guide explains what's going on underneath.

run() waits for the command to finish and returns a CompletedProcess with args, returncode, stdout, and stderr.

stdout, stderr, and Exit Codes

Programs report results through three channels: standard output for normal output, standard error for diagnostics, and an exit code where 0 means success. Here's a child process using all three:

import subprocess
import sys

result = subprocess.run(
    [sys.executable, "-c", "import sys; print('out'); print('err', file=sys.stderr); sys.exit(3)"],
    capture_output=True,
    text=True,
)
print(repr(result.stdout), repr(result.stderr), result.returncode)

'out\n' 'err\n' 3

sys.executable is the path of the Python interpreter running your script. Use it whenever you launch another Python process, rather than hardcoding "python" or "python3", so the child runs in the same environment (including the same virtual environment).

Handling Failure

By default, run() does not raise when a command fails. A non-zero exit code just ends up in returncode, and if you don't check it your script carries on as if everything worked. That's the most common subprocess bug.

Pass check=True to turn a non-zero exit into an exception:

import subprocess
import sys

try:
    subprocess.run(
        [sys.executable, "-c", "import sys; sys.exit('config file missing')"],
        capture_output=True,
        text=True,
        check=True,
    )
except subprocess.CalledProcessError as exc:
    print(f"Command failed with exit code {exc.returncode}")
    print(exc.stderr.strip())

Command failed with exit code 1
config file missing

CalledProcessError carries returncode, cmd, stdout, and stderr, so you can report exactly what went wrong. (Passing a string to sys.exit() prints it to stderr and exits with code 1.)

There are two other failures to know about, and neither is a CalledProcessError:

The program doesn't exist. You get FileNotFoundError before anything runs.
The program takes too long. With timeout=, you get subprocess.TimeoutExpired.

import subprocess

try:
    subprocess.run(["no-such-tool", "--help"])
except FileNotFoundError as exc:
    print(exc)

[Errno 2] No such file or directory: 'no-such-tool'

To check whether a program is available before calling it, use shutil.which(). It returns the full path, or None if the program isn't on PATH:

import shutil

print(shutil.which("git"))           # e.g. /usr/bin/git
print(shutil.which("no-such-tool"))  # None

Timeouts

A command waiting on the network or on input it will never get can hang your script forever. Always set a timeout (in seconds) for anything that could stall:

import subprocess

try:
    subprocess.run(["sleep", "5"], timeout=1)
except subprocess.TimeoutExpired as exc:
    print(f"Timed out: {exc.cmd}")

Timed out: ['sleep', '5']

When the timeout expires, run() kills the child process and waits for it before raising, so you don't leave orphans behind. Any output collected so far is available on the exception as exc.stdout and exc.stderr (as bytes, or None).

Sending Input

Use input= to feed data to the program's standard input. With text=True it's a string; otherwise it must be bytes:

import subprocess

result = subprocess.run(
    ["sort"],
    input="pear\napple\nfig\n",
    capture_output=True,
    text=True,
    check=True,
)
print(result.stdout, end="")

apple
fig
pear

If the command shouldn't read input at all, pass stdin=subprocess.DEVNULL. That's a good habit for scripts run from cron or CI, where an unexpected prompt would otherwise wait forever.

Working Directory and Environment

cwd= runs the command in a different directory without changing your own script's working directory:

import subprocess

subprocess.run(["git", "status", "--short"], cwd="/path/to/repo", check=True)

env= replaces the child's environment entirely. Usually you want to add or override a variable while keeping everything else (especially PATH), so copy os.environ first:

import os
import subprocess
import sys

env = {**os.environ, "APP_ENV": "staging"}
result = subprocess.run(
    [sys.executable, "-c", "import os; print(os.environ['APP_ENV'])"],
    env=env,
    capture_output=True,
    text=True,
    check=True,
)
print(result.stdout, end="")

staging

Passing env={"APP_ENV": "staging"} alone would wipe PATH, HOME, and everything else, which breaks many programs in confusing ways.

Why You Should Avoid shell=True

You can pass a single string with shell=True, and the command runs through /bin/sh (or cmd.exe on Windows):

import subprocess

subprocess.run("ls -l *.py | wc -l", shell=True)

It's tempting because it lets you use pipes, wildcards, &&, and redirection exactly as you would in a terminal. The problem is that the string is interpreted by a shell, so any value you insert into it is interpreted too. Consider a script that prints a user-supplied file:

filename = "report; rm -rf ~"
subprocess.run(f"cat {filename}", shell=True)  # runs: cat report; rm -rf ~

That's a shell injection vulnerability, and it applies whenever any part of the command comes from user input, a filename, a web request, or a config file you don't fully control. The list form has no such problem: ["cat", filename] passes the whole string as one argument to cat, semicolons and all.

Other downsides of shell=True:

Behavior depends on the platform's shell, so commands often aren't portable.
Exit codes come from the shell, which can mask failures in the middle of a pipeline.
It starts an extra process.

Tools for When You Need Shell-Like Behavior

The standard library covers most of what people reach for a shell to do:

Shell feature	Python alternative
`*.py` wildcards	`pathlib.Path.glob("*.py")` or the `glob` module
`> out.txt`	`stdout=open(...)`, or capture and write the file
Pipes between commands	Two `Popen` objects (shown below) or `input=`
`cd dir && cmd`	`cwd="dir"`
`VAR=x cmd`	`env={**os.environ, "VAR": "x"}`
`which cmd`	`shutil.which("cmd")`
`cp`, `mv`, `rm -r`	`shutil.copy`, `shutil.move`, `shutil.rmtree`

For file operations specifically, prefer pathlib and shutil over shelling out at all. They're faster, portable, and raise proper Python exceptions. See pathlib in Python.

The shlex module helps when you're converting between strings and argument lists:

import shlex

print(shlex.split('grep -n "hello world" notes.txt'))
print(shlex.quote("report; rm -rf ~"))
print(shlex.join(["echo", "it's", "a b"]))

['grep', '-n', 'hello world', 'notes.txt']
'report; rm -rf ~'
echo 'it'"'"'s' 'a b'

shlex.split turns a command string into a list the way a POSIX shell would, which is handy for commands read from config.
shlex.quote escapes one value for safe inclusion in a shell string, if you truly must use shell=True.
shlex.join does the reverse of split, which is useful for printing a command in logs or error messages so it can be copied and re-run.

If you do use shell=True, keep the string a constant, or quote every dynamic piece with shlex.quote.

A Reusable Helper

In a real project, you'll call commands from many places and want consistent error handling. A small wrapper keeps that in one spot:

# shell.py
import shlex
import subprocess
from collections.abc import Sequence
from pathlib import Path


class CommandError(RuntimeError):
    pass


def run(
    args: Sequence[str],
    *,
    cwd: str | Path | None = None,
    timeout: float = 60,
) -> str:
    """Run a command and return its stdout, raising CommandError on failure."""
    try:
        result = subprocess.run(
            args,
            cwd=cwd,
            capture_output=True,
            text=True,
            encoding="utf-8",
            errors="replace",
            timeout=timeout,
            check=True,
        )
    except FileNotFoundError:
        raise CommandError(f"command not found: {args[0]}") from None
    except subprocess.TimeoutExpired:
        raise CommandError(f"timed out after {timeout}s: {shlex.join(args)}") from None
    except subprocess.CalledProcessError as exc:
        detail = exc.stderr.strip() or exc.stdout.strip()
        raise CommandError(
            f"exit {exc.returncode}: {shlex.join(args)}\n{detail}"
        ) from None
    return result.stdout


if __name__ == "__main__":
    print(run(["git", "rev-parse", "--abbrev-ref", "HEAD"]).strip())
    print(run(["git", "log", "-3", "--pretty=%h %s"]))
    try:
        run(["git", "checkout", "no-such-branch"])
    except CommandError as exc:
        print(exc)
    try:
        run(["sleep", "3"], timeout=0.5)
    except CommandError as exc:
        print(exc)

Run inside a Git repository, it prints something like:

main
b3206eb Fix rounding in totals
4bb8e50 Add order model
1f59e28 Initial commit

exit 1: git checkout no-such-branch
error: pathspec 'no-such-branch' did not match any file(s) known to git
timed out after 0.5s: sleep 3

The wrapper turns all three failure modes into one exception type with a readable message that includes the exact command (via shlex.join) and the program's own error output. Callers only need to catch CommandError. The from None suppresses the chained traceback from the original exception, since the message already contains everything useful. If you'd rather keep the chain for debugging, use from exc instead. For more on designing exception types like this, see creating custom exceptions.

Streaming Output with Popen

run() waits until the command finishes before giving you anything. For long-running commands, like a build, a test suite, or a download, you often want to show output as it arrives. That's what Popen is for: it starts the process and returns immediately, and you read from its pipes while it runs.

# stream_output.py
import subprocess
import sys

child_code = """
import time
for step in range(1, 4):
    print(f"step {step}/3", flush=True)
    time.sleep(0.5)
"""

with subprocess.Popen(
    [sys.executable, "-c", child_code],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    text=True,
    bufsize=1,
) as proc:
    for line in proc.stdout:
        print(f"[child] {line}", end="")

print(f"exit code: {proc.returncode}")

[child] step 1/3
[child] step 2/3
[child] step 3/3
exit code: 0

Each line appears half a second after the previous one, as the child prints it. The details:

stdout=subprocess.PIPE connects the child's output to proc.stdout, a file-like object you can iterate line by line.
stderr=subprocess.STDOUT merges standard error into the same stream, so you see errors in order with regular output.
bufsize=1 requests line buffering on your side of the pipe (only meaningful in text mode).
The with block waits for the process to exit and closes its pipes when you leave it, so proc.returncode is set afterward.

Line-by-line streaming only works if the child flushes its output. Many programs buffer output when they detect they're writing to a pipe instead of a terminal, so lines may arrive in bursts. For Python children, flush=True or the -u flag fixes it. Other tools often have a flag like --line-buffered.

Popen with check-like behavior is up to you: after the with block, check proc.returncode and raise if it's non-zero.

Avoiding Deadlocks

If you capture both stdout and stderr with separate pipes and read only one of them, the child can fill the other pipe's buffer and block forever, while you wait forever for it. Two safe patterns:

Merge them with stderr=subprocess.STDOUT and read one stream, as above.
Use proc.communicate(), which reads both streams concurrently until the process exits. This is what run() does internally.

Building a Pipeline

To connect commands the way a shell pipe does, without shell=True, wire one process's stdout to the next process's stdin:

# pipeline.py
import subprocess
import sys

producer = subprocess.Popen(
    [sys.executable, "-c", "print('banana'); print('apple'); print('cherry')"],
    stdout=subprocess.PIPE,
)
consumer = subprocess.Popen(
    ["sort"],
    stdin=producer.stdout,
    stdout=subprocess.PIPE,
    text=True,
)
producer.stdout.close()  # let producer get SIGPIPE if consumer exits early
output, _ = consumer.communicate()
producer.wait()
print(output, end="")

apple
banana
cherry

Closing producer.stdout in the parent is the step people miss. The consumer now owns that pipe; if the parent keeps its copy open, the producer won't notice when the consumer exits early. If the first command's output is small, a simpler approach is to run it with run(), capture its output, and pass it to the second command with input=.

Platform Notes

Windows built-ins. Commands like dir and copy are built into cmd.exe, not standalone programs, so they only work with shell=True. Prefer Python equivalents instead.
Batch files. Running .bat or .cmd files on Windows goes through cmd.exe even without shell=True, so treat their arguments with the same care as shell strings.
Executable lookup. With a list command, the first element is looked up on PATH. Passing a full path (for example from shutil.which) removes any ambiguity.

Async Code

If you're inside an asyncio application, blocking on subprocess.run() stalls the event loop. Use asyncio.create_subprocess_exec() instead; it accepts the same list-of-arguments style and lets you await proc.communicate(). The asyncio beginner's guide covers the event loop itself.

Quick Reference

Task	Code
Run, fail loudly	`subprocess.run(args, check=True)`
Capture output as text	`subprocess.run(args, capture_output=True, text=True)`
Get stdout only	`subprocess.check_output(args, text=True)`
Send input	`subprocess.run(args, input="data", text=True)`
Silence output	`stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL`
Limit run time	`timeout=30`
Different directory	`cwd="path"`
Extra environment variable	`env={**os.environ, "KEY": "value"}`
Stream output live	`Popen(args, stdout=PIPE, text=True)` and iterate `proc.stdout`

Conclusion

subprocess.run() with a list of arguments covers almost every case: add capture_output=True and text=True to read the output, check=True so failures raise instead of slipping by, and a timeout so nothing hangs forever. Avoid shell=True unless the command string is a constant, and reach for pathlib, shutil, and shlex before reaching for the shell. When you need live output or a pipeline, Popen gives you the lower-level control, as long as you read pipes in a way that can't deadlock.