Speeding Up Python Code: Practical Optimization Techniques

Python has a reputation for being slow, and for tight numeric loops written in pure Python, it is. But most slow Python programs aren't slow because of the interpreter. They're slow because of a list used where a set belongs, a function recomputing the same answer thousands of times, or a database commit inside a loop. Those problems are cheap to fix and the gains are often 10x to 1000x, far more than any micro-tweak.

This post is a practical catalog of techniques, roughly ordered from the biggest typical payoff to the smallest. Every benchmark here was run with timeit on Python 3.13 on a laptop, so treat the numbers as relative rather than absolute. Your results will differ, which is exactly why the first rule matters.

Rule Zero: Profile Before You Optimize

Don't guess where the time goes. Run the program under cProfile, find the function with the highest cumulative or internal time, and focus there. Optimizing code that accounts for 2% of the runtime can, at best, make the program 2% faster.

python -m cProfile -s cumulative your_script.py

Once you have a candidate fix, compare old and new with timeit and check they return the same result. The details of both tools are in Profiling Python Code with cProfile and timeit. Everything below assumes you've already found your hotspot.

1. Use the Right Data Structure

This is where the largest wins usually hide. The classic case is membership testing: x in some_list scans the list element by element (O(n)), while x in some_set is a hash lookup (O(1) on average).

# membership.py
import timeit

setup = """
banned = [f"user{i}" for i in range(2_000)]
banned_set = set(banned)
requests = [f"user{i}" for i in range(0, 20_000, 3)]
"""
for name, stmt in [
    ("list lookup", "[r for r in requests if r in banned]"),
    ("set lookup", "[r for r in requests if r in banned_set]"),
]:
    best = min(timeit.repeat(stmt, setup, number=5, repeat=5)) / 5
    print(f"{name:<11} {best * 1000:.2f} ms")

list lookup 76.75 ms
set lookup  0.19 ms

A 400x difference from changing one variable. Some other swaps worth knowing:

If you're doing this	Use this instead
Repeated x in list	A set or dict
list.pop(0) or list.insert(0, x) in a loop (O(n) each)	collections.deque with popleft() / appendleft()
Searching a list for an item by key	A dict keyed by that field
Keeping a list sorted while inserting	bisect.insort, or sort once at the end
Repeatedly finding the smallest item	heapq
Removing duplicates while keeping order	list(dict.fromkeys(items))

The point isn't to memorize the table. It's to ask, whenever a loop runs over a collection, "what operation am I doing on this collection inside the loop, and what does it cost?"

2. Lean on Built-ins and the Standard Library

Built-in functions like sum(), min(), max(), sorted(), any(), all(), and str.join() are implemented in C. They do the looping without executing Python bytecode for each element.

import timeit

setup = "data = list(range(1_000_000))"
manual = """
total = 0
for x in data:
    total += x
"""
for name, stmt in [("manual loop", manual), ("sum()", "total = sum(data)")]:
    best = min(timeit.repeat(stmt, setup, number=10, repeat=5)) / 10
    print(f"{name:<12} {best * 1000:.2f} ms")

manual loop  19.77 ms
sum()        2.91 ms

The same applies to specialized tools in the standard library. Counting with collections.Counter is about twice as fast as a hand-written dictionary loop:

import timeit

setup = """
import random
from collections import Counter
random.seed(1)
words = [random.choice(["red", "green", "blue", "cyan", "pink"]) for _ in range(200_000)]
"""
manual = """
counts = {}
for w in words:
    if w in counts:
        counts[w] += 1
    else:
        counts[w] = 1
"""
for name, stmt in [("manual dict", manual), ("Counter", "counts = Counter(words)")]:
    best = min(timeit.repeat(stmt, setup, number=10, repeat=5)) / 10
    print(f"{name:<12} {best * 1000:.2f} ms")

manual dict  11.73 ms
Counter      5.54 ms

It's also shorter and clearer. Before you write a loop, check whether itertools, collections, functools, heapq, bisect, math, or statistics already does the job. For example, math.hypot(x, y) beats math.sqrt(x * x + y * y), and math.prod(), math.fsum(), and statistics.fmean() are all C-backed.

3. Build Strings with join()

Strings are immutable, so s += piece in a loop may create a new string each time. CPython has an optimization that sometimes avoids the copy, but you can't rely on it, and str.join() is consistently faster:

import timeit

setup = "words = [f'word{i}' for i in range(100_000)]"
concat = """
s = ""
for w in words:
    s += w + ","
"""
join = 's = ",".join(words)'
for name, stmt in [("+= in a loop", concat), ("str.join()", join)]:
    best = min(timeit.repeat(stmt, setup, number=20, repeat=5)) / 20
    print(f"{name:<13} {best * 1000:.2f} ms")

+= in a loop  5.65 ms
str.join()    0.71 ms

If you build output piece by piece, append the pieces to a list and "".join() them at the end, or write to an io.StringIO.

4. Cache Results You Compute Repeatedly

If a pure function (same input, same output, no side effects) is called with the same arguments many times, cache it. functools.cache memoizes results with one decorator. Recursive algorithms with overlapping subproblems benefit the most:

# cache_demo.py
import time
from functools import cache


def edit_distance(a: str, b: str) -> int:
    if not a:
        return len(b)
    if not b:
        return len(a)
    if a[0] == b[0]:
        return edit_distance(a[1:], b[1:])
    return 1 + min(
        edit_distance(a[1:], b),      # delete
        edit_distance(a, b[1:]),      # insert
        edit_distance(a[1:], b[1:]),  # replace
    )


@cache
def edit_distance_cached(a: str, b: str) -> int:
    if not a:
        return len(b)
    if not b:
        return len(a)
    if a[0] == b[0]:
        return edit_distance_cached(a[1:], b[1:])
    return 1 + min(
        edit_distance_cached(a[1:], b),
        edit_distance_cached(a, b[1:]),
        edit_distance_cached(a[1:], b[1:]),
    )


for func in (edit_distance, edit_distance_cached):
    start = time.perf_counter()
    result = func("kitten sitting", "sitting kit")
    print(f"{func.__name__:<22} {result}  {time.perf_counter() - start:.4f}s")

edit_distance          8  3.1882s
edit_distance_cached   8  0.0001s

From three seconds to a tenth of a millisecond. The uncached version recomputes the same substring pairs an exponential number of times; the cached one computes each pair once.

Things to keep in mind:

• @cache is unbounded. For functions called with many distinct arguments, use @lru_cache(maxsize=1024) to cap memory.
• Arguments must be hashable. Lists and dicts won't work; convert them to tuples or frozensets.
• Only cache pure functions. Caching something that reads the clock or a database returns stale data.
• On methods, the cache holds a reference to self and keeps instances alive. Prefer functools.cached_property for per-instance computed values.

Caching also applies outside functions: hoist work that doesn't change out of loops. Compiling a regex, parsing a config file, or building a lookup table should happen once, not on every iteration.

5. Be Lazy: Don't Do Work You Don't Need

Generators produce values on demand, so you can stop early and avoid building large intermediate lists. Combined with short-circuiting functions like any(), all(), and next(), the difference can be enormous:

import timeit

setup = "data = list(range(1_000_000))"
for name, stmt in [
    ("list then check", "len([x for x in data if x > 10]) > 0"),
    ("any() generator", "any(x > 10 for x in data)"),
]:
    best = min(timeit.repeat(stmt, setup, number=10, repeat=5)) / 10
    print(f"{name:<16} {best * 1e6:.1f} usec")

list then check  21293.0 usec
any() generator  0.6 usec

The first version builds a list of nearly a million items just to see whether it's non-empty. The second stops at the 12th element. Other forms of laziness:

• next((x for x in items if pred(x)), None) to find the first match
• itertools.islice() to take the first N items of any iterable
• Reading files line by line (for line in f) instead of f.read().splitlines()
• Returning early from functions once the answer is known

Generators are covered in depth in What Is a Generator in Python.

6. Batch I/O and Database Work

For many real applications, the CPU isn't the bottleneck at all. The time goes into round trips: network calls, disk syncs, database commits. Each one has a fixed cost, so doing them one at a time multiplies that cost.

# batching.py
import sqlite3
import time
from pathlib import Path

rows = [(i, f"item-{i}") for i in range(2_000)]


def fresh_db(path: Path) -> sqlite3.Connection:
    path.unlink(missing_ok=True)
    conn = sqlite3.connect(path)
    conn.execute("CREATE TABLE items (id INTEGER PRIMARY KEY, name TEXT)")
    conn.commit()
    return conn


conn = fresh_db(Path("one_by_one.db"))
start = time.perf_counter()
for row in rows:
    conn.execute("INSERT INTO items VALUES (?, ?)", row)
    conn.commit()
print(f"commit per row:   {time.perf_counter() - start:.3f}s")
conn.close()

conn = fresh_db(Path("batched.db"))
start = time.perf_counter()
with conn:
    conn.executemany("INSERT INTO items VALUES (?, ?)", rows)
print(f"one executemany:  {time.perf_counter() - start:.3f}s")
conn.close()

commit per row:   0.578s
one executemany:  0.001s

Committing after every row forces a disk sync each time. One transaction with executemany() does the same inserts hundreds of times faster. The same principle applies elsewhere:

• Use bulk endpoints and bulk inserts instead of one request or query per item
• Fetch related data in one query (a join or IN (...)) instead of a query per row, the "N+1" problem
• Reuse HTTP connections with a session object instead of opening a new connection per request
• Buffer writes to files rather than flushing after each line

If you're working with databases from Python, How to Use Python with Databases covers the basics.

7. Vectorize Numeric Work with NumPy

When you're doing arithmetic over large arrays of numbers, a pure-Python loop pays interpreter overhead and boxes every number as a Python object. NumPy stores numbers in contiguous typed arrays and runs operations in compiled code.

# vectorize.py
import timeit

import numpy as np

prices = [float(i % 500) for i in range(1_000_000)]
arr = np.array(prices)


def with_tax_loop(values: list[float]) -> list[float]:
    return [v * 1.2 + 0.5 for v in values]


def with_tax_numpy(values: np.ndarray) -> np.ndarray:
    return values * 1.2 + 0.5


assert np.allclose(with_tax_loop(prices), with_tax_numpy(arr))
for name, call in [
    ("list comprehension", lambda: with_tax_loop(prices)),
    ("NumPy", lambda: with_tax_numpy(arr)),
]:
    best = min(timeit.repeat(call, number=10, repeat=5)) / 10
    print(f"{name:<18} {best * 1000:.2f} ms")

list comprehension 26.83 ms
NumPy              0.35 ms

About 75 times faster. The catch: converting between Python lists and NumPy arrays has a cost, so vectorization pays off when data stays in arrays across several operations. For tabular data, pandas and Polars give you the same benefit with a DataFrame API. See How to Use Python for Data Analysis.

8. Micro-Optimizations: Know Their Limits

There's a lot of folklore about small tricks: binding a global function to a local variable, avoiding attribute lookups with . in loops, preferring comprehensions over append(). Some of it was valuable in older Pythons. Since Python 3.11, the specializing adaptive interpreter has made many of these differences small.

# hoisting.py
import math
import timeit

points = [(i, i + 1) for i in range(200_000)]


def distances_slow(pts: list[tuple[int, int]]) -> list[float]:
    out = []
    for x, y in pts:
        out.append(math.sqrt(x * x + y * y))
    return out


def distances_fast(pts: list[tuple[int, int]]) -> list[float]:
    sqrt = math.sqrt
    return [sqrt(x * x + y * y) for x, y in pts]


def distances_hypot(pts: list[tuple[int, int]]) -> list[float]:
    return [math.hypot(x, y) for x, y in pts]


for f in (distances_slow, distances_fast, distances_hypot):
    best = min(timeit.repeat(lambda: f(points), number=10, repeat=5)) / 10
    print(f"{f.__name__:<16} {best * 1000:.2f} ms")

distances_slow   18.90 ms
distances_fast   17.59 ms
distances_hypot  11.70 ms

Hoisting math.sqrt into a local and using a comprehension saved about 7%. Using the right built-in, math.hypot, saved 38%. That pattern repeats: picking a better tool beats tweaking how you call the same tool.

Comprehensions are still worth using because they're clearer, and they're usually a little faster than an explicit append() loop. Just don't expect them to rescue a slow algorithm.

9. Upgrade Python

The cheapest optimization is often a version bump. The "Faster CPython" work made Python 3.11 significantly faster than 3.10 on typical benchmarks, and 3.12, 3.13, and 3.14 have continued to improve. If you're still on 3.9 or 3.10, upgrading can speed up your whole codebase with no code changes.

Python 3.13 also added an experimental JIT compiler (opt-in, disabled by default) and 3.14 includes it in the official Windows and macOS binaries, still off by default and enabled with the PYTHON_JIT=1 environment variable. It's not yet a reliable speedup for most code, so measure before relying on it.

10. Go Parallel When the Work Allows It

If the hotspot is CPU-bound and splits into independent chunks, you can spread it across cores. On a standard CPython build, threads don't help pure-Python CPU work because of the GIL, so use processes:

# parallel_primes.py
from concurrent.futures import ProcessPoolExecutor


def count_primes(limit: int) -> int:
    count = 0
    for n in range(2, limit):
        if all(n % d for d in range(2, int(n**0.5) + 1)):
            count += 1
    return count


if __name__ == "__main__":
    with ProcessPoolExecutor() as executor:
        print(list(executor.map(count_primes, [100_000] * 8)))

Parallelism multiplies your throughput by, at most, the number of cores. It's worth doing after you've fixed the algorithm, not instead of it. Using concurrent.futures for Simple Parallelism covers pools in detail, and Understanding the GIL and Free-Threaded Python explains when threads can help.

11. Compile the Hot Path

When a well-designed pure-Python hotspot is still too slow, you can move it out of the interpreter:

• PyPy is an alternative Python implementation with a tracing JIT. Long-running pure-Python code often runs several times faster with no changes, but C-extension compatibility varies.
• Cython compiles Python-like code with optional static types to C extensions.
• mypyc compiles type-annotated Python modules to C extensions; it's what makes mypy itself fast.
• Numba JIT-compiles numeric functions that use NumPy arrays with a single decorator.
• Rust or C extensions (for example with PyO3) are the heavy option, for libraries where every millisecond matters.

These add build complexity, so reach for them last and only for code you've profiled.

A Quick Checklist

When you've found a slow function, ask in this order:

1. Can I avoid doing this work at all, or do less of it (laziness, early exit)?
2. Am I using the right data structure for the operations inside the loop?
3. Am I recomputing something that could be cached or hoisted?
4. Am I doing I/O one item at a time that could be batched?
5. Is there a built-in, standard library function, or NumPy operation that does this?
6. Can independent chunks run in parallel?
7. Is it still too slow? Then consider compiling it.

Conclusion

Most Python speedups come from doing less work, not from doing the same work with cleverer syntax. Profile to find the hotspot, then fix it with the right data structure, built-ins, caching, laziness, and batched I/O. Move numeric work into NumPy, use processes for CPU-bound chunks, and keep your Python version current. Micro-optimizations are the last few percent; the first 100x is almost always algorithmic. And whatever you change, measure it and check the result is still correct.