psleak

Project Links

Meta

Author: Giampaolo Rodola

Requires Python: >=3.6

Classifiers

Development Status

4 - Beta

Intended Audience

Developers

Operating System

OS Independent

Programming Language

Python :: 3

Topic

Software Development :: Debuggers
Software Development :: Testing
System :: Monitoring

A testing framework for detecting memory leaks and unclosed resources created by Python functions, particularly those implemented in C extensions.

It was originally developed as part of psutil test suite, and later split out into a standalone project.

Note: this project is still experimental. API and internal heuristics may change.

Features

Memory leak detection

The framework measures process memory before and after repeatedly calling a function, tracking:

Heap metrics from psutil.heap_info
USS, RSS and VMS from psutil.Process.memory_full_info

The goal is to catch cases where C native code allocates memory without freeing it, such as:

malloc() without free()
mmap() without munmap()
HeapAlloc() without HeapFree() (Windows)
VirtualAlloc() without VirtualFree() (Windows)
HeapCreate() without HeapDestroy() (Windows)
Python C objects for which you forget to call Py_DECREF, Py_CLEAR, PyMem_Free, etc.

Because memory usage is noisy and influenced by the OS, allocator and garbage collector, the function is called repeatedly with an increasing number of invocations. If memory usage continues to grow across runs, it is marked as a leak and a MemoryLeakError exception is raised.

Unclosed resource detection

Beyond memory, the framework also detects resources that the target function allocates but fails to release after it’s called once. The following categories are monitored:

File descriptors (POSIX): e.g. open() without close(), shm_open() without shm_close(), unclosed sockets, pipes, and similar objects.
Windows handles: kernel objects created via calls such as OpenFile(), OpenProcess(), CreatePipe() and others that are not released with CloseHandle()
Python threads: threading.Thread objects that were started but never joined or otherwise stopped.
Native system threads: low-level threads created directly via pthread_create() or CreateThread() (Windows) that remain running or unjoined. These are not Python threading.Thread objects, but OS threads started by C extensions without a matching pthread_join() or WaitForSingleObject() (Windows).

Uncollectable GC objects: objects that cannot be garbage collected because they form reference cycles and / or define a __del__ method, e.g.:

class Leaky:
    def __init__(self):
        self.ref = None

def create_cycle():
    a = Leaky()
    b = Leaky()
    a.ref = b
    b.ref = a
    return a, b  # cycle preventing GC from collecting

Each category raises a specific assertion error describing what was leaked.

Install

pip install psleak

Usage

Subclass MemoryLeakTestCase and call execute() inside a test:

from psleak import MemoryLeakTestCase

class TestLeaks(MemoryLeakTestCase):
    def test_fun(self):
        self.execute(some_function)

If the function leaks memory or resources, the test will fail with a descriptive exception, e.g.:

psleak.MemoryLeakError: memory kept increasing after 10 runs
Run # 1: heap=+388160  | uss=+356352  | rss=+327680  | (calls= 200, avg/call=+1940)
Run # 2: heap=+584848  | uss=+614400  | rss=+491520  | (calls= 300, avg/call=+1949)
Run # 3: heap=+778320  | uss=+782336  | rss=+819200  | (calls= 400, avg/call=+1945)
Run # 4: heap=+970512  | uss=+1032192 | rss=+1146880 | (calls= 500, avg/call=+1941)
Run # 5: heap=+1169024 | uss=+1171456 | rss=+1146880 | (calls= 600, avg/call=+1948)
Run # 6: heap=+1357360 | uss=+1413120 | rss=+1310720 | (calls= 700, avg/call=+1939)
Run # 7: heap=+1552336 | uss=+1634304 | rss=+1638400 | (calls= 800, avg/call=+1940)
Run # 8: heap=+1752032 | uss=+1781760 | rss=+1802240 | (calls= 900, avg/call=+1946)
Run # 9: heap=+1945056 | uss=+2031616 | rss=+2129920 | (calls=1000, avg/call=+1945)
Run #10: heap=+2140624 | uss=+2179072 | rss=+2293760 | (calls=1100, avg/call=+1946)

Configuration

MemoryLeakTestCase exposes several tunables as class attributes or per-call overrides:

warmup_times: warm-up calls before starting measurement (default: 10)
times: number of times to call the tested function in each iteration. (default: 200)
retries: maximum retries if memory keeps growing (default: 10)
tolerance: allowed memory growth (in bytes or per-metric) before it is considered a leak. (default: 0)
trim_callback: optional callable to free caches before starting measurement (default: None)
checkers: config object controlling which checkers to run (default: None)
verbosity: diagnostic output level (default: 0)

You can override these either when calling execute():

from psleak import MemoryLeakTestCase, Checkers

class MyTest(MemoryLeakTestCase):
    def test_fun(self):
        self.execute(
            some_function,
            times=500,
            tolerance=1024,
            checkers=Checkers.exclude("gcgarbage")
         )

…or at class level:

from psleak import MemoryLeakTestCase, Checkers

class MyTest(MemoryLeakTestCase):
    times = 500
    tolerance = {"rss": 1024}
    checkers = Checkers.only("memory")

    def test_fun(self):
        self.execute(some_function)

Auto-generate tests

In order to avoid writing many similar test methods by hand, MemoryLeakTestCase can auto-generate test methods from a declarative specification:

import numpy
from psleak import MemoryLeakTestCase, LeakTest

class TestNumpyLeaks(MemoryLeakTestCase):

    @classmethod
    def auto_generate(cls):
        return {
            "zeros": LeakTest(numpy.zeros, 10),
            "add": LeakTest(numpy.add, 1, 2),
            "custom": LeakTest(numpy.prod, [1, 2], times=10, tolerance=1024),
        }

This will define the following test methods: test_leak_zeros, test_leak_add, test_leak_custom.

Recommended test environment

For more reliable results, it is important to run tests with:

PYTHONMALLOC=malloc PYTHONUNBUFFERED=1 python3 -m pytest test_memleaks.py

Why this matters:

PYTHONMALLOC=malloc: disables the pymalloc allocator, which caches small objects (<= 512 bytes) and therefore makes leak detection less reliable. With pymalloc disabled, all memory allocations go through the system malloc(), making them visible in heap, USS, RSS, and VMS metrics.
PYTHONUNBUFFERED=1: disables stdout/stderr buffering, making memory leak detection more reliable.

Memory leak tests should be run separately from other tests, and not in parallel (e.g. via pytest-xdist).

Run psleak own tests

git clone git@github.com:giampaolo/psleak.git
cd psleak
make install-pydeps build test

References

(2025) Blog post about psutil heap APIs
(2018) History of psutil heap APIs: psutil issue #1275
(2016) Blog post about USS and PSS memory
Usage of psleak in psutil tests: test_memleaks.py

Files in release

psleak-0.1.5.tar.gz (25.8KiB)

Extras:

Dependencies:

psutil (>=7.2.1)

psleak 0.1.5