memory_profiler

Monitor Memory usage of Python code

4,321

375

4,321

134

View on GitHub

Top Related Projects

pympler

1,197

Development tool to measure, monitor and analyze the memory behavior of Python objects in a running Python application.

profiling

2,966

Was an interactive continuous Python profiler.

pyinstrument

6,432

🚴 Call stack profiler for Python. Shows you why your code is slow!

Quick Overview

Memory Profiler is a Python module for monitoring memory consumption of a process as well as line-by-line analysis of memory consumption for Python programs. It provides a simple way to measure and visualize the memory usage of Python code, helping developers identify memory leaks and optimize their applications.

Pros

Easy to use with minimal setup required
Provides line-by-line memory usage analysis
Can be used as a decorator or in command-line mode
Supports both Python 2 and Python 3

Cons

May introduce performance overhead when profiling
Limited to Python programs only
Accuracy can be affected by garbage collection timing
Might not capture short-lived objects in some cases

Code Examples

Using Memory Profiler as a decorator:

from memory_profiler import profile

@profile
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

if __name__ == '__main__':
    my_func()

Line-by-line memory usage:

@profile
def my_func():
    a = [1] * (10 ** 6)  # Line 1
    b = [2] * (2 * 10 ** 7)  # Line 2
    del b  # Line 3
    return a  # Line 4

my_func()

Using Memory Profiler in a Jupyter notebook:

%load_ext memory_profiler

@profile
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

%mprun -f my_func my_func()

Getting Started

To get started with Memory Profiler:

Install the package:
```
pip install memory_profiler
```

Import and use the @profile decorator:

from memory_profiler import profile

@profile
def my_function():
    # Your code here
    pass

my_function()

Run your script with:

python -m memory_profiler your_script.py

This will display the line-by-line memory usage of your profiled function.

Competitor Comparisons

pympler

1,197

Development tool to measure, monitor and analyze the memory behavior of Python objects in a running Python application.

Pros of Pympler

More comprehensive memory analysis tools, including object tracker and muppy
Provides detailed memory breakdowns by object type
Supports asynchronous profiling

Cons of Pympler

Steeper learning curve due to more complex API
May have higher overhead for simple memory profiling tasks
Less straightforward integration with existing code

Code Comparison

memory_profiler:

@profile
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

Pympler:

from pympler import tracker
tr = tracker.SummaryTracker()
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a
my_func()
tr.print_diff()

Both tools offer valuable memory profiling capabilities, but they cater to different use cases. memory_profiler is simpler and more straightforward for basic profiling needs, while Pympler provides more advanced features for in-depth memory analysis. The choice between them depends on the specific requirements of your project and the level of detail needed in memory profiling.

profiling

2,966

Was an interactive continuous Python profiler.

Pros of profiling

More comprehensive profiling capabilities, including CPU, memory, and I/O profiling
Interactive visualization tools for easier analysis of profiling data
Supports both synchronous and asynchronous code profiling

Cons of profiling

Steeper learning curve due to more advanced features
May have higher overhead for simple profiling tasks
Less focused on memory profiling specifically

Code Comparison

memory_profiler:

@profile
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

if __name__ == '__main__':
    my_func()

profiling:

from profiling import profile

@profile
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

if __name__ == '__main__':
    my_func()

Both libraries use a decorator approach for profiling functions, but profiling offers more advanced features and visualization tools. memory_profiler is more focused on memory usage, while profiling provides a broader range of profiling capabilities. The choice between the two depends on the specific profiling needs and the complexity of the project.

pyinstrument

6,432

🚴 Call stack profiler for Python. Shows you why your code is slow!

Pros of pyinstrument

Provides a more detailed and hierarchical view of performance bottlenecks
Offers both command-line and programmatic interfaces for profiling
Generates interactive HTML reports for easy analysis

Cons of pyinstrument

Focuses on CPU time profiling rather than memory usage
May have a higher overhead compared to memory_profiler for certain use cases
Requires manual integration into code for profiling specific functions

Code Comparison

memory_profiler:

from memory_profiler import profile

@profile
def my_function():
    # Function code here

pyinstrument:

from pyinstrument import Profiler

profiler = Profiler()
profiler.start()
# Code to profile
profiler.stop()
profiler.print()

Key Differences

memory_profiler specializes in memory usage tracking, while pyinstrument focuses on CPU time profiling
pyinstrument provides more detailed call stack information and visualization options
memory_profiler offers line-by-line memory usage analysis, which is not available in pyinstrument

Use Cases

Use memory_profiler when analyzing memory consumption is the primary concern
Choose pyinstrument for identifying performance bottlenecks and optimizing execution time
Consider using both tools in conjunction for comprehensive performance analysis of Python applications

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

.. image:: https://travis-ci.org/pythonprofilers/memory_profiler.svg?branch=master :target: https://travis-ci.org/pythonprofilers/memory_profiler

================= Memory Profiler

Note: This package is no longer actively maintained. I won't be actively responding to issues.

This is a python module for monitoring memory consumption of a process as well as line-by-line analysis of memory consumption for python programs. It is a pure python module which depends on the psutil <http://pypi.python.org/pypi/psutil>_ module.

============== Installation

Install via pip::

$ pip install -U memory_profiler

The package is also available on conda-forge <https://github.com/conda-forge/memory_profiler-feedstock>_.

To install from source, download the package, extract and type::

$ pip install .

=========== Quick Start

Use mprof to generate a full memory usage report of your executable and to plot it.

.. code-block:: bash

mprof run executable
mprof plot

The plot would be something like this:

.. image:: https://i.stack.imgur.com/ixCH4.png

======= Usage

line-by-line memory usage

The line-by-line memory usage mode is used much in the same way of the line_profiler <https://pypi.python.org/pypi/line_profiler/>_: first decorate the function you would like to profile with @profile and then run the script with a special script (in this case with specific arguments to the Python interpreter).

In the following example, we create a simple function my_func that allocates lists a, b and then deletes b:

.. code-block:: python

@profile
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

if __name__ == '__main__':
    my_func()

Execute the code passing the option -m memory_profiler to the python interpreter to load the memory_profiler module and print to stdout the line-by-line analysis. If the file name was example.py, this would result in::

$ python -m memory_profiler example.py

Output will follow::

Line #    Mem usage    Increment  Occurrences   Line Contents
============================================================
     3   38.816 MiB   38.816 MiB           1   @profile
     4                                         def my_func():
     5   46.492 MiB    7.676 MiB           1       a = [1] * (10 ** 6)
     6  199.117 MiB  152.625 MiB           1       b = [2] * (2 * 10 ** 7)
     7   46.629 MiB -152.488 MiB           1       del b
     8   46.629 MiB    0.000 MiB           1       return a

The first column represents the line number of the code that has been profiled, the second column (Mem usage) the memory usage of the Python interpreter after that line has been executed. The third column (Increment) represents the difference in memory of the current line with respect to the last one. The fourth column (Occurrences) shows the number of times that profiler has executed each line. The last column (Line Contents) prints the code that has been profiled.

Decorator

A function decorator is also available. Use as follows:

.. code-block:: python

from memory_profiler import profile

@profile
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

In this case the script can be run without specifying -m memory_profiler in the command line.

In function decorator, you can specify the precision as an argument to the decorator function. Use as follows:

.. code-block:: python

from memory_profiler import profile

@profile(precision=4)
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

If a python script with decorator @profile is called using -m memory_profiler in the command line, the precision parameter is ignored.

Time-based memory usage

Sometimes it is useful to have full memory usage reports as a function of time (not line-by-line) of external processes (be it Python scripts or not). In this case the executable mprof might be useful. Use it like::

mprof run <executable>
mprof plot

The first line run the executable and record memory usage along time, in a file written in the current directory. Once it's done, a graph plot can be obtained using the second line. The recorded file contains a timestamps, that allows for several profiles to be kept at the same time.

Help on each mprof subcommand can be obtained with the -h flag, e.g. mprof run -h.

In the case of a Python script, using the previous command does not give you any information on which function is executed at a given time. Depending on the case, it can be difficult to identify the part of the code that is causing the highest memory usage.

Adding the profile decorator to a function(ensure no from memory_profiler import profile statement) and running the Python script with

mprof run --python python <script>

will record timestamps when entering/leaving the profiled function. Running

mprof plot

afterward will plot the result, making plots (using matplotlib) similar to these:

.. image:: https://camo.githubusercontent.com/3a584c7cfbae38c9220a755aa21b5ef926c1031d/68747470733a2f2f662e636c6f75642e6769746875622e636f6d2f6173736574732f313930383631382f3836313332302f63623865376337382d663563632d313165322d386531652d3539373237623636663462322e706e67 :target: https://github.com/scikit-learn/scikit-learn/pull/2248 :height: 350px

or, with mprof plot --flame (the function and timestamp names will appear on hover):

.. image:: ./images/flamegraph.png :height: 350px

A discussion of these capabilities can be found here <http://fa.bianp.net/blog/2014/plot-memory-usage-as-a-function-of-time/>_.

.. warning:: If your Python file imports the memory profiler from memory_profiler import profile these timestamps will not be recorded. Comment out the import, leave your functions decorated, and re-run.

The available commands for mprof are:

mprof run: running an executable, recording memory usage
mprof plot: plotting one the recorded memory usage (by default, the last one)
mprof list: listing all recorded memory usage files in a user-friendly way.
mprof clean: removing all recorded memory usage files.
mprof rm: removing specific recorded memory usage files

Tracking forked child processes

In a multiprocessing context the main process will spawn child processes whose system resources are allocated separately from the parent process. This can lead to an inaccurate report of memory usage since by default only the parent process is being tracked. The mprof utility provides two mechanisms to track the usage of child processes: sum the memory of all children to the parent's usage and track each child individual.

To create a report that combines memory usage of all the children and the parent, use the include-children flag in either the profile decorator or as a command line argument to mprof::

mprof run --include-children <script>

The second method tracks each child independently of the main process, serializing child rows by index to the output stream. Use the multiprocess flag and plot as follows::

mprof run --multiprocess <script>
mprof plot

This will create a plot using matplotlib similar to this:

.. image:: https://cloud.githubusercontent.com/assets/745966/24075879/2e85b43a-0bfa-11e7-8dfe-654320dbd2ce.png :target: https://github.com/pythonprofilers/memory_profiler/pull/134 :height: 350px

You can combine both the include-children and multiprocess flags to show the total memory of the program as well as each child individually. If using the API directly, note that the return from memory_usage will include the child memory in a nested list along with the main process memory.

Plot settings

By default, the command line call is set as the graph title. If you wish to customize it, you can use the -t option to manually set the figure title.

mprof plot -t 'Recorded memory usage'

You can also hide the function timestamps using the n flag, such as

mprof plot -n

Trend lines and its numeric slope can be plotted using the s flag, such as

mprof plot -s

.. image:: ./images/trend_slope.png :height: 350px

The intended usage of the -s switch is to check the labels' numerical slope over a significant time period for :

>0 it might mean a memory leak.
~0 if 0 or near 0, the memory usage may be considered stable.
<0 to be interpreted depending on the expected process memory usage patterns, also might mean that the sampling period is too small.

The trend lines are for ilustrative purposes and are plotted as (very) small dashed lines.

Setting debugger breakpoints

It is possible to set breakpoints depending on the amount of memory used. That is, you can specify a threshold and as soon as the program uses more memory than what is specified in the threshold it will stop execution and run into the pdb debugger. To use it, you will have to decorate the function as done in the previous section with @profile and then run your script with the option -m memory_profiler --pdb-mmem=X, where X is a number representing the memory threshold in MB. For example::

$ python -m memory_profiler --pdb-mmem=100 my_script.py

will run my_script.py and step into the pdb debugger as soon as the code uses more than 100 MB in the decorated function.

.. TODO: alternatives to decoration (for example when you don't want to modify the file where your function lives).

===== API

memory_profiler exposes a number of functions to be used in third-party code.

memory_usage(proc=-1, interval=.1, timeout=None) returns the memory usage over a time interval. The first argument, proc represents what should be monitored. This can either be the PID of a process (not necessarily a Python program), a string containing some python code to be evaluated or a tuple (f, args, kw) containing a function and its arguments to be evaluated as f(*args, **kw). For example,

.. code-block:: python

>>> from memory_profiler import memory_usage
>>> mem_usage = memory_usage(-1, interval=.2, timeout=1)
>>> print(mem_usage)
[7.296875, 7.296875, 7.296875, 7.296875, 7.296875]

Here I've told memory_profiler to get the memory consumption of the current process over a period of 1 second with a time interval of 0.2 seconds. As PID I've given it -1, which is a special number (PIDs are usually positive) that means current process, that is, I'm getting the memory usage of the current Python interpreter. Thus I'm getting around 7MB of memory usage from a plain python interpreter. If I try the same thing on IPython (console) I get 29MB, and if I try the same thing on the IPython notebook it scales up to 44MB.

If you'd like to get the memory consumption of a Python function, then you should specify the function and its arguments in the tuple (f, args, kw). For example:

.. code-block:: python

>>> # define a simple function
>>> def f(a, n=100):
    ...     import time
    ...     time.sleep(2)
    ...     b = [a] * n
    ...     time.sleep(1)
    ...     return b
    ...
>>> from memory_profiler import memory_usage
>>> memory_usage((f, (1,), {'n' : int(1e6)}))

This will execute the code f(1, n=int(1e6)) and return the memory consumption during this execution.

========= REPORTING

The output can be redirected to a log file by passing IO stream as parameter to the decorator like @profile(stream=fp)

.. code-block:: python

>>> fp=open('memory_profiler.log','w+')
>>> @profile(stream=fp)
>>> def my_func():
    ...     a = [1] * (10 ** 6)
    ...     b = [2] * (2 * 10 ** 7)
    ...     del b
    ...     return a

For details refer: examples/reporting_file.py

Reporting via logger Module:

Sometime it would be very convenient to use logger module specially when we need to use RotatingFileHandler.

The output can be redirected to logger module by simply making use of LogFile of memory profiler module.

.. code-block:: python

>>> from memory_profiler import LogFile
>>> import sys
>>> sys.stdout = LogFile('memory_profile_log')

Customized reporting:

Sending everything to the log file while running the memory_profiler could be cumbersome and one can choose only entries with increments by passing True to reportIncrementFlag, where reportIncrementFlag is a parameter to LogFile class of memory profiler module.

.. code-block:: python

>>> from memory_profiler import LogFile
>>> import sys
>>> sys.stdout = LogFile('memory_profile_log', reportIncrementFlag=False)

For details refer: examples/reporting_logger.py

===================== IPython integration

After installing the module, if you use IPython, you can use the %mprun, %%mprun, %memit and %%memit magics.

For IPython 0.11+, you can use the module directly as an extension, with %load_ext memory_profiler

To activate it whenever you start IPython, edit the configuration file for your IPython profile, ~/.ipython/profile_default/ipython_config.py, to register the extension like this (If you already have other extensions, just add this one to the list):

.. code-block:: python

c.InteractiveShellApp.extensions = [
    'memory_profiler',
]

(If the config file doesn't already exist, run ipython profile create in a terminal.)

It then can be used directly from IPython to obtain a line-by-line report using the %mprun or %%mprun magic command. In this case, you can skip the @profile decorator and instead use the -f parameter, like this. Note however that function my_func must be defined in a file (cannot have been defined interactively in the Python interpreter):

.. code-block:: python

In [1]: from example import my_func, my_func_2

In [2]: %mprun -f my_func my_func()

or in cell mode:

.. code-block:: python

In [3]: %%mprun -f my_func -f my_func_2
   ...: my_func()
   ...: my_func_2()

Another useful magic that we define is %memit, which is analogous to %timeit. It can be used as follows:

.. code-block:: python

In [1]: %memit range(10000)
peak memory: 21.42 MiB, increment: 0.41 MiB

In [2]: %memit range(1000000)
peak memory: 52.10 MiB, increment: 31.08 MiB

or in cell mode (with setup code):

.. code-block:: python

In [3]: %%memit l=range(1000000)
   ...: len(l)
   ...:
peak memory: 52.14 MiB, increment: 0.08 MiB

For more details, see the docstrings of the magics.

For IPython 0.10, you can install it by editing the IPython configuration file ~/.ipython/ipy_user_conf.py to add the following lines:

.. code-block:: python

# These two lines are standard and probably already there.
import IPython.ipapi
ip = IPython.ipapi.get()

# These two are the important ones.
import memory_profiler
memory_profiler.load_ipython_extension(ip)

=============================== Memory tracking backends

memory_profiler supports different memory tracking backends including: 'psutil', 'psutil_pss', 'psutil_uss', 'posix', 'tracemalloc'. If no specific backend is specified the default is to use "psutil" which measures RSS aka "Resident Set Size". In some cases (particularly when tracking child processes) RSS may overestimate memory usage (see example/example_psutil_memory_full_info.py for an example). For more information on "psutil_pss" (measuring PSS) and "psutil_uss" please refer to: https://psutil.readthedocs.io/en/latest/index.html?highlight=memory_info#psutil.Process.memory_full_info

Currently, the backend can be set via the CLI

$ python -m memory_profiler --backend psutil my_script.py

and is exposed by the API

.. code-block:: python

>>> from memory_profiler import memory_usage
>>> mem_usage = memory_usage(-1, interval=.2, timeout=1, backend="psutil")

============================ Frequently Asked Questions

* Q: How accurate are the results ?
* A: This module gets the memory consumption by querying the
  operating system kernel about the amount of memory the current
  process has allocated, which might be slightly different from
  the amount of memory that is actually used by the Python
  interpreter. Also, because of how the garbage collector works in
  Python the result might be different between platforms and even
  between runs.

* Q: Does it work under windows ?
* A: Yes, thanks to the
  `psutil <http://pypi.python.org/pypi/psutil>`_ module.

=========================== Support, bugs & wish list

For support, please ask your question on stack overflow <http://stackoverflow.com/>_ and add the *memory-profiling* tag <http://stackoverflow.com/questions/tagged/memory-profiling>. Send issues, proposals, etc. to github's issue tracker <https://github.com/pythonprofilers/memory_profiler/issues> .

If you've got questions regarding development, you can email me directly at f@bianp.net

.. image:: http://fa.bianp.net/static/tux_memory_small.png

============= Development

Latest sources are available from github:

https://github.com/pythonprofilers/memory_profiler

=============================== Projects using memory_profiler

Benchy <https://github.com/python-recsys/benchy>_

IPython memory usage <https://github.com/ianozsvald/ipython_memory_usage>_

PySpeedIT <https://github.com/peter1000/PySpeedIT>_ (uses a reduced version of memory_profiler)

pydio-sync <https://github.com/pydio/pydio-sync>_ (uses custom wrapper on top of memory_profiler)

========= Authors

This module was written by Fabian Pedregosa <http://fseoane.net>_ and Philippe Gervais <https://github.com/pgervais>_ inspired by Robert Kern's line profiler <http://packages.python.org/line_profiler/>_.

Tom <http://tomforb.es/>_ added windows support and speed improvements via the psutil <http://pypi.python.org/pypi/psutil>_ module.

Victor <https://github.com/octavo>_ added python3 support, bugfixes and general cleanup.

Vlad Niculae <http://vene.ro/>_ added the %mprun and %memit IPython magics.

Thomas Kluyver <https://github.com/takluyver>_ added the IPython extension.

Sagar UDAY KUMAR <https://github.com/sagaru>_ added Report generation feature and examples.

Dmitriy Novozhilov <https://github.com/demiurg906>_ and Sergei Lebedev <https://github.com/superbobry>_ added support for tracemalloc <https://docs.python.org/3/library/tracemalloc.html>_.

Benjamin Bengfort <https://github.com/bbengfort>_ added support for tracking the usage of individual child processes and plotting them.

Muhammad Haseeb Tariq <https://github.com/mhaseebtariq>_ fixed issue #152, which made the whole interpreter hang on functions that launched an exception.

Juan Luis Cano <https://github.com/Juanlu001>_ modernized the infrastructure and helped with various things.

Martin Becker <https://github.com/mgbckr>_ added PSS and USS tracking via the psutil backend.

========= License

BSD License, see file COPYING for full text.

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot