4. Profiling and Optimization

Full list of references in Chapter 4 of the IPython Cookbook, the definitive guide to high-performance scientific computing and data science in Python, by Dr. Cyrille Rossant, Packt Publishing, 400 pages, 2014.

Profiling

• Program optimization on Wikipedia.
• Python's timeit module.
• Python's profile module.
• easy_profiler that simplifies the use of Python profiling tools.
• line_profiler, a line-by-line profiler, for Python 2 only.
• How to deal with @profile in profiled code.
• RunSnakeRun, a viewer for profiling results.

Memory profiling

• memory_profiler package.
• psutil package, required by memory_profiler.
• Guppy-PE, a memory profiling package.
• PySizer, a memory profiler for Python.
• Pympler, another memory profiler.

Code tracing

• Python's trace module.
• Online Python Tutor, a great educational interactive tool to visualize the step-by-step execution of Python code.
• Some Python profiling tools.

NumPy code optimization

• Broadcasting rules and examples.
• Array interface in NumPy.
• Locality of reference.
• Internals of NumPy in the SciPy lectures notes.
• The complete list of NumPy routines is available on the NumPy Reference Guide.
• List of indexing routines.
• NumPy Performance Tricks.
• 100 NumPy exercices.

Memory mapping

• Documentation of memmap.

Sparse matrices

• SciPy lecture notes about sparse matrices.
• Reference documentation on sparse matrices.

HDF5

• PyTables
• h5py
• Partial list of HDF5 users
• HDF5 chunking.
• PyTables optimization guide, a must-read for PyTables users.
• In-kernel queries in PyTables.
• An alternative to PyTables and HDF5 might come from the Blaze project, still in early development at the time of writing.
• Difference between PyTables and h5py, from the perspective of h5py.
• Difference between PyTables and h5py, from the perspective of PyTables.