Python Multiprocessing: No Performance Gain With Multiple Processes

Using multiprocessing, I tried to parallelize a function but I have no performance improvement: from MMTK import * from MMTK.Trajectory import Trajectory, TrajectoryOutput, Snapsho

Solution 1:

Pool.apply is a blocking operation:

[Pool.apply is the] equivalent of the apply() built-in function. It blocks until the result is ready, so apply_async() is better suited for performing work in parallel ..

In this case Pool.map is likely more appropriate for collecting the results; the map itself blocks but the sequence elements / transformations are processed in parallel.

---

It addition to using partial application (or manual realization of such), also consider expanding the data itself. It's the same cat in a different skin.

data = ((trajectory, r) for r in [range(0,2001), ..])
result = pool.map(.., data)

This can in turn be expanded:

def apply_data(d):
    return calpha_2dmap_mult(*d)

result = pool.map(apply_data, data)

The function (or simple argument-expanded proxy of such of such) will need to be written to accept a single argument but all the data is now mapped as a single unit.