Python Multiple Subprocess With A Pool/queue Recover Output As Soon As One Finishes And Launch Next Job In Queue

I'm currently launching a subprocess and parsing stdout on the go without waiting for it to finish to parse stdout. for sample in all_samples: my_tool_subprocess = subprocess.P

Solution 1:

ThreadPool could be a good fit for your problem, you set the number of worker threads and add jobs, and the threads will work their way through all the tasks.

from multiprocessing.pool import ThreadPool
import subprocess


defwork(sample):
    my_tool_subprocess = subprocess.Popen('mytool {}'.format(sample),shell=True, stdout=subprocess.PIPE)
    line = Truewhile line:
        myline = my_tool_subprocess.stdout.readline()
        #here I parse stdout..


num = None# set to the number of workers you want (it defaults to the cpu count of your machine)
tp = ThreadPool(num)
for sample in all_samples:
    tp.apply_async(work, (sample,))

tp.close()
tp.join()

Solution 2:

well as i understood your question your problem is that the result of the first process after its finished is supplied to the second process, then to the third and so on, to achieve this you should import threading module and use the class Thread:

proc = threading.Thread(target=func, args=(func arguments) # Thread classproc.start()                                   # starting the threadproc.join()                                    # this ensures that the next thread does no 

start until the previous one has finished.....

Solution 3:

well if this is the case you should write the same code above without proc.join() in this case the main thread (main) will start the other four threads, this the case that multithreading in a single process (in other words no benefits of multicore processor) to benefit from multicore processor you should use the multiprocessing module like this:

proc = multiprocessing.Process(target=func, args=(funarguments))      
proc.start()

this way each would be a separate process and separate processes can run completely independently of one another