Splitting Large File Into Smaller File Giving Memory Error

This is the python code i'm using. I have a 5gb file which i need to split in around 10-12 files according to line numbers. But this code gives a memory error. Please can someone t

Solution 1:

Just use groupby, so you don't need to create 386972 iterators:

from itertools import groupby

n = 386972
with open('reviewsNew.txt','rb') as f:
    for idx, lines in groupby(enumerate(iterable), lambda (idx, _): idx // n):
        with open('small_file_{0}'.format(idx * n), 'wb') as fout:
            fout.writelines(l for _, l in lines)