Memory Error When Opening The File With Read()

I'm new to python and I'm editing a program where I need to open I file but it's more than 1.5 Gb so I get memory error. Code is: f=open('thumbdata3.dat','rb') tdata = f.read() f.c

Solution 1:

From the description it seems that the memory footprint is the problem here. So we can use the generators to reduce the memory footprint of the data , so that it loads the part of data being used one by one.

from itertools import chain, islice

def piecewise(iterable, n):
    "piecewise(Python,2) => Py th on"
    iterable = iter(iterable)
    while True:
        yield chain([next(iterable)], islice(iterable, n-1))

l = ...
file_large = 'large_file.txt'
with open(file_large) as bigfile:
   for i, lines in enumerate(piecewise(bigfile, l)):
      file_split = '{}.{}'.format(file_large, i)
      with open(file_split, 'w') as f:
         f.writelines(lines)