Python Generator Of Generators?

I wrote a class that reads a txt file. The file is composed of blocks of non-empty lines (let's call them 'sections'), separated by an empty line: line1.1 line1.2 line1.3 line2.1

Solution 1:

Python 2:

map(list, generator_of_generators)

Python 3:

list(map(list, generator_of_generators))

or for both:

[list(gen) for gen in generator_of_generators]

---

Since the generated objects are generator functions, not mere generators, you'd want to do

[list(gen()) for gen in generator_of_generator_functions]

If that doesn't work I have no idea what you're asking. Also, why would it return a generator function and not a generator itself?

---

Since in the comments you said you wanted to avoid list(generator_of_generator_functions) from crashing mysteriously, this depends on what you really want.

• It is not possible to overwrite the behaviour of list in this way: either you store the sub-generator elements or not

• If you really do get a crash, I recommend exhausting the sub-generator with the main generator loop every time the main generator iterates. This is standard practice and exactly what itertools.groupby does, a stdlib generator-of-generators.

eg.

defmetagen():
    definnergen():
        yield1yield2yield3for i inrange(3):
        r = innergen()
        yield r

        for _ in r: pass

• Or use a dark, secret hack method that I'll show in a mo' (I need to write it), but don't do it!

As promised, the hack (for Python 3, this time 'round):

from collections import UserList
from functools import partial


defobjectitemcaller(key):
    definner(*args, **kwargs):
        try:
            returngetattr(object, key)(*args, **kwargs)
        except AttributeError:
            returnNotImplementedreturn inner


classListable(UserList):
    def__init__(self, iterator):
        self.iterator = iterator
        self.iterated = Falsedef__iter__(self):
        return self

    def__next__(self):
        self.iterated = Truereturnnext(self.iterator)

    def_to_list_hack(self):
        self.data = list(self)
        del self.iterated
        del self.iterator
        self.__class__ = UserList

for key in UserList.__dict__.keys() - Listable.__dict__.keys():
    if key notin ["__class__", "__dict__", "__module__", "__subclasshook__"]:
        setattr(Listable, key, objectitemcaller(key))


defmetagen():
    definnergen():
        yield1yield2yield3for i inrange(3):
        r = Listable(innergen())
        yield r

        ifnot r.iterated:
            r._to_list_hack()

        else:
            for item in r: passfor item in metagen():
    print(item)
    print(list(item))
#>>> <Listable object at 0x7f46e4a4b850>#>>> [1, 2, 3]#>>> <Listable object at 0x7f46e4a4b950>#>>> [1, 2, 3]#>>> <Listable object at 0x7f46e4a4b990>#>>> [1, 2, 3]list(metagen())
#>>> [[1, 2, 3], [1, 2, 3], [1, 2, 3]]

It's so bad I don't want to even explain it.

The key is that you have a wrapper that can detect whether it has been iterated, and if not you run a _to_list_hack that, I kid you not, changes the __class__ attribute.

Because of conflicting layouts we have to use the UserList class and shadow all of its methods, which is just another layer of crud.

Basically, please don't use this hack. You can enjoy it as humour, though.

Solution 2:

A rather pragmatic way would be to tell the "generator of generators" upon creation whether to generate generators or lists. While this is not as convenient as having list magically know what to do, it still seems to be more comfortable than having a special to_list function.

defgengen(n, listmode=False):
    for i inrange(n):
        defgen():
            for k inrange(i+1):
                yield k
        yieldlist(gen()) if listmode else gen()

Depending on the listmode parameter, this can either be used to generate generators or lists.

for gg in gengen(5, False):
    print gg, list(gg)
printlist(gengen(5, True))