Can I Store An Iterator In A File Which I Can Read From Later? Will This Reduce Space Consumption?

Let's say I have a very large integer, around the order of 10**200. Now storing the integer in a file will take some amount of space. If I convert it into an iterator using yield,

Solution 1:

I'm assuming this is what you'd like to do:

deff():
    yield10**200

Then save f() in a file. The answer is no, that won't work. An generator like f() (note: that's generator, not iterable) cannot be pickled or otherwise serialized unless you turn it into a custom iterator with special-purpose pickling support.

Solution 2:

Building on larsmans answer, a custom iterator can be built to do this:

classmy_large_num(object):def__init__(self):
        self.num_iterations = 0def__iter__(self):
        returnselfdefnext(self):
        ifself.num_iterations < 1:
            self.num_iterations += 1return10**200else:
            raise StopIteration()

You can then:

importpicklepickled_repr= pickle.dumps(my_large_num())
restored_object = pickle.loads(pickled_repr)
sum(restored_object)

This works because underneath, iterable objects have a next() function which raises StopIteration when done. All we're doing is creating a class that implements this functionality.

In this specific case, regardless of the fact you have stored the class in a file, you still need to perform the iteration, and thus store 10**200 in memory, so you gain no functionality except generating the number on demand, which you can do without serializing the object.

You might be thinking of mmap style space saving. This maps memory to a file - note however this still affects the usable memory of your program.

Solution 3:

You can use the Shelve Module to store this.

A “shelf” is a persistent, dictionary-like object. The difference with “dbm” databases is that the values (not the keys!) in a shelf can be essentially arbitrary Python objects — anything that the pickle module can handle.

The following types can be pickled

Solution 4:

An integer of the value 10**200 does not take a large amount of space. Encoded in base 10 ASCII, that takes only 201 characters. If you're willing to store your data in binary, then you're looking at only 85ish

---

If you mean "iterable", that doesn't make much sense either - an iterable is essentially a function, and you already have the function saved - it's in the source file.