How To Get Chunks Of Elements From A Queue?

I have a queue from which I need to get chunks of 10 entries and put them in a list, which is then processed further. The code below works (the 'processed further' is, in the examp

Solution 1:

You could use iter twice: iter(q.get, 'END') returns an iterator which can iterate over the values in the queue until 'END' is returned by q.get().

Then you could use the grouper recipe

iter(lambda: list(IT.islice(iterator, 10)), [])

to group the iterator into chunks of 10 items.

import itertools as IT
import multiprocessing as mp

q = mp.Queue()
for i inrange(22):
    q.put(i)
q.put("END")

iterator = iter(q.get, 'END')
for chunk initer(lambda: list(IT.islice(iterator, 10)), []):
    print(chunk)

yields

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
[20, 21]

Solution 2:

With Python 3.8 you can use the walrus operator.

import threading
from itertools import islice
from queue import Queue

defproducer(queue, _end):
    for i inrange(10, 100):
        queue.put(i)
    else:
        queue.put(_end)

defconsumer(queue, _end):
    iterator = iter(queue.get, _end)
    # Consumes the queue in batcheswhile batch := list(islice(iterator, 10)):
        print(batch)

defmain():

    queue = Queue()
    _end = object()

    t1 = threading.Thread(target=consumer, args=(queue, _end))
    t2 = threading.Thread(target=producer, args=(queue, _end))

    t1.start()
    t2.start()

    t2.join()
    t1.join()

if __name__ == '__main__':
    main()

In[2]: main()                                                                  
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19][20, 21, 22, 23, 24, 25, 26, 27, 28, 29][30, 31, 32, 33, 34, 35, 36, 37, 38, 39][40, 41, 42, 43, 44, 45, 46, 47, 48, 49][50, 51, 52, 53, 54, 55, 56, 57, 58, 59][60, 61, 62, 63, 64, 65, 66, 67, 68, 69][70, 71, 72, 73, 74, 75, 76, 77, 78, 79][80, 81, 82, 83, 84, 85, 86, 87, 88, 89][90, 91, 92, 93, 94, 95, 96, 97, 98, 99]