Python For Loop Extending

I wonder why the for loop is not extending it's iteration: for link in frontLinks: #try: getCurlink = self.getHref(link) # get current site links

Solution 1:

The for loop evaluates its expression once, to get an iterator. Later, you re-bind the name frontLinks to be a new list. The new list won't have anything to do with the for loop.

Although it is tricky to modify a list while iterating over it, it's OK to add to the end of the list, it will work. Change your last line to this:

frontLinks.extend(getCurlink)

Solution 2:

You will not see changes in a list while iterating over it.

You need to use something like:

while i < len(frontLinks):
     link = frontLinks[i]
     ...
     if condition:
       frontLinks.append(item)
     i += 1

Solution 3:

Although you can append to a list while iterating, I would use one list and a deque (you can use a list instead of deque, just replace todo.popleft() with todo.pop(0)):

from collections import deque

done = []
todo = deque(frontLinks) # your initial frontLinkswhile todo:
    link = todo.popleft() # take the first element from todo
    getCurlink = self.getHref(link) # get current site linksif getCurlink:
        todo.extend(getCurlink) # extend the todo list
    done.append(link)

This way you always have two collections with clear roles: todo and done. You can stop/resume the iteration, dump/load the current state, etc. without having to restart the whole process.