Every for loop you've written in this section has relied on a mechanism worth understanding directly: Python's iterator protocol. A generator is a simple, common way to build your own.
Lists, tuples, dictionaries, and strings are all iterable — for x in ... works on all of them because each knows how to produce its items one at a time when asked, via iter() and next() under the hood:
numbers = [1, 2, 3]
iterator = iter(numbers)
print(next(iterator)) # 1
print(next(iterator)) # 2
print(next(iterator)) # 3
# next(iterator) # StopIteration — nothing leftA for loop is really just this pattern, wrapped in convenient syntax — it calls next() repeatedly until it runs out.
Building a full list to loop over it once wastes memory if the list is huge, or impossible if it's infinite. A generator function — using yield instead of return — produces values one at a time, on demand, without ever holding the whole sequence in memory at once:
def count_up_to(n):
i = 1
while i <= n:
yield i
i += 1
for number in count_up_to(5):
print(number) # 1, 2, 3, 4, 5 — computed one at a time, not built as a list first
A generator expression is the lazy cousin of the list comprehension from earlier in this section — swap the square brackets for parentheses:
squares_list = [n ** 2 for n in range(1000000)] # builds all million values right now
squares_gen = (n ** 2 for n in range(1000000)) # builds nothing yet — values come one at a timeList vs. generator — when each makes sense
Use a list when you need to loop over the data more than once, check its length, or index into it. Reach for a generator when you're processing a large or open-ended sequence just once, in order — reading a huge file line by line is a classic real-world case.
This is the gotcha that catches almost everyone the first time: once a generator has been fully looped over, it's exhausted — looping over it again produces nothing, silently, with no error:
squares = (n ** 2 for n in range(3))
print(list(squares)) # [0, 1, 4]
print(list(squares)) # [] — already exhausted, nothing left to produceA list can be looped over as many times as you like; a generator is single-use by design. If you need the data more than once, either convert it to a list (list(...)) or build a fresh generator each time you need it.