Generators provide a simple, elegant way to create iterators using regular functions. By replacing the return statement with yield, Python handles the entire Iterator Protocol automatically behind the scenes.
1. What is yield?
When a function contains the yield keyword, Python automatically compiles it into a Generator Function. Calling this function does not execute its body immediately; instead, it returns a Generator Object.
return: Terminates the function completely and returns a value, destroying its local stack frame.
yield: Produces a value, pauses function execution, and preserves its entire execution context (all local variables and instruction pointer) in memory until the next value is requested.
1defsimple_number_stream():
2print("-> Generator started")
3yield10
4print("-> Resumed at second yield")
5yield20
6print("-> Resumed at third yield")
7yield30
8print("-> Generator finished")
9
10# 1. Calling the function returns a generator object (no execution yet!)
11gen = simple_number_stream()
12print(type(gen))# <class 'generator'>
13
14# 2. Each next() call executes until the next yield
15print(next(gen))# Prints: "-> Generator started", then returns 10
16print(next(gen))# Prints: "-> Resumed at second yield", then returns 20
17print(next(gen))# Prints: "-> Resumed at third yield", then returns 30
18
19# 3. Next call raises StopIteration automatically
20try:
21next(gen)
22except StopIteration:
23print("Stream ended.")
2. Memory Efficiency: Processing Huge Files
The superpower of generators is lazy evaluation—values are computed strictly on-demand.
Consider reading a 10 GB log file. Reading it into a standard list with .readlines() will immediately exhaust system RAM with an OutOfMemory crash. A generator reads one line at a time with near-zero memory footprint:
1defread_large_log(filepath):
2"""Streams file lines lazily without reading whole file into RAM."""
3withopen(filepath,"r", encoding="utf-8")as f:
4for line in f:
5if"ERROR"in line:
6yield line.strip()
7
8# Memory usage is constant (a few kilobytes) even for 100 GB files!
9for error_line in read_large_log("server.log"):
10print(f"Logged Incident: {error_line}")
3. Building Generator Pipelines
You can chain multiple generators together like unix pipes (|) to form clean, modular data processing pipelines:
4. Advanced Generator Features: send() and close()
Generators can also receive data back from the caller via .send(value):
1defrunning_average():
2"""Calculates running average of values sent into it."""
3 total =0.0
4 count =0
5 average =None
6
7whileTrue:
8# yield produces current average AND receives incoming value
9 val =yield average
10if val isNone:
11break
12 total += val
13 count +=1
14 average = total / count
15
16avg_calculator = running_average()
17
18# Prime the generator (advance to first yield)
19next(avg_calculator)# Returns None
20
21print(avg_calculator.send(10))# 10.0
22print(avg_calculator.send(20))# 15.0
23print(avg_calculator.send(30))# 20.0
24
25# Terminate generator cleanly
26avg_calculator.close()
5. Comparison: Regular Function vs. Generator
Attribute
Regular Function
Generator Function
Keyword
Uses return
Uses yield
Execution
Executes to completion in one call
Pauses and resumes on demand
Return Value
Single value or collection
Generator object
Memory
Stores entire result set in RAM
Generates values lazily one at a time
Protocol
Not iterable by default
Implements Iterator Protocol natively
⚠️ Common Intermediate Mistakes & Gotchas
1. Re-using an Exhausted Generator
A generator can only be consumed once. Once it raises StopIteration, iterating over it again yields zero items silently. Create a new generator instance if you need a fresh pass.
2. Calling a Generator Function Without Iterating
Calling a generator function returns a generator object; it does not execute the function body until next() is called or a for loop begins.
💻 Try It Yourself: Chunked Data Generator
Scenario
Build a generator function chunk_stream(iterable, chunk_size) that yields fixed-size sublists lazily without loading the entire collection into memory.
Complete Solution
1defchunk_stream(iterable, chunk_size):
2 chunk =[]
3for item in iterable:
4 chunk.append(item)
5iflen(chunk)== chunk_size:
6yield chunk
7 chunk =[]
8if chunk:
9yield chunk
10
11# Test chunk generator
12numbers =list(range(1,11))
13for batch in chunk_stream(numbers,3):
14print(f"Batch: {batch}")
Expected Output
1Batch: [1, 2, 3]
2Batch: [4, 5, 6]
3Batch: [7, 8, 9]
4Batch: [10]
Interactive Quiz• Knowledge Check
Practice Quiz
Question 1
What does Python return when a function containing a yield statement is invoked?
Question 2
How does yield differ from return?
Question 3
What exception is automatically raised when a generator function reaches its end or an empty return?
Question 4
Why are generators essential when parsing massive datasets (such as a 50 GB log file)?
Question 5
Which generator method allows sending a value back into the generator function at the point of yield?
🚀 What's Next?
In the next lesson, we will continue your intermediate Python journey with Generator Expressions (7: Iterators and Generators).