๐ Day 19 : Generators and Iterators
๐ฏ Enterprise Objective
Generators are capable of much more than simple iteration. Today we master yield from for recursive pipelines, Coroutines for two-way data streaming (.send()), and the elegant @contextmanager to build our own robust with blocks.
๐ Strategic Overview
| # | Topic | Concept |
|---|---|---|
| 1 | Yield From | Chaining & recursion |
| 2 | Coroutines | .send() data inwards |
| 3 | Contexts | Building with blocks |
1. Advanced Generators : yield from & State
We've learned that yield pauses a function. Advanced generators can yield from OTHER generators using yield from, allowing you to chain pipelines cleanly. Generators also maintain complex internal state across iterations.
def sub_gen():
yield 1
yield 2
def main_gen():
yield from sub_gen() # Chains the sub-generator seamlessly
yield 3๐ผ Why Data Analysts Care
โข Tree Traversal: Flattening complex nested JSON structures recursively
โข Data Pipelining: Chaining multiple generator functions (extract -> transform -> load)
โ ๏ธ Infinite Recursion
When using yield from recursively, ensure you have a base case that stops yielding, just like standard recursion, to prevent a RecursionError.
๐งช Concept Checks: Yield From
Q1. Write a generator gen_a() yielding 1, 2. Write gen_b() yielding yield from gen_a() then 3. Test it.
Q2. Create a generator that yields numbers 1 to 3, then yields from range(4, 6). Print as list.
Q3. Write a generator that maintains state: a running total. It takes a list of numbers and yields the running total at each step.
Q4. Try to call len() on a generator. Catch the TypeError. Why doesn't it work?
Q5. Explain why yield from is cleaner than for x in sub_gen(): yield x.
2. Coroutines : Sending Data to Generators
Generators don't just produce data; they can consume it! By using the .send() method, you can pass data into a paused generator. A generator used this way is called a Coroutine.
def accumulator():
total = 0
while True:
value = yield total # Pauses here, waits for .send()
total += value๐ผ Why Data Analysts Care
โข Event Loops: Coroutines are the historical foundation of Python's asyncio for asynchronous programming
โข State Machines: Building state machines that wait for external inputs to transition states
๐ง Pro Tip
Before you can .send() a value into a coroutine, you MUST 'prime' it by calling next(gen) or gen.send(None) so it reaches the first yield statement.
๐งช Concept Checks: Coroutines
Q1. Write a coroutine greeter() that waits for a name (yield) and prints "Hello [name]". Prime it, then send "Alice".
Q2. Modify the averager above. Send 5, 10, 15. Print the final average.
Q3. What happens if you try to send() to a coroutine without calling next() first? Catch the TypeError.
Q4. Send None to the averager coroutine above to break the loop. Catch the StopIteration.
Q5. Explain how a coroutine differs fundamentally from a standard function (hint: pausing vs returning).
3. Context Managers : Building Your Own 'with'
You know how to use with open(). You can build your own Context Managers using the contextlib module and a generator. This is the cleanest way to handle setup and teardown logic.
from contextlib import contextmanager
@contextmanager
def my_timer():
start = time.time()
yield # Code inside the 'with' block runs here
print(f'Took {time.time() - start}s')๐ผ Why Data Analysts Care
โข Database Connections: Yielding a DB connection and ensuring conn.close() happens after the with block
โข Temporary State: Temporarily changing a directory, then switching back safely
๐ง Pro Tip
If an exception occurs inside the with block, it will be raised at the yield statement in your context manager. Wrap the yield in a try/finally to ensure cleanup!
๐งช Concept Checks: Contextlib
Q1. Import contextmanager. Write a @contextmanager called tag(name) that prints .
Q2. Use your tag("div") in a with block and print "Hello" inside it.
Q3. Write a context manager ignore_errors that yields inside a try/except block catching Exception. Use it to ignore 1/0.
Q4. Why must you use try/finally around the yield in a context manager for resource cleanup?
Q5. Does a context manager have to yield a value? (e.g., yield f vs just yield). Explain.
๐ ๏ธ Professional Practice Tasks
Theory is useless without muscle memory. Complete these tasks to solidify your understanding.
Task 1 (Recursive File Finder): Write a generator find_txt_files(path) using pathlib. If a path is a directory, iterate over its children. If a child is a directory, yield from find_txt_files(child). If it's a .txt file, yield it.
Task 2 (Filter Coroutine): Write a coroutine filter_even(target_list). In an infinite loop, wait for a value using val = yield. If val is even, append it to target_list.
Task 3 (Custom Open Context): Write a @contextmanager safe_open(file, mode) that prints 'Opening', yields the file object, and prints 'Closing' in a finally block.
Task 4 (Stateful Generator): Write a generator moving_average(window_size). It keeps an internal list of the last N values. Yield the average of the list at each step. Test by sending it 1,2,3,4,5.
Task 5 (HTML Builder): Write nested context managers to build HTML strings. with tag('body'): with tag('h1'): print('Title') should result in valid HTML structure.
๐ป Pure Coding Interview Questions
Q1.
What is the yield from keyword used for in Python 3.3+?
Q2.
Explain how coroutines in Python differ from traditional threads.
Q3.
Write a simple coroutine and explain why you must call next(coro) before sending a value.
Q4.
What happens if a generator used as a context manager does not have a try/finally block and an exception occurs in the with block?
Q5.
How do you return a value from a generator (i.e., when StopIteration is raised)?
Q6.
Implement a simple Event Broker using coroutines (subscribers wait for messages via yield).
Q7.
Explain the contextlib.suppress context manager. Write a custom version of it.
Q8.
Write a generator that yields chunks of a specific size from a larger list.
Q9.
What is the difference between implementing a context manager using classes (enter/exit) vs @contextmanager?
Q10.
Write a context manager that temporarily redirects sys.stdout to a file.
Q11.
Write a coroutine that acts as a grep command: it receives lines of text and prints those that contain a specific substring.
Q12.
Explain what generator.throw() does.
Q13.
Explain what generator.close() does and how a generator can catch GeneratorExit.
Q14.
How would you use yield from to flatten a deeply nested dictionary?
Q15.
Write a context manager that acquires a lock from the threading module and releases it safely.
Q16.
Explain how asyncio built upon the concept of generator-based coroutines.
Q17.
Write a generator that produces an infinite sequence of prime numbers.
Q18.
What is the memory complexity of traversing a massive tree structure using yield from?
Q19.
Write a coroutine broadcast that takes a list of other coroutines and sends any received value to all of them.
Q20.
Explain the performance differences between [x for x in gen] and list(gen).
Q21.
Write a context manager that temporarily modifies an environment variable and restores it afterward.
Q22.
How do you chain multiple generators together into a data pipeline?
Q23.
Write a generator that acts like a sliding window over an iterable.
Q24.
Explain why generators cannot be pickled (serialized) natively.
Q25.
Write a @contextmanager to mock a database transaction (print 'begin', yield, print 'commit', or 'rollback' on exception).
๐ Day 19 Executive Summary
| # | Topic | Key Takeaway |
|---|---|---|
| 1 | Yield From | Flattens nested generator logic effortlessly |
| 2 | Coroutine | Generators can pause to receive data, not just emit it |
| 3 | Context | @contextmanager makes setup/teardown logic perfectly clean |
โ Instructor's End-of-Day Checklist
โข [ ] I can use yield from to delegate to another generator.
โข [ ] I understand how to .send() data to a coroutine.
โข [ ] I can write a custom @contextmanager.