Multiprocessing Basics
Multiprocessing Basics
When executing CPU-intensive tasks—such as mathematical matrix crunching, cryptographic hashing, image transformation, or machine learning model inference—Python's Global Interpreter Lock (GIL) prevents threads from utilizing multiple CPU cores simultaneously.
The multiprocessing module bypasses the GIL entirely by spawning independent operating system processes. Each process runs its own isolated CPython interpreter instance with a dedicated memory heap, enabling true multi-core parallelism across modern hardware.
1. Process Architecture & The GIL Bypass
Unlike threads that share a single process heap and contend for the same GIL mutex, separate processes run in parallel across distinct CPU physical cores:
2. Process Start Methods: spawn vs fork
Operating systems spawn child processes differently, configured via multiprocessing.set_start_method():
| Start Method | Operating System Default | Mechanism | Characteristics |
|---|---|---|---|
spawn | Windows & macOS (default) | Launches fresh Python executable; re-imports main module | Safest; clean memory; requires if __name__ == '__main__': |
fork | Linux / POSIX | Uses OS fork() to clone parent memory address space | Blazing fast start; dangerous with open threads/locks |
forkserver | Supported on POSIX | Pre-forks clean server process to spawn children | Avoids multi-threaded fork deadlocks |
The __name__ == '__main__' Invariant
spawn mode): When a child process is spawned, the new interpreter re-imports the main Python script from the beginning. If process creation logic is not protected by if __name__ == '__main__':, the child process will spawn another child, triggering an uncontrolled infinite recursion known as a Fork/Spawn Bomb that crashes the operating system.3. Basic Process Creation with multiprocessing.Process
Visual Architecture & Process Flow
How data and code flow step-by-step
4. High-Throughput Worker Pools with multiprocessing.Pool
For embarrassingly parallel data processing, manually managing individual processes and combining results is cumbersome. multiprocessing.Pool distributes tasks across a fixed pool of worker processes and collects return values automatically:
5. Architectural Summary Table
| Feature | Details |
|---|---|
| Concurrency Type | True parallel multi-core execution (bypasses the GIL) |
| Memory Model | Isolated private address space per process (no shared heap) |
| Communication | Inter-Process Communication (IPC) via serialized pipes/queues |
| Overhead | Higher memory footprint and slower startup than threads |
| Best Workloads | CPU-bound computation, data transforms, media encoding |
| Safety Invariant | Must guard startup code with if __name__ == '__main__': |
Multiple Choice Questions
1.
Why can multiprocessing achieve true multi-core CPU parallelism in Python while threading cannot? A. multiprocessing converts Python code into assembly language. B. Each process runs its own independent CPython interpreter instance with its own private GIL, allowing simultaneous execution on separate CPU cores. C. multiprocessing only runs on Linux. D. Processes ignore the operating system scheduler.
2.
Why is the if __name__ == '__main__': guard strictly mandatory when spawning processes on Windows and macOS? A. Windows does not support Python functions. B. In spawn mode, the child process re-imports the main script; without the guard, the child would recursively spawn additional child processes in an infinite loop. C. It prevents memory leaks in the GPU. D. It is an optional code styling convention from PEP 8.
spawn start method, child processes re-import the entry point script to load target functions. Without the __name__ == '__main__': check, each child would execute the process-spawning code again, causing a spawn bomb.3.
What method on a multiprocessing.Pool instance applies a function concurrently across an iterable of inputs and blocks until all results are gathered? A. pool.apply_async() B. pool.map() C. pool.fork() D. pool.run()
pool.map(func, iterable) chops the iterable into chunks, distributes them to worker processes, and blocks until all results are gathered in order, mimicking built-in map().4.
What does process.pid represent on an active multiprocessing.Process object? A. The number of iterations executed. B. The operating system Process Identifier. C. The thread identity hash. D. The port number for network sockets.
process.pid provides the OS-level Process ID assigned by the operating system kernel to the running process.5.
What is the primary architectural trade-off of multiprocessing compared to multithreading? A. Processes cannot open files. B. Processes have higher memory overhead and inter-process communication costs because memory is isolated and data must be serialized (pickled). C. Processes cannot run on 64-bit systems. D. Processes cannot return values.
Process vs Thread
Continue learning with hands-on practice, examples, and exercises in the upcoming topic.
Related Lessons
| Previous Lesson | Next Lesson |
|---|---|
| Project: Multithreaded Downloader | Process vs Thread |
Practice Quiz
Test your understanding of this lesson with 5 questions. Each question has one correct answer.