Buffers and Streams: Readable, Writable & Pipeline for Large Files0%
Native HTTP Server vs Express Framework Philosophy

Buffers and Streams: Readable, Writable & Pipeline for Large Files

Beginner12 min read•Updated: Jul 11, 2026
Study MaterialsCheatsheet

Buffers and Streams: Readable, Writable & Pipeline for Large Files

When dealing with large files, video feeds, or high-throughput network packets, loading the entire payload into RAM via fs.readFile() will instantly exhaust server memory and crash the process. Node.js solves this with Buffers (raw binary memory) and Streams (processing data chunk-by-chunk in real time).

1. Understanding Buffers

A Buffer is a fixed-size chunk of memory allocated outside the V8 JavaScript heap. It stores raw binary bytes (octets):

JavaScript
// Allocate buffer of 10 bytes initialized with zeros
const buf = Buffer.alloc(10);
 
// Create buffer from UTF-8 string
const strBuf = Buffer.from('Hello Node.js', 'utf-8');
console.log(strBuf); // <Buffer 48 65 6c 6c 6f 20 4e 6f 64 65 2e 6a 73>
console.log(strBuf.toString('hex')); // Hexadecimal representation
console.log(strBuf.toString('base64')); // Base64 encoding

2. The 4 Fundamental Stream Types

  1. 1
    Readable Streams: Source of data to read from (fs.createReadStream, http.IncomingMessage).
  2. 2
    Writable Streams: Destination to write data into (fs.createWriteStream, http.ServerResponse).
  3. 3
    Duplex Streams: Both Readable and Writable simultaneously (TCP socket).
  4. 4
    Transform Streams: Modifies or transforms the data chunk as it passes through (zlib.createGzip, crypto cipher).

3. Streaming Huge Files with Backpressure and pipeline

Using .pipe() is convenient, but if a stream errors mid-transmission, old .pipe() calls often leak open file descriptors. Modern Node.js standardizes on stream.promises.pipeline to handle errors and cleanup automatically:

JavaScript
import fs from 'node:fs';
import zlib from 'node:zlib';
import { pipeline } from 'node:stream/promises';
 
async function compressLargeLogFile(inputPath, outputPath) {
try {
const readStream = fs.createReadStream(inputPath);
const gzipTransform = zlib.createGzip();
const writeStream = fs.createWriteStream(outputPath);
 
// pipeline manages backpressure and ensures all streams close cleanly on failure
await pipeline(readStream, gzipTransform, writeStream);
console.log('File compressed successfully with zero memory overhead!');
} catch (err) {
console.error('Pipeline failed:', err.message);
}
}
 
compressLargeLogFile('./access-5gb.log', './access-5gb.log.gz');

4. What is Backpressure?

If a Readable stream produces data faster than the Writable destination can write to disk or network (e.g. reading from an NVMe SSD at 3 GB/s and sending over a 10 Mbps internet connection), incoming chunks accumulate in RAM. Backpressure pauses the readable stream until the writable stream drains its buffer, maintaining a constant, predictable memory footprint under 30 MB regardless of file size.

Interactive Quiz

Multiple Choice Questions

Question 1
Where is the memory for a Node.js Buffer allocated?
Question 2
Why is streaming preferred over fs.readFile() when handling multi-gigabyte files in a backend API?
Question 3
What is "backpressure" in Node.js streams?
Question 4
Which built-in utility handles error cleanup and automatically tears down streams when an error occurs during piped execution?
Question 5
Which type of stream is zlib.createGzip(), which takes uncompressed bytes and outputs compressed bytes?
Next Lesson

Native HTTP Server vs Express Framework Philosophy

Continue learning with hands-on practice, examples, and exercises in the upcoming topic.

Related Lessons

Previous LessonNext Lesson
EventEmitter Pattern & Custom Event ArchitectureNative HTTP Server vs Express Framework Philosophy

Practice Quiz

Test your understanding of this lesson with 5 questions. Each question has one correct answer.

PreviousNext
PrevNext