JavaScript Streams: From Beginner to Expert

JS Syntax

Binary

2025 Apr 131493 words·8 min read

Streams in JavaScript are an elegant pattern for handling data, allowing us to process data as if we were dealing with a flowing stream—bit by bit, rather than loading everything all at once. Imagine the water in a stream, continuously flowing; this is the approach streams take, enabling efficient handling of videos, large files, and real-time data, even in resource-constrained environments.

Why Streams Are Needed

Do you remember the last time you tried to upload a large video file in a browser? The waiting, the freezing, and even the crashes can be quite frustrating. This is where the limitations of the traditional "load everything at once" approach become apparent.

When dealing with large files, the traditional method loads the entire file into memory at once, like pouring a bucket of water into a small cup—overflow is inevitable.

I encountered this issue in a project where users were uploading large files, causing the server's memory usage to spike instantly, eventually leading to a complete service outage. In such scenarios, stream programming is clearly the most appropriate solution. Let's compare the traditional method with stream processing:

// Traditional method - Load the entire file into memory
fs.readFile('huge-video.mp4', (err, data) => {
  if (err) throw err;
  // At this point, the data variable contains the entire file content
  processFile(data);
});

// Stream processing - Read the file in chunks
const readStream = fs.createReadStream('huge-video.mp4');
readStream.on('data', (chunk) => {
  // Process a small portion of data at a time
  processChunk(chunk);
});

After using stream programming, you'll notice that the server's memory usage drops significantly, and the freezing seems to never have occurred. This is the power of stream programming.

In fact, the strength of streams lies in "processing on demand"—you don't have to wait for all the data to start working. It's like sipping coffee while continuously refilling the cup, rather than waiting for a whole pot of coffee to brew before drinking. With streams, the memory footprint for a 1GB file might only be 64KB (default chunk size), a significant efficiency improvement compared to the traditional method's 1GB usage.

Even if the client has enough memory, processing large files all at once can cause noticeable performance issues. The main thread gets blocked, the user interface may freeze, and the overall responsiveness of the application is affected. Similarly, in a Node.js server environment, this blocking directly impacts the ability to handle other requests.

Moreover, modern applications often need to handle real-time data streams, such as live video streaming, real-time log analysis, or financial transaction data. Traditional data processing methods can no longer meet the demand for "processing as it is generated," leading to increased data processing delays and reduced real-time performance.

Therefore, what we need is a data processing mechanism that can quickly handle real-time data, has low memory costs, and is feature-rich. This mechanism is implemented in JavaScript as streams.

Basic Concepts of Streams

Note: Unless otherwise specified, the streams mentioned below refer to streams in the Node.js environment by default.

Streams provide a new paradigm for data processing, allowing us to split data into small chunks (chunks) and process them step by step, just like a continuous flow of water.

The best way to understand the concept of streams is through analogies from everyday life. Imagine water flowing through a pipe: water flows from the source (such as a reservoir) through the pipe system, eventually reaching the destination (such as a faucet). Throughout the process, the water is continuously flowing, not all delivered at once.

Technically, a stream is an abstract representation of an asynchronous sequence of data. It provides a standardized interface for handling continuously arriving data chunks. These data chunks can be:

Binary data in files
HTTP request bodies
Terminal input
Any segmentable continuous data

In Node.js, the stream module provides the core API for stream processing. Almost all I/O operations are built on streams, from the file system to HTTP requests and responses, streams are everywhere.

const { Readable, Writable, Transform, Duplex } = require('stream');

Stream processing enables a "pipeline" operation of data, where producers and consumers can work in parallel. Data can be processed immediately as it is generated, greatly improving the system's responsiveness and real-time performance. This is particularly prominent in scenarios such as video transcoding and real-time data analysis.

Performance Comparison Example:

Processing Method	Memory Usage	Processing Delay	Applicable Scenarios
Traditional Method	~1GB	High	Small Files
Stream Processing	~64KB(default chunk size)	Low	Large Files/Real-time Data

The main types of streams in Node.js are (we will cover these in detail in the next chapter):

Readable Streams
Writable Streams
Duplex Streams
Transform Streams

Each type of stream is an instance of EventEmitter, meaning they communicate through events. Common stream events include:

data - Triggered when there is data available to read
end - Triggered when no more data is available to read
error - Triggered when an error occurs
finish - Triggered when all data has been flushed to the underlying system

In the following chapters, we will delve into the various types of streams in JavaScript, their usage methods, and best practices, helping you master this powerful data processing tool.

The Four Basic Types of Streams

Readable Streams

Readable streams are the source of data, like a reservoir or water source. File reading, HTTP requests, and user input are typical scenarios for readable streams. To describe it professionally, a readable stream is a producer of data, representing a source of data. Its core characteristics are:

Data can only be read from the stream, not written to it
Supports two data consumption modes: flowing mode and paused mode
Automatically handles the backpressure mechanism
Data can be piped to writable streams

Typical Use Cases:

Reading data from files
Receiving HTTP request bodies
Reading database query results
Any scenario that requires sequential reading of data

Here are some common implementations of readable streams:

javascript

const fs = require("fs");

// 1. File Read Stream
const fileStream = fs.createReadStream("./data.txt");

// 2. HTTP Request Stream
const http = require("http");
http.createServer((req, res) => {
  // req is a readable stream
  req.on("data", (chunk) => {
    console.log(`Received ${chunk.length} bytes of data`);
  });
});

// 3. Custom Readable Stream
const { Readable } = require("stream");
const myReadable = new Readable({
  read(size) {
    // Custom data generation logic
    this.push("some data");
    this.push(null); // Indicates the end of data
  },
});

Writable Streams

Writable streams are the consumers of data, representing the destination of data. Its characteristics are:

Data can only be written to the stream, not read from it
Supports buffering to handle differences in write speeds
Provides a drain event to handle backpressure
Can receive data piped from readable streams

Typical Use Cases:

Writing to files
Sending HTTP responses
Writing to databases
Any scenario that requires sequential writing of data

Here are some common implementations of writable streams:

javascript

// 1. File Write Stream
const fileWriter = fs.createWriteStream("./output.txt");

// 2. HTTP Response Stream
http.createServer((req, res) => {
  // res is a writable stream
  res.write("Hello World");
  res.end();
});

// 3. Custom Writable Stream
const { Writable } = require("stream");
const myWritable = new Writable({
  write(chunk, encoding, callback) {
    console.log("Writing data:", chunk.toString());
    callback(); // Indicates the write is complete
  },
});

Duplex Streams

Duplex streams are bidirectional streams, implementing both readable and writable interfaces, with both reading and writing capabilities. Its characteristics are:

Can read and write
The read and write ends are independent
Commonly used in bidirectional communication scenarios

Here are some common implementations of duplex streams, such as TCP socket servers:

javascript

const net = require('net');

// Create a TCP server
const server = net.createServer((socket) => {
  // socket is a duplex stream
  socket.write('Welcome to the connection!\n');
  
  socket.on('data', (data) => {
    console.log('Received data from client:', data.toString());
    socket.write('Server received your message\n');
  });
});

server.listen(8080, () => {
  console.log('Server started');
});

// Custom Duplex Stream
const { Duplex } = require('stream');
const myDuplex = new Duplex({
  write(chunk, encoding, callback) {
    console.log('Writing:', chunk.toString());
    callback();
  },
  
  read(size) {
    this.push('Data from duplex stream\n');
    this.push(null);
  }
});

Transform Streams

Transform streams are a special type of duplex stream, specifically used for data transformation. Its characteristics are:

Can read and write simultaneously
Data written to the write end is transformed and appears at the read end
Commonly used in data format conversion, encryption/decryption, etc.

There are many types of transform stream implementations, such as compression/decompression streams, encryption streams, etc.:

javascript

const { Transform } = require("stream");
const zlib = require("zlib");

// 1. Custom Transform Stream: Convert input to uppercase
class UpperCaseTransform extends Transform {
  _transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  }
}

// Usage Example
process.stdin.pipe(new UpperCaseTransform()).pipe(process.std...