npm, Yarn, pnpm: From File Copying to Content Addressing — Three Storage Philosophies Behind Package Managers

Devops

Front-end Build Tools

8 days ago4378 words·22 min read

What You'll Learn

By the end of this article, you will be able to:

Read a node_modules directory: why is it flat? Why do some packages appear multiple times?
Understand the root cause of phantom dependencies: why can you import a package you never declared?
Choose the right package manager: grasp the essential differences between npm, Yarn, and pnpm from the storage layer up

Prerequisites

You've used at least one of npm install, yarn add, or pnpm i
You have a basic understanding of Node.js require / import
No OS-level knowledge needed (but you'll pick up some interesting filesystem concepts along the way)

Let's Start with a Real Scenario

Picture this afternoon:

You just cloned a new project. git clone done, you reflexively run npm install. The terminal starts scrolling like mad. You get up to grab some water, come back, and it's still going. When it finally finishes, you glance at Finder: node_modules is 800MB.

Your project has maybe a few dozen source files. How did dependencies eat up that much space? Is there any way to trim it down?

Then something even more puzzling happens. Somewhere in your code, you casually write:

import dayjs from 'dayjs';

And it runs fine. But when you search package.json for dayjs — it's not there. You never declared it. So where did it come from? Why does it work? And if it works without a declaration, do you even need to list all your dependencies?

You ask a coworker. They shrug: "Try deleting node_modules and reinstalling. It might not work the second time." You try. They're right — this time, dayjs throws an error.

Which leads to the third question: why would the same package.json produce different results across two installs?

These three headaches — where the space went, the ghost dependency, and non-deterministic installs — all trace back to the same root cause: what exactly does a package manager do to your filesystem when you hit install?

This article is not a tutorial on npm install --save-dev. We're going one level deeper: when you run that install command, what happens on disk? npm, Yarn, and pnpm each approached the same problem with a fundamentally different mindset.

First, the Foundation: How Does Node.js Find a Module?

Before asking "how does a package manager store things," you need to understand "how does Node.js find things." This is the starting point for everything that follows.

Say you have a file src/index.js with this line:

const _ = require('lodash');

When Node.js reaches this line, it doesn't search the internet for lodash. It does exactly one simple thing: starting from the current file's directory, walk upward one level at a time, looking for a directory called node_modules/lodash.

Think of Node.js as someone searching an office building for a room. They start on the src/ floor, peek into src/node_modules/lodash — is it there? Nope. Go up one floor to the project root, check node_modules/lodash — found it! That's the one.

If they climb all the way to the root directory / without finding anything — error: Cannot find module 'lodash'.

In code, the core logic is just a handful of lines:

function resolveModule(id, fromDir) {
  // Start from the current file's directory
  let dir = fromDir;
  while (dir !== '/') {
    // Build the candidate path: currentDir/node_modules/moduleName
    const candidate = `${dir}/node_modules/${id}`;
    // Does it exist? Return if yes, go up one level if no
    if (fs.existsSync(candidate)) return candidate;
    dir = path.dirname(dir);
  }
  throw new Error(`Cannot find module '${id}'`);
}

This code couldn't be simpler, but it encodes a crucial rule: whether a module is accessible depends entirely on where it sits in the filesystem tree — if it's "on your way up," you can reach it; if it isn't, you can't.

This rule is the foundation upon which npm, Yarn, and pnpm built all their storage strategies. All three tools are doing the same thing: assembling a node_modules directory, each in their own way, so that Node.js finds every package it needs along that upward walk.

Now that we understand the rule, let's see how the three tools solved the problem.

Stop One: npm — From Russian Dolls to Open Floors

npm v2: The Most Straightforward Approach

The earliest npm (v2 era) took the most direct route: map the dependency tree onto the filesystem, as-is.

What does that mean? Imagine your project only depends on package A, and A depends on B, and B depends on C. Your node_modules would look like this:

txt

node_modules/
└── A/
    ├── index.js
    └── node_modules/
        └── B/
            ├── index.js
            └── node_modules/
                └── C/
                    └── index.js

This is essentially a Russian nesting doll — each package tucks its own dependencies inside its own node_modules folder. Remember the rule we just learned? Node.js walks upward from the current file. So when A/index.js calls require('B'), Node.js finds B at A/node_modules/B. When B/index.js calls require('C'), Node.js finds C at B/node_modules/C. Everything lines up perfectly — each layer of the dependency tree is faithfully reflected on disk.

There's nothing wrong with this approach in terms of correctness. But the problems are easy to imagine.

Think about it: if you install Express, it pulls in roughly 30 dependencies. Each of those 30 has its own dependencies. What you end up with isn't a tidy 3-layer doll — it's a monster nesting dozens of layers deep. On Windows, where file paths are capped at 260 characters, deep node_modules paths just explode.

Even worse: disk waste. Suppose your project directly depends on both A and B, and both happen to depend on the exact same version of lodash. In npm v2's structure, lodash appears twice — once at A/node_modules/lodash, once at B/node_modules/lodash. Two identical copies of the same files, taking up double the space.

npm v2 traded simplicity for a hidden cost: the dependency tree's complexity was passed directly onto the filesystem. However deep the tree, however much duplication — the disk paid the price.

The npm team recognized this and made a major architectural change in v3.

npm v3: Everyone into the Lobby

npm v3's idea: instead of each package hoarding its own dependencies in its own little room, pull all shareable dependencies up to the root node_modules so everyone can share them.

This operation is called hoisting. Its core philosophy in one sentence: hoist when you can, nest only when you must.

Let me walk you through the process with a concrete example. Say your project depends on A and B. Both need lodash, but with different version ranges: A wants lodash@^4.0.0, B wants lodash@^3.0.0.

npm v3 processes dependencies one by one. Suppose it starts with A:

Sees that A needs lodash@^4.0.0.
Checks root node_modules — empty, no conflict.
Places lodash@4.17.21 in the root node_modules.

Then it processes B:

Sees that B needs lodash@^3.0.0.
Checks root node_modules — lodash@4.17.21 is already there.
Now the critical question: can the root's lodash@4 satisfy B's ^3.0.0 requirement?
Semver tells you ^3.0.0 means >=3.0.0 <4.0.0 — and 4.17.21 is clearly outside that range. Conflict!
So lodash@3.10.1 can't go to the root. It stays nested inside B's own node_modules.

The final structure looks like this:

txt

node_modules/
├── A/
├── B/
│   └── node_modules/
│       └── lodash@3.10.1/     # conflict — stays nested
├── lodash@4.17.21/             # successfully hoisted, shared by all

This is significantly more efficient than v2's pure nesting — lodash@4 exists only once, instead of being duplicated under every package that uses it. But it introduced three classic problems.

Problem 1: Install Order Changes Everything

Let's flip the order. What if npm v3 processes B first instead of A?

B wants lodash@^3.0.0 → root is empty, no conflict → lodash@3.10.1 goes to root node_modules.
A wants lodash@^4.0.0 → root has lodash@3.10.1 → ^4.0.0 means >=4.0.0 <5.0.0 → 3.10.1 doesn't satisfy it → conflict!
So lodash@4.17.21 gets nested inside A's own node_modules.

See what happened? Same package.json, same dependency declarations, just a different processing order — and the resulting node_modules structure is different. The lock file (package-lock.json) pins versions, but it can't pin hoisting decisions. Which package gets hoisted to root and which stays nested still depends on install order.

In mathematical terms, hoisting produces a "partially ordered" result — there's no single optimal solution. Each install order can yield a different, yet locally reasonable, arrangement.

Problem 2: Phantom Dependencies

This is the pitfall you're most likely to trip over in daily development.

Once a package gets hoisted to root node_modules, it becomes "visible" to all your project code — even if you never declared it in package.json.

Let's go back to dayjs. Say your project depends on antd, and antd internally depends on dayjs. During npm v3's hoisting process, if dayjs doesn't run into a version conflict, it will almost certainly get hoisted to root node_modules.

So your project code can just write:

import dayjs from 'dayjs'; // no error! it's sitting right there in node_modules

You never ran npm install dayjs, but it works. Until one day — antd updates, drops dayjs, and switches to a different date library. You upgrade antd, run npm install again — and your project explodes: Cannot find module 'dayjs'.

That's a phantom dependency: your code sneakily borrowed someone else's dependency, and when they took it back, you broke. The proper practice is simple: whatever you use, you declare in package.json. But npm's flat structure doesn't enforce this discipline.

Problem 3: Doppelgangers

This is a side effect of the first two problems. Different versions of the same package can coexist at different locations in your project:

txt

node_modules/
├── lodash@4.17.21/           # one copy at root
└── some-package/
    └── node_modules/
        └── lodash@3.10.1/    # another copy nested

Disk waste is the obvious issue. The more insidious one is runtime behavior: if both versions get loaded into memory, you can end up with subtle bugs — two modules maintaining their own internal state, unaware of each other's existence.

npm v3's hoisting solved a real problem (excessive nesting), but it also acknowledged a reality: between compatibility and purity, npm chose compatibility — it wanted the existing ecosystem to keep working without any changes. That choice has merit, but the price is the pitfalls we just walked through.

At this point you might wonder: with all these hoisting headaches, is there a fundamentally different way to think about this? Yarn's answer: yes.

Stop Two: Yarn PnP — "Why Do We Even Need a `node_modules` Directory?"

Asking a Different Question

npm spent years optimizing node_modules — tweaking hoisting algorithms, improving lock files — all focused on "how to organize this directory better." The Yarn team asked a more fundamental question:

If the ultimate goal is for Node.js to find each module's location — why not just tell it where each module is, rather than constructing a whole node_modules directory tree?

Let me use an analogy to help you grasp the leap in thinking.

Think of a traditional library. Books sit on shelves. To find one, you walk shelf by shelf following a call number. If a book is misplaced, you simply can't find it. node_modules is like this shelf system — a package's physical location determines whether it can be found.

Now think of a digital library. The books are still stored in a warehouse, but you don't walk the shelves. You open an electronic catalog, type the book title, and the catalog tells you: "Warehouse Section 3, Row 12." You walk straight there and grab it. You don't need to know or care what other books are nearby.

Yarn PnP (Plug'n'Play) built this "digital catalog" — it replaces the entire node_modules directory tree with a single file (.pnp.cjs).

What .pnp.cjs Actually Does

When you install dependencies with Yarn PnP, your project directory has no node_modules folder. Instead, you get:

A .pnp.cjs file (a few hundred KB)
A .yarn/cache/ directory full of .zip files

The .pnp.cjs file is, at its core, a giant lookup table. It records which packages this project needs, which zip file each package lives in, and the dependency relationships between packages. When Node.js executes require('lodash'), PnP intercepts the operation — it doesn't walk the directory tree, it does a direct table lookup:

// This is essentially what .pnp.cjs does — the actual table is much larger
// It hijacks Node.js's module loading mechanism
const packageMap = {
  'lodash': '/project/.yarn/cache/lodash-npm-4.17.21-xxxx.zip',
  'react': '/project/.yarn/cache/react-npm-18.2.0-yyyy.zip',
  // ... hundreds or thousands of entries
};

// When someone calls require('lodash'), PnP doesn't search directories
// It looks up the zip path from this table and reads the zip contents directly

This approach brings several immediate benefits:

Benefit 1: Atomic installs. Traditional npm install works like this: download a tar → extract to a temp directory → create thousands of directories and files → move to node_modules. If power cuts out mid-process, you're left with a half-baked node_modules. PnP's process: download a zip file → move it to the cache directory. That's a single rename system call — it either succeeds completely or fails completely. No "half-finished" states.

Benefit 2: No randomness. No hoisting algorithm, no "first come, first hoisted" logic. Dependencies are precisely recorded in .pnp.cjs. As long as package.json and the lock file are the same, every install produces exactly the same result.

Benefit 3: Space savings. First, zip itself compresses. More importantly — global caching. The ~/.yarn/berry/cache/ directory is shared across projects. A package you've already installed won't be downloaded again. And packages live as zip files, not as thousands of tiny files scattered on disk (filesystems are notoriously inefficient with many small files).

Benefit 4: No phantom dependencies. This follows naturally — .pnp.cjs precisely records who depends on whom. There's no concept of "accidentally hoisted to root." A package you didn't declare simply cannot be required, no matter what code you write.

So What's the Catch?

Sounds wonderful, right? But this approach has one significant real-world issue: it requires ecosystem cooperation.

Many Node.js tools (Babel, ESLint, Jest, various Webpack loaders) were built with the assumption that modules live on the filesystem. They concatenate strings to form paths and look things up:

// What traditional tools do: concatenate a path and look it up on the filesystem
const babelConfig = {
  presets: ['@babel/preset-react'] // Babel tries to require this string directly
};

In a PnP environment, this breaks — because the path node_modules/@babel/preset-react simply doesn't exist. Tools need to be updated to resolve module paths through the PnP API:

// What PnP requires tools to do instead
const { resolveRequest } = require('pnpapi');
const babelConfig = {
  presets: [
    resolveRequest('@babel/preset-react', __dirname)
  ]
};

The Yarn team knows this, of course. They provide compatibility bridges like @yarnpkg/pnpify to help traditional tools work in PnP environments. But this adds a layer of complexity and ongoing maintenance. Not every team wants to take that extra step for PnP's philosophy.

This is why Yarn v2+ split into two paths: PnP mode (zero-install, extreme performance) and node_modules compatibility mode (keeping the traditional structure). PnP is the forward-looking vision, but compatibility mode exists so today's engineering practices can transition smoothly.

Stop Three: pnpm — The Art of Linking

pnpm's Position: Can We Have Both Compatibility and Efficiency?

By now, we've seen two paths:

The npm / Yarn Classic path: keep the node_modules directory structure, use hoisting to save space. Great compatibility, but with phantom dependencies and non-determinism.
The Yarn PnP path: abandon node_modules entirely, use a lookup table to solve everything. Perfect, but requires toolchain cooperation.

pnpm asked a question that seems like a compromise but has deep engineering insight behind it: can we keep node_modules compatibility while achieving deduplication and isolation at the filesystem level?

The answer: leverage two built-in filesystem "linking" mechanisms.

To understand this, we first need to clarify two easily confused OS concepts: hard links and symbolic links. Don't worry — I'll explain them with the simplest analogies possible.

First, Understand the Two Link Types

Hard links: imagine you have a file — the actual data sits on disk. Normally, one filename points to one chunk of data. But the filesystem allows you to give the same data chunk multiple names — like one person having several nicknames that all refer to the same person. That's a hard link.

The key property: the data is only truly deleted when you remove the last name pointing to it. As long as at least one hard link exists, the data survives.

Symbolic links: think of these as road signs. A symlink isn't the data itself — it's just a slip of paper that says "the target is at such-and-such location." When you access the symlink, the OS automatically redirects you to the target. If the target gets deleted, the symlink becomes a "dead link" — the note is still there, but the place it points to is gone.

pnpm's ingenuity: hard links for deduplication (same physical data, multiple names), symbolic links for isolation (controlling who can access what).

The Global Store: Same Content, One Home

When you install lodash@4.17.21 for the very first time with pnpm, here's what happens:

Download the package files
Compute a SHA-256 hash of all those files' contents
Use that hash as the "address" and store the package in a global store (on macOS, typically ~/.pnpm-store/v3/)
The resulting path looks like ~/.pnpm-store/v3/e8/a2b4c7... — using the first few hash characters as subdirectory names to avoid too many entries in a single directory

From then on, any project that needs lodash@4.17.21 — pnpm won't download it again, and won't copy files. It simply looks up that hash "address" in the global store.

This approach is called content-addressable storage. The core philosophy: same content should have the same storage location. Your disk stores exactly one copy of lodash@4.17.21, no matter how many projects use it.

The .pnpm Directory: Your Project's "Virtual Warehouse"

Once the global store has the package, pnpm needs to connect your project to it. It doesn't put packages directly in the root node_modules. Instead, it creates a .pnpm directory inside node_modules — this is where packages actually "land."

txt

node_modules/
└── .pnpm/
    ├── lodash@4.17.21/
    │   └── node_modules/
    │       └── lodash/           ← hard link, pointing to global store
    └── react@18.2.0/
        └── node_modules/
            ├── react/            ← hard link, pointing to global store
            └── loose-envify/     ← symbolic link, pointing to loose-envify inside .pnpm

The folder name for each package in .pnpm includes not just the package name and version, but also a hash suffix — this hash is computed from the package's peer dependencies. This means the same package under different peer dependency combinations gets different directory names, ensuring proper isolation.

Root nodemodules: Only What You Ask For

So .pnpm stores everything, but what about those few package names you see directly in node_modules?

They are all symbolic links, pointing to the real locations inside .pnpm:

txt

node_modules/
├── react → .pnpm/react@18.2.0_xxx/node_modules/react   ← symbolic link
├── lodash → .pnpm/lodash@4.17.21_yyy/node_modules/lodash ← symbolic link
└── .pnpm/
    └── ... (actual homes of all packages)

Do you see the key point? The root node_modules only contains what you explicitly declared as direct dependencies — whatever you wrote in package.json is what appears at root level.

This is the secret to how pnpm prevents phantom dependencies. react depends on loose-envify, but loose-envify's symlink only exists inside .pnpm/react@18.2.0/node_modules/ — it's react's own node_modules. Your project code can't reach it, because root node_modules has no road sign pointing to it.

Only when you explicitly add loose-envify to your package.json and install it does pnpm create a corresponding symlink in root node_modules, making it accessible to your code.

That's strict dependency isolation. Your code can only see the packages you declared. Transitive dependencies stay locked inside their owners' directories.

Why Is Installation So Fast?

With a global store and hard links, installing a new project works like this:

pnpm walks the dependency tree and calculates which packages are already in the global store and which are missing
For existing packages — zero downloads, just create links
For new packages — download once, store globally, then create links
Creating links is just a few system calls (link() and symlink()) — millisecond operations

This is why pnpm i in daily development typically takes only a few seconds — most packages are already sitting in your global store from previous projects.

Side by Side: One Table to Compare All Three

Dimension	npm / Yarn Classic	Yarn PnP	pnpm
Storage method	Copy files to `node_modules`, deduplicate via hoisting	Zip archives, no `node_modules` directory, lookup table	Hard links for dedup + symbolic links for isolation
Dedup scope	Within a single project, partial	Global zip cache	Global store, content-addressed dedup
Phantom dependencies	Present (side effect of hoisting)	Absent (strict declare-to-access)	Absent (symlinks only expose direct deps)
Install determinism	Low (hoisting order depends on install order)	High (no hoisting algorithm)	High (content hashing guarantees consistency)
Ecosystem compatibility	Full (no changes needed)	Needs adaptation layer (e.g., `@yarnpkg/pnpify`)	Full (preserves `node_modules` directory structure)
Cross-project disk sharing	None (each project copies independently)	Yes (zip files shared across projects)	Yes (hard links share physical data blocks)
Install speed	Slower (extract + many small file writes + hoisting algorithm)	Fast (download zip, no small file explosion)	Very fast (cached packages only need links created)
Core philosophy	Engineering compromise: balance compatibility and efficiency	Paradigm shift: abandon filesystem conventions for determinism and performance	Engineering elegance: leverage system primitives, achieve both compatibility and efficiency

Which One Should You Choose?

Beginners or personal projects: pnpm. Space-efficient, fast, no phantom dependencies to trip you up, and zero extra configuration — it looks just like the node_modules structure you're used to.
Monorepos (one repo managing multiple packages): pnpm's workspace mode is a natural fit. pnpm-workspace.yaml is clean and simple. Cross-package references go through the workspace protocol, automatically resolved via symlinks.
Highly customized toolchain, and your team is willing to invest in adaptation: Yarn PnP's extreme performance and zero-install experience are compelling, but verify that all tools your team uses are compatible or can be bridged.
Legacy projects or CI/CD pipelines deeply dependent on npm's default behavior: npm is the most conservative — and therefore safest — choice.

There is no perfect tool, only the tool best suited to your context. Once you understand the storage mechanisms behind all three, you can make decisions based on your project's actual needs, rather than blindly believing "X is best."

Hands-On Practice

Theory is good, verification is better. The three exercises below need no special setup.

Exercise 1: Catch a Phantom Dependency in the Wild

bash

# Create a fresh npm project
mkdir test-npm && cd test-npm
npm init -y

# Install express (it pulls in many transitive dependencies)
npm install express

# Count how many folders are in root node_modules
ls node_modules | wc -l

# Now check how many packages you actually declared in package.json
cat package.json | grep -A 5 "dependencies"

You'll see far more folders in node_modules than the ones you declared. Those extras are hoisted transitive dependencies — every single one of them is a phantom dependency. You could require any of them in your code, even though you never declared them.

Exercise 2: Compare Install Speeds

bash

# Clean up previous test projects, start fresh

# npm
mkdir test-speed && cd test-speed
npm init -y && time npm install react

# yarn (install with npm install -g yarn first if needed)
rm -rf node_modules package-lock.json
time yarn add react

# pnpm (install with npm install -g pnpm first if needed)
rm -rf node_modules yarn.lock
time pnpm add react

If you've previously installed react with pnpm in other projects, the third run (pnpm) will be dramatically faster — react is already in your global store.

Exercise 3: Verify pnpm's Hard Links

bash

# Install a package with pnpm
mkdir test-link && cd test-link
pnpm init && pnpm add express

# Check the inode number of express's package.json inside .pnpm
ls -i node_modules/.pnpm/express@*/node_modules/express/package.json

# Find the same file in the global store and compare inode numbers
ls -i ~/.pnpm-store/v3/*/*/express@*/package.json

If both files show the same inode number, they're using the same physical data blocks — that's a hard link. The file "appears" in two different paths, but only one copy exists on disk.

Summary & Key Terms

Core Takeaways

The essence of Node.js module resolution: walk upward looking for node_modules — where a package is placed determines who can access it. All three package managers are strategizing around this rule.
npm v3's hoisting: solved deep nesting but paid three costs — non-deterministic structures, phantom dependencies, and disk waste from multi-versioned packages.
Yarn PnP's revolution: replaced the node_modules directory tree with a lookup table. Elegant, but requires toolchain cooperation. It forces us to reconsider: what even is a "dependency"?
pnpm's elegance: hard links for global deduplication, symbolic links for strict isolation. Fully compatible with the node_modules ecosystem while solving phantom dependencies and disk waste — finding an ingenious balance between compatibility and efficiency.

Key Terms

hoisting: the strategy of pulling transitive dependencies up to root node_modules to reduce nesting depth
phantom dependency: a dependency accessible in your code but undeclared in package.json — a side effect of hoisting
hard link: multiple filenames pointing to the same physical data blocks
symbolic link / symlink: a special file that points to another file path (a "road sign")
content-addressable storage: using a content hash as the storage address — identical content naturally deduplicates
deterministic install: the property that identical inputs always produce identical outputs

npm, Yarn, pnpm: From File Copying to Content Addressing — Three Storage Philosophies Behind Package Managers

What You'll Learn

Prerequisites

Let's Start with a Real Scenario

First, the Foundation: How Does Node.js Find a Module?

Stop One: npm — From Russian Dolls to Open Floors

npm v2: The Most Straightforward Approach

npm v3: Everyone into the Lobby

Problem 1: Install Order Changes Everything

Problem 2: Phantom Dependencies

Problem 3: Doppelgangers

Stop Two: Yarn PnP — "Why Do We Even Need a node_modules Directory?"

Asking a Different Question

What .pnp.cjs Actually Does

So What's the Catch?

Stop Three: pnpm — The Art of Linking

pnpm's Position: Can We Have Both Compatibility and Efficiency?

First, Understand the Two Link Types

The Global Store: Same Content, One Home

The .pnpm Directory: Your Project's "Virtual Warehouse"

Root nodemodules: Only What You Ask For

Why Is Installation So Fast?

Side by Side: One Table to Compare All Three

Which One Should You Choose?

Hands-On Practice

Exercise 1: Catch a Phantom Dependency in the Wild

Exercise 2: Compare Install Speeds

Exercise 3: Verify pnpm's Hard Links

Summary & Key Terms

Core Takeaways

Key Terms

Stop Two: Yarn PnP — "Why Do We Even Need a `node_modules` Directory?"