Kubernetes Node Internals — Part 5: CSI, Volumes, and Mounts on the Node

"Networking gives a Pod an IP address. Storage gives it a place to put bytes. Both feel simple from YAML. Neither is simple on the node."

By now, the main node story is already familiar.

• Part 1 mapped the stack
• Part 2 explained trust and bootstrap
• Part 3 followed Pod startup
• Part 4 explained how the node stays alive under pressure

But there is still one important path left to understand:

How does a Pod ask for storage in YAML and end up seeing a real mounted directory inside the container?

That path is where volumes, kubelet, CSI, and the Linux mount system meet.

And it is one of the easiest parts of Kubernetes to use without really understanding.

Series roadmap

1. Part 1 — The anatomy of a node
2. Part 2 — Bootstrap and the secret handshake
3. Part 3 — A pod is born
4. Part 4 — Keeping the node alive
5. Part 5 — CSI, volumes, and mounts on the node

The simple mental model first

Before we talk about CSI RPCs or mount points, start with the shortest accurate model.

When a Pod asks for a volume, Kubernetes must solve three different problems:

1. What storage object should back this request?
2. How does that storage become available on this specific node?
3. How does the container see it at /data, /cache, or some other mount path?

That is the whole storage story in one sentence:

Kubernetes chooses or creates the storage, gets it onto the node, and then mounts it into the Pod.

The reason this feels slippery is that those three steps do not all happen in the same place.

• some happen in the control plane
• some happen in storage-controller components
• some happen on the node
• and the final visible effect is a normal Linux mount

So if networking is, "Give the Pod an IP," storage is:

"Give the Pod a filesystem path backed by something real, and make it show up at exactly the right moment in startup."

Why storage feels harder than networking

Networking usually feels like a shared utility.

Storage often feels like a promise.

If a Pod says:

volumeMounts:
  - name: data
    mountPath: /var/lib/app

the application is assuming several things are already true:

• the right volume exists
• it belongs to this Pod or can be shared safely
• it is attached to the node if attachment is required
• it is mounted with the correct filesystem semantics
• the path is visible inside the container before the process starts

If any one of those fails, the Pod may stay stuck in ContainerCreating, or the container may start with the wrong filesystem view.

That is why node-side storage is such an important missing piece in a Kubernetes mental model.

The cast of characters

To keep the story clear, here are the main components and what job each one owns.

The key idea is that kubelet is still the orchestrator on the node.

CSI does not replace kubelet. CSI gives kubelet a standard way to ask storage drivers to do their job.

Two different stories: provisioning vs mounting

People often say, "Kubernetes mounted my volume," as if that were one step.

It is usually at least two stories:

1. Provisioning / binding

This is the question:

"What actual storage object will back this claim?"

That may involve:

• binding a PVC to an existing PV
• dynamically provisioning a new disk, filesystem, or network share
• waiting until a node is chosen so topology-aware provisioning can happen correctly

2. Node-local mounting

This is the question:

"Now that a volume exists, how does this node make it visible inside the Pod?"

This post is mainly about the second story.

We will touch the first only enough to make the node path understandable.

Before the node starts work

By the time kubelet is preparing a Pod, some storage decisions may already be done.

For example, with a PVC-backed volume, the cluster may already have:

1. created or selected a PV
2. bound the PVC to that PV
3. decided which node the Pod will run on

But storage is tricky because some drivers use topology-aware provisioning.

That means the exact storage object may depend on where the Pod lands.

This is the intuition behind things like WaitForFirstConsumer in a StorageClass:

"Do not commit to a disk too early. Wait until scheduling gives us a real node or zone context."

So even before kubelet mounts anything, the scheduler and storage controllers may already be part of the story.

The most important distinction: attach vs mount

This is probably the single most useful storage distinction for Kubernetes operators.

Attach and mount are not the same thing.

Not every volume type needs both.

Volumes that often need attach + mount

• cloud block volumes such as EBS-like disks
• SAN-style block devices

Volumes that may only need mount-like behavior

• NFS or network filesystems
• projected volumes such as Secrets or ConfigMaps
• emptyDir

So when a Pod is stuck waiting for storage, one of the first questions should be:

"Is the failure in the attach phase, or the mount phase?"

That one question removes a lot of confusion.

The node-side sequence at a glance

Here is the high-level path once a Pod with volumes has been assigned to a node.

1. kubelet sees the Pod and notices the required volumes
2. kubelet's volume manager begins reconciling desired vs actual volume state
3. if the volume type requires attachment, kubelet waits for that to complete
4. kubelet asks the CSI node plugin to prepare the volume on this node
5. the volume is staged or mounted at a node-local path
6. kubelet bind-mounts that prepared path into the Pod's volume directory
7. the container runtime makes that mount visible inside the container at the configured mountPath
8. only then does the application process start with the expected filesystem view

Here is that flow visually.

This is why volume mounting belongs conceptually next to Pod startup from Part 3. The container should not start until the filesystem view is ready.

The kubelet volume manager

Inside the node, kubelet has to manage more than containers and probes.

It also has to answer:

"For every Pod on this machine, which volumes should exist, and which ones are actually mounted right now?"

That work is handled by kubelet's volume manager and related reconciliation loops.

This matters because storage is not a one-shot action.

The node has to keep reasoning about:

• desired volumes for each Pod
• current mounts already present on disk
• cleanup when a Pod goes away
• retries when mounts fail
• remounts after kubelet restarts or node disruptions

The same Kubernetes pattern shows up again:

desired state vs actual state, reconciled locally by kubelet.

Where CSI fits

Now we can place CSI correctly.

CSI stands for Container Storage Interface.

It is the storage equivalent of what CRI did for container runtimes and what CNI did for networking.

Its job is to standardize how orchestration systems talk to storage drivers.

That means Kubernetes does not want hard-coded storage logic for every cloud disk, every file share, and every vendor appliance.

Instead, Kubernetes can say:

"If you are a CSI-compatible driver, here is the contract. Implement these operations, and kubelet plus the controller components will know how to use you."

The easiest way to think about CSI

• CSI controller side handles storage actions that are cluster-wide or backend-facing
• CSI node side handles storage actions that must happen on a specific node

So CSI fits between kubelet's intent and the actual Linux mount operation.

It is a contract layer, not magic.

The two CSI node calls worth remembering

You do not need to memorize every CSI RPC to reason well about node storage.

But two names are worth knowing because they represent the most important node-local phases.

NodeStageVolume

This is roughly:

"Prepare this volume on the node at a staging location."

For some drivers, this might involve:

• discovering the attached device
• formatting it if appropriate
• mounting it at a global path on the node

Not every driver uses staging, but when it exists, staging is usually a node-global preparation step.

NodePublishVolume

This is roughly:

"Make the prepared volume available for this Pod at the target path."

In practice, this often means mounting or bind-mounting the volume into a Pod-specific path managed by kubelet.

So the short mental model is:

• stage = prepare once on the node
• publish = expose to this workload at this path

That distinction is much easier to remember than the raw RPC names.

A more complete end-to-end flow

Let us walk through a common PVC-backed volume flow without drowning in every implementation detail.

1. Pod references a PVC

Your Pod spec says, in effect:

volumes:
  - name: data
    persistentVolumeClaim:
      claimName: app-data

At this point, the Pod is not asking for a mount directly. It is asking for the storage represented by the PVC.

2. PVC is bound to a PV

The cluster makes sure app-data is backed by a real volume.

That might be a provisioned disk, a network share, or some other backend depending on the driver.

3. Pod lands on a node

Now the request becomes node-local.

Kubelet on that node sees:

"This Pod needs volume data, and I must make it appear before the container starts."

4. Attachment happens if the storage type needs it

For attachable storage, the volume must become visible to the machine first.

If that does not happen, kubelet cannot mount it, because there is nothing usable on the node yet.

5. kubelet asks the CSI node plugin to prepare it

The CSI node plugin performs node-specific steps such as:

• device discovery
• filesystem checks
• stage mount creation
• target path mount creation

6. kubelet places it into the Pod's world

Kubelet keeps Pod-specific directories under its own state paths, commonly under locations like:

• /var/lib/kubelet/pods/...
• /var/lib/kubelet/plugins/...

The exact paths are less important than the idea:

the node has a kubelet-managed place where Pod volume mounts live before the container ever sees them.

7. container runtime bind-mounts it into the container

When the container starts, the runtime includes the correct mounts so the process sees the volume at the configured mountPath.

From inside the container, it just looks like:

"There is a directory at /var/lib/app, and my files are there."

But underneath that simple view is a chain of kubelet, CSI, mount points, and namespace-aware bind mounts.

The mount is real Linux, not Kubernetes magic

This is worth stating as clearly as possible.

At the end of the storage path, Kubernetes is still relying on ordinary Linux behavior.

The kernel understands mounts.

Kubelet and CSI are coordinating which thing should be mounted, where, and when.

But the final effect is still a real mount in the node's filesystem and then a container-visible mount in the container's mount namespace.

That is the same pattern we saw elsewhere in the series:

• Kubernetes expresses intent
• a node agent coordinates the work
• a lower-level subsystem makes it real

For containers, the lower-level subsystem was the kernel plus runtime.

For storage mounts, the lower-level subsystem is still the Linux mount machinery, often reached through CSI driver logic.

Pod sandbox, container mounts, and bind mounts

One subtle point confuses many people the first time they debug storage.

The container does not usually mount a cloud disk directly by itself.

Instead, the node prepares the volume, and then that prepared path is made visible inside the container.

This is why bind mounts are such a useful mental model.

You can think of it like this:

1. node obtains access to storage
2. node mounts it at a kubelet-managed path
3. kubelet/runtime bind-mount that path into the Pod/container view

That is also why many volume debugging steps still happen on the node, not inside the container.

If the node mount never succeeded, there is nothing useful for the container to see.

Not all volumes go through CSI in the same way

Another source of confusion is assuming every volume behaves like a cloud disk.

That is not true.

emptyDir

emptyDir is node-local ephemeral storage.

It does not require an external storage backend. Kubelet prepares a directory on the node for the Pod, and that directory lives as long as the Pod does.

ConfigMap, Secret, and projected volumes

These are also mounted into the Pod, but the backing data comes from Kubernetes objects, not from a block device.

The node still prepares something mountable, but the storage path is very different from a CSI-backed persistent volume.

CSI-backed persistent volumes

These are the cases where CSI is most visible, especially when you are working with cloud disks, network filesystems, or vendor storage systems.

So the right mental model is not:

"All volumes are CSI mounts."

It is:

"Kubernetes has multiple volume types, and CSI is the standard driver interface for many persistent storage backends."

The failure patterns you should recognize fast

Most storage incidents on nodes collapse into a small number of categories.

1. PVC is not bound yet

If the claim is still pending, the node is often waiting on a problem that has not even reached the mount stage.

Typical symptoms:

• Pod stuck pending or creating
• PVC remains Pending
• provisioner or topology issues in events

2. Attach succeeded nowhere, so mount cannot begin

For attachable volumes, if the storage never becomes visible to the node, the mount phase is blocked.

Typical symptoms:

• repeated attach timeouts
• cloud-provider or CSI controller errors
• kubelet waiting for attached volume

3. Node plugin is missing or unhealthy

If the CSI node plugin is not running correctly on the node, kubelet may know what it wants but have nobody to perform the node-local storage work.

Typical symptoms:

• NodePublishVolume or NodeStageVolume failures
• CSI socket errors
• volume setup repeatedly retried

4. Mount path or filesystem operation fails

Sometimes attachment is fine and the driver is present, but the actual Linux mount step fails.

Typical symptoms:

• wrong filesystem type
• target path already busy
• stale mount state after node disruption
• permission or device discovery issues

5. Cleanup or unmount is stuck later

The storage story is not only about startup.

If volumes do not unmount cleanly, later Pods can fail too.

Typical symptoms:

• terminating Pods that linger
• device still busy during detach
• stale Pod directories under kubelet paths

Again, the fastest operator question is often:

"Did we fail before attach, during node staging, during publish, or during cleanup?"

Where this connects back to Part 4

Part 4 focused on node survival: pressure, eviction, OOM behavior, and cleanup.

Storage plugs into that story in several ways.

DiskPressure is not only about images

When nodes run low on storage, the pressure may come from multiple places:

• image layers
• container writable layers
• logs
• emptyDir
• volume-related artifacts and mount state

So storage is not just a startup topic. It is also part of steady-state node health.

Volume problems can block Pod startup entirely

Unlike some application-level failures, mount problems often happen before the app process even exists.

That is why storage failures frequently show up as:

• long ContainerCreating
• FailedMount
• FailedAttachVolume

The process never got a chance to fail, because its filesystem view was never ready.

A compact debugging order

When a Pod with volumes is stuck, use a short, repeatable inspection order.

1. kubectl describe pod for mount-related events
2. kubectl get pvc,pv to confirm claim and binding state
3. kubelet logs on the node for volume manager and mount errors
4. CSI controller and CSI node plugin logs
5. node mount state, kubelet Pod volume paths, and device visibility

This order helps you classify whether the problem is:

• API object state
• controller-side storage workflow
• node-local CSI work
• or raw Linux mount behavior

Concepts introduced

Let us make the new vocabulary crisp.

PersistentVolumeClaim (PVC)

The Pod asks for storage indirectly through a PVC.

That lets the Pod describe the storage it needs without hard-coding the backend details.

PersistentVolume (PV)

The PV is the storage resource that actually satisfies the claim.

It is the cluster's representation of the underlying storage.

Attach vs mount

Remember this split:

• attach = make storage available to the node
• mount = expose storage at a filesystem path

Many debugging mistakes come from treating those as the same event.

NodeStageVolume and NodePublishVolume

These are the two most useful CSI node operations to recognize.

• stage = prepare globally on the node
• publish = expose to a workload path

You do not need to memorize every RPC around them to reason correctly.

Kubelet volume manager

This is kubelet's local reconciliation machinery for volumes.

It tracks what should be mounted, what is already mounted, and what needs cleanup.

Final mental model

If you remember only one thing from this post, remember this:

A Pod volume is not "just there." It is prepared on the node and then made visible inside the container.

• the Pod asks through a volume spec, often via a PVC
• cluster-side logic ensures a real storage backend exists
• kubelet notices the required volume on the assigned node
• CSI performs node-local storage operations through a standard contract
• the Linux mount system makes the volume real on the node
• kubelet and the runtime make that mounted path visible inside the container

Once that clicks, storage stops feeling like a side topic and starts feeling like what it really is:

one more essential node path that must be correct before an application can run.

At this point, the full node picture is much more complete:

• execution stack
• bootstrap and trust
• Pod birth
• node survival
• and now the storage path that gives containers their data

That is the difference between seeing a mountPath in YAML and actually understanding what the node has to do to make it true.