Lately I’ve been experimenting with a year home server running in a ZFS pool on a hard disk array. When I prioritize capacity and reliability, which I obviously look for in a NAS, this setup works great. I have no problem streaming large files and the system is exceptionally stable for me. But over time, I started to notice a few instances where the whole system felt slower than it should. It wasn’t something that showed up in the benchmarks, but opening folders with thousands of files like images certainly took a lot longer than expected. Programs that scan large media folders were pausing more often than I thought. Elsewhere, container workloads that relied on multiple reads were also less responsive than they should have been. Hard drives were not saturated, the problem was latency.
It took me quite a bit of digging and studying before I realized that most of these delays were caused by metadata access. Apparently, ZFS has a feature designed to solve this particular problem. It uses a special allocation class called metadata vdev. Instead of storing its file system metadata on the main pool disks, it moves these structures to faster storage such as an SSD or NVMe drive. To solve the problem, I added the spare NVMe drive to the pool as a metadata vdev and the difference was immediate.
Migrating file system metadata to NVMe reduces latency and speeds up day-to-day operations
The thing is, when we think about storage performance, file transfer speed is at the top of the priority list. But day-to-day file system activity has little to do with large sequential reads or writes. Instead, the system constantly searches for metadata. Metadata is information that the file system stores about each file. This includes filenames, timestamps, permissions, directory structures, and even the location of the actual data blocks. First ZFS pool if it can read any file, it must check this metadata to determine where that file lives on disk.
Since metadata resides on spinning disks, each data retrieval requires random access. Hard drives are great at transferring large data, but the speed limitations become apparent when it comes to jumping around the board for small read cycles. And that’s exactly what happens when it comes to reading metadata.
Adding a metadata vdev changes where those metadata files live. After a custom vdev is added to the pool, ZFS starts storing the new metadata blocks on the faster device rather than the original data vdev. This is a fair idea. Hard drives continue to store most of the data while SSD or NVMe folds the system index. When the system needs to find a file, it can solve the search on fast SSD storage instead of searching on slower hard drives.
As I mentioned before, this fast memory is small on my system NVMe driver connected to the server. There is no need to change anything about the main pool. Your hard disk array works as before; only metadata is transferred to flash memory. Note that this change does not change the maximum throughput either. This only reduces the latency associated with finding files. But it makes a bigger difference than you might expect for real workloads.
Faster directory searches, faster library scans, and what you need to know before adding one
The speed increases are most noticeable when you browse the catalogs. Like many their ownersI store large media libraries and photo archives on a server with some folders containing tens of thousands of files. Previously, listing these folders could take several minutes, as the system fetched metadata from different drives. After adding vdev metadata, those listings now appear almost instantly.
Media library scans are also faster. Especially for programs that index music, movies or photos. These tasks involve reading metadata rather than actual files. And moving metadata to SSD dramatically reduces read time. Container storage also benefits from the switch. Many self-hosted services rely on databases or folders full of small files. These workloads can generate significant amounts of random file system activity. Instead of the hard drive handling these searches, the NVMe device now takes care of them. All said and done, the end result is a much more responsive system.
There are some specific details you should know. It is important to understand that vdev is not a cache. ZFS stores metadata permanently on vdev, which means that the storage pool depends on it. If the vdev fails and is not mirrored, the storage pool may be unavailable. It’s a good idea to mirror a custom vdev across two SSDs for redundancy. That said, unless you’re running a mission-critical workload, building vdev on a single NVMe might do the trick.
Why does metadata vdev make ZFS feel faster?
It’s a good idea to keep your expectations in check when working with vdev. Adding a metadata vdev will not change your data storage server to a full flash array. Your transfer speed won’t double either. Reading the actual files will still work at the speeds you’re used to, but the system senses how you’re accessing files faster. Directory navigation is faster and library scans are significantly faster. Instead of waiting for hard drives to search between platters every time you need to open a large directory, ZFS resolves these searches in fast flash memory, greatly increasing responsiveness.
