Quantum StorNext 7 — Exploring the new tiering features

A summary of the new File System Pooling feature with example commands

Without further ado

Quantums StorNext file system has enabled a large number of post-production workflows over the years, so it’s no surprise that the release of a new version has a lot of people excited, including us. The most notable changes and additions in the newest version include another (separate) UI, a switch to a capacity-and subscription-based licensing model and the addition of the File System Pooling feature, which is the main focus of this article.

What is File System Pooling?

Due to the intensely high demands of modern media workflows, post-production houses are always on the lookout for a storage solution that can consistently deliver ever-increasing performance. Consequently, the extremely fast NVMe drives have been receiving a lot of (well-deserved) attention in recent years. However, a Terabyte of storage on an NVMe comes at a hefty price-premium compared to the good-old Hard Drive. This is why media storage manufactures develop different strategies to tie these storage mediums into a single environment — one in which the user can profit from the performance of NVMe drives or even SSDs while at the same time relying on Hard Drives to provide the bulk of relatively low-cost storage. File System Pooling is Quantums solution to tackle this challenge. This feature allows users to group different types of storage mediums together inside one file system and define policies for moving the files between the pools.

How does File System Pooling work?

Pooling moves the data without changing the metadata associated with the file or the file attributes. Most importantly, pooling works on a block level and doesn’t actually move the files in the namespace — only the data of the file itself. Pooling can be initiated in two ways: Manually by a user or automatically by a preset pooling policy. Policies are scheduled tasks that use a set of criteria to identify the files to execute a pooling job with. Policy criteria relates to file size, files modification time, files location and more. It is important to mention that File System Pooling is handily integrated as part of StorNexts core filesystem components and not related to the tiering features of the separate StorNext Manager software suite, which are in our opinion historically complex and hard to use on a daily basis.

Pooling Jobs

Jobs define a set of instructions to be executed on specified content — either manually selected or identified by a policy. A pooling job can be either executed immediately or scheduled to run on a specific date and time. An internal locking mechanism of the StorNext filesystem prohibits changes to the file that is currently being used by a job. There are seven different job types offered and described in the StorNext documentation as follows:

Useful CLI commands

Before you can use “sntier”, you have to enable the service. This requires the StorNext filesystem services to be running and the SN Web API enabled (which has been the default setting for some time now). Run the following command to enable the service on the node:

# sntier service --enable
# sntier pool --add --sg fast nvme --sg slow qxs --mount /stornext/xcellis01/
# sntier pool
Pool status host localhost
Mount: /stornext/xcellis01
PoolName Capacity Used Exclusive Stripe Groups
fast 13.10 TiB 11.47% No nvme
slow 4.91 TiB 0.00% No qxs
Mount: /usr/adic/HAM/shared
Unassigned SGs: sg0
# sntier submit --action promote --target fast --paths /stornext/pooling/dir1/
# sntier submit --action demote --target slow --paths /stornext/pooling/dir1/
# sntier jobs
# sntier jobs --show 1000
{
"content": {
"paths": [
"/stornext/xcellis01/test/"
],
"mount": "/stornext/xcellis01"
},
"originator": "admin",
"job": 1000,
"task": {
"action": "demote",
"target_pool": "slow",
"io_size": 1048576
},
"comp_report": {
"finishTime": "Tue Apr 6 13:48:26 2021",
"childjobs": 46,
"elapsedSec": 101.946394,
"filesSkipped": 0,
"filesMoved": 1000,
"bytesMoved": 50987008000,
"bytesPerSec": "477M",
"dirsChanged": 1,
"state": "completed",
"queueTime": "Tue Apr 6 13:46:44 2021",
"startTime": "Tue Apr 6 13:46:44 2021",
"filesScanned": 1000
}
}

Expanding the functionality of File System Pooling

StorNext Pooling Jobs can easily be set up and triggered through the ELEMENTS Workflow Automation Engine. This integration allows you to execute Pooling Jobs together with a chain of other tasks, including: automatic transcoding, cloud synching, Slack notifications and even Media Library operations.
As an example, we have created a short video that shows how easy it is to integrate this feature flexibly into the ELEMENTS Automation Engine:

Conclusion

When it comes to media workflows, storage environments consisting of multiple types of storage mediums simply make a lot of sense. Using flash-based storage for extreme performance-demanding work and Hard Drives for basically everything else can drastically increase the efficiency of your workflows. However, this approach is only as effective as the data management mechanism itself. The new File System Pooling feature seems to offer all the features needed to profit from this kind of infrastructure and we are very excited to put it to work.

human-centered media storage