Understanding storage tiering can boost efficiency in your post-production workflows
Hybrid storage is a term that is often used by IT storage vendors to describe a storage environment that consists of multiple types of storage devices with mechanisms that determine when and how the data should be moved between them to ensure the highest-possible performance at an affordable price. Typically, a mix of Solid-State Drives (SSD) and Hard Disks (HDD) is used. Over the years, we have had a lot of experience with these systems and we’d like to share some general information, as well as specific benefits that hybrid storage could provide to your company, whilst also highlighting where it might not work so well.
Why do we need hybrid storage?
The amazing rate of development of flash technology over the last few years has had a truly remarkable impact on the price of SSDs. This trend has allowed us to build blisteringly fast flash-based storage for many of our clients at a much lower cost than they expected. Our confidence in flash-based storage led us to develop the ELEMENTS BOLT in 2019, whilst our first “hybrid” storage product was in fact the ELEMENTS NAS with a NVMe cache, which we released in 2017. But with over 25GB/s of internal performance in a single 2U chassis, the NVMe-based ELEMENTS BOLT is still one of the fastest storage solutions on the market. And it comes at a very competitive price per TB when compared to other all-NVMe solutions. However, the still present price premium of SSDs compared to HDDs makes an all-out move to all-SSD storage a stretch for tight budgets in many companies — especially the smaller ones. This is where IT vendors and their hybrid storage solutions usually show up.
Hybrid storage is neither the highest performing nor the cheapest storage that money can buy, but it is certainly one of the most efficient if you look at its data sheets. Combining different storage mediums allows you to reap their individual benefits and thereby achieve a great balance of performance and cost. This approach has proven itself time and time again as a solution which allows projects of smaller companies to compete with those of the big dogs.
Benefits in media workflows
One thing is certain, media workflows can be relied upon to always demand some of the highest storage requirements. Continually-increasing bitrates of the used footage place high demands on storage performance while shear amounts of material with huge file sizes fuel the hunger for ever-higher capacity. Simply said, storage is always full and always too small.
Hybrid storage is an elegant solution for both of these demands and one that lets you keep the frequently accessed media with high performance requirements on the SSD drives while at the same time having the bulk of the systems capacity provided by comparably low-cost HDDs. Due to their lower latency and higher speed, SSDs are much better suited to power the colour grading or VFX process. In some cases, the “slow” layer can even be backed by a cloud storage, parking all “cold” assets off-site in the cloud.
File management
IT based hybrid storage has one big problem and that is that many products work based on caching algorithms that bring a file to the fast layer after it has been accessed for the first time. However, when working in editing environments, this is not good enough. Editors require the video to play back smoothly at the first play, not at the second. Caching a full timeline with the client sitting next to you (aka. watching a stuttering playhead move across the timeline) is far from efficient. What also needs to be taken into account is that the caching occupies a sizeable chunk of performance from the “slow” layer, because it runs at the fastest possible pace. This in turn grinds all other read processes (that are not already on the fast tier) to a halt. Simply said — the storage does not know which file will be required on the fast tier and when it will be required. This is typically why hybrid solutions are not used in finishing or editing but rather in VFX and rendering. Here, real time playback is not an issue and the same datasets (textures, layers, images) are accessed much more frequently.
This means that in order to get the most benefit out of a hybrid storage environment, an efficient file management mechanism is needed. This mechanism should ideally provide more functionality than a mere assignment of directories or workspaces to different storage pools, which consist of different storage media. Single files or whole collections will often have to be moved from one pool to another and doing this by hand would put a burden on the IT team and could easily lead to mistakes. Automating this process makes a lot of sense. Automation applications, running on a client on the network, are sometimes used to solve these problems but can easily lead to a new one — an increased network load as the files now have to traverse the client.
Ideally, this automation process should be flexible, easy to understand and use, and running in a way that allows for the fastest possible data distribution without burdening the network. Due to our extensive experience with hybrid storage environments, we at ELEMENTS have recognised this need early. To solve it, we have developed and equipped every ELEMENTS system with a powerful Workflow Automation Engine. Set-up custom workflows of any size and complexity and profit from maximum performance, as all tasks are running exactly where they should be — directly on your ELEMENTS system, or even on 3rd party storage environments. Permissions management tools allow you to determine who can start the workflows and even how — through our Media Asset Management tool the Media Library or through the context menu in the macOS Finder and Windows Explorer. The Workflow Automation Engine can also be used to transcode media, run python scripts, sync your storage with a cloud and much more.
Small-scale vs. large-scale
The concept of using different storage technologies for different use-cases works just as well in a single system as it does in large environments. Just as different types of devices can be used in a single storage system, an environment consisting of multiple, different systems can be strung together. For instance: We have made a number of our customers very happy by implementing an environment consisting of one or more NVMe-based ELEMENTS BOLT to provide the performance, while the bulk of the capacity is delivered by ELEMENTS ONE, or even JBODs. Add an LTO Tape Library for archiving and a cloud instance for off-site backup purposes and an extremely powerful environment is built, which is capable of supporting even the most performance-demanding workflows in a secure and consistent way.
Conclusion
Hybrid storage can provide a way to serve the different needs of every step of the post-production process in the most efficient manner, if it is utilised correctly. Just installing a typical “hybrid raid” with a few NVMes or SSDs and hoping that clever caching will solve the performance bottleneck is not going to work in the typical post-production environment. But using different storage layers in a well-designed manner allows us to build an environment that can support the highest performance-demanding workflows, while at the same time reducing cost significantly. However, it is important to keep in mind that the efficacy of this approach is highly dependent on the file management strategy.
Our approach
In media workflows, storage planning and management is usually quite straight-forward. The typical post-production deals with either a number of individual projects (think movies and commercials) or with a longer-running venture consisting of individual projects (think of series and their episodes for example).
In this case, it makes a lot of sense to simply keep the currently running projects and those that are planned to run soon on the faster storage layer. This approach will provide you with the maximum performance, even when accessing files for the first time. To simplify, speed up and automate this periodic data distribution on the system, we recommend using our Workflow Automation Engine. This is by far the most efficient strategy for moving data between storage layers and it requires minimal interaction, thus greatly minimising the chances of human error.
But what is ultimately clear is that in a few years time, the cost of flash-based storage will have come down enough to make HDDs in a performance sensitive environment fully obsolete. It’s just a matter of patience.
StorNext 7 tiering
The newly-released Quantum StorNext 7 file system offers a number of new and efficient tiering features. The integration of these tools into our Workflow Automation Engine allows for efficient moving of files between storage pools with different performance. To find out more, read our blog: Quantum StorNext 7 — Exploring the new tiering features.
