This vendor-written tech primer has been edited by Network World to eliminate product promotion, but readers should note it will likely favor the submitter’s approach.
Video is everywhere, and growing exponentially. According to a recent report, 35 billion video ads were viewed in December, representing year-over-year growth of more than 100%. And every industry is seeing video growth, which creates a problem for data managers because video challenges storage management in four ways:
- Performance requirements are not served well by traditional storage architecture.
- Rapid video growth can overwhelm storage environments while resource utilization is masked by virtualization
- Use of traditional backup tools make data protection expensive and challenging
- Long-term data value means this is not a temporary problem
If, however, you move video to specialty storage, you’ll achieve five amazing benefits:
* Higher Performance. Video has different characteristics and performance requirements from traditional corporate data. A single video file is often up to a TB or more. Collaborative video processing applications (e.g., post production) may or may not be IOPS- or bandwidth-centric, but they often require almost zero latency.
This combination of large file and latency sensitivity is a challenge for traditional storage architectures. Storage arrays typically utilize intelligent memory caches to deliver performance; having written data to the cache, the application is freed to perform its next operation. Read transactions from memory buffers can occur at much higher speed than a read from disk. For traditional data, this model works well. But these software-managed caches are not large or flexible enough to manage unpredictable video data streams.
Video files overflow storage caches, causing the array to pause while it pages data to and from the disk. Meanwhile, the latency-sensitive video application continues to send data. This results in “frame drop”; writes which are not serviced rapidly enough are “dropped” by the storage system. This results in a performance problem and may, in some cases, cause data to be lost.
When video causes this data traffic jam, other transactional or productivity data which is better suited for this architecture isn’t getting serviced either. Resources are being used but nobody’s getting good performance.
Field engineers from the best array vendors will anticipate these problems and deal with them by turning the caches off and overprovisioning disk heads in the storage system to address the needs of the video/image data. But, besides adding expense, this leaves traditional data without the tools needed to address performance.
Segregating out large, latency-sensitive video data – and applying specialty storage which is engineered to match its needs – gives both sets of applications better data access and performance.
* Faster, smoother, cheaper backups. In addition to being super-sized, video data is unique in a second way: these files are static. While users may clip pieces of these files to make new files, the original files aren’t modified. For this reason, they can create an unnecessary burden in your backup process. Smart incremental backup technologies will avoid backing up these files, but when you need to do a full backup, these large static files will get copied – over and over again, using lots of bandwidth and storage.
Moreover, unless a file is an exact copy of a prior video, these files do not de-dupe. But their volume and capacity will consume compute resources as the deduplication process attempts to do its (fruitless) job. Meanwhile, other non-video files are waiting to be protected.
This challenge is exacerbated by video’s long useful life. This data is often high value, causing it to be saved for years. As a first step, simply identifying this data and segregating it from your normal backups will have a strong positive impact on backup schedules and costs.
Moving it to specialty storage will have additional benefits. Video-optimized storage is architected with smart metadata that only store copies of new versions as files, groupings of files, or directories are added. This model avoids the need for a traditional file walk process and optimizes the capacity of protected storage. Backups of all your data will operate more smoothly.
* Increased customer satisfaction. Having segregated video from primary storage, your applications and users will all get better data access, availability and performance. Work will complete more rapidly. Avoiding backup window overruns also means avoiding production maintenance outages or performance issues. Finally, because you won’t need to respond to the challenges that this data can cause, you’ll have more time to plan for the future – and to consider and deploy other options for improving quality of service. All your customers (and you!) will be happier.
* Managing data growth. Video data is growing exponentially and it needs kept for a long time. However, when this data is combined with traditional data, optimizing its placement (for performance and access) is a challenge. Moving the data to specialty storage solves this issue by providing intelligent storage tiering, driven by either workflow or data usage.
Video is often integrated with a business workflow – a combination of multiple applications that deliver business results. Specialized storage for video is architected to work in combination with these applications (“workflow-managed”) so data is moved to the right storage automatically.
As an example, content associated with completed projects can be automatically archived on direction by an application process. The specialty storage controller is also aware of content usage, so it can automatically move data to lower-cost storage while maintaining transparent online access.
These combined capabilities give the user the right quality of service while the data is stored at the lowest cost. This can even occur at initial data write – e.g., video surveillance data can be transparently moved from camera to digital tape, and only restored to disk if needed for analysis. This capability allows you to make much more efficient use of your primary storage. If you are running near capacity on primary, the system automatically migrates data to next tier disk, object storage, or tape, allowing you to maintain access until your primary disk upgrade is completed.
Segregating video data also enables accurate tracking of this high growth data for applying billing. You can better manage overall data growth.
* Lower operating cost. Finally, all of the prior benefits of applying specialty storage combine to deliver a lower total cost of ownership. Specialty storage will need fewer disks to deliver performance. Improving your backup process will decrease labor costs and reduce needed software licenses. Having more satisfied users means less work. And better management of growth saves hardware, power and cooling, floor space and labor. The application of specialty storage for video data is a win for everyone.
Stow Lee has more than 30 years of experience in the storage industry, including 10 years working with a variety of companies in data reduction and file system software and hardware.
resource