Most organisations who are looking to add Solid-State Drives (SSD) to their storage environment, are often mindful of finding ways to avoid under-provisioning and ensure performance and scalability. However, to meet cost goals and avoid unnecessary spending they should steer away from over-provisioning. Workload profiling can help orgnisations achieve the critical balance.
A recent survey of 115 Global 500 companies by GatePoint Research showed that 65 percent of storage architects say they are doing some sort of pre-deployment testing before making their investment decision. Alarmingly, only 36 percent understand their application workload I/O profiles and performance requirements. They don’t know what workload profiling is and how it can be used to accurately evaluate vendors against the actual applications that will be running over their particular storage infrastructure.
Workload profiling explained
Workload profiles, often called I/O profiles, are data and statistics that directly relate to storage activity and loading in real (observed) production storage arrays. It characterises the realistic, sometimes massive, application workloads that strains networked storage at an infrastructure level. Profiles typically comprise a mix of virtualised applications, such as databases, that can have significant random I/O content.
I/O profiles contain information on reads vs. writes, the mix of random versus sequential data access, the data and metadata commands, file and directory structures and IOPs to name a few of the key metrics.
Two key steps to creating a workload profile:
Gather production data: The first step entails accessing the storage array logs and other statistics available from each of the production storage array’s proprietary tools. Every vendor has their own way of reporting storage I/O and utilisation data and most storage admins know how to run these tools and utilities over a predetermined period of time. This data provides the foundation of the workload-to-performance relationship. The data can be input into a storage workload modelling application.
Workload modelling: The second step entails creating the models based on the array data. There are tools available commercially that can take storage array log data and directly import it into the workload model. These models can then be used to vary the assumptions and perform a number of possible outcomes and worst case scenarios that users can review. Such applications can maintain libraries of repeatable scenarios that can be used to stress a storage system under a realistic simulation of the workload(s) it will be supporting in production. Workload modeling enables the comprehensive performance testing of any flash or hybrid storage system under the actual conditions that it is expected to operate. With highly accurate simulations, storage performance can be fully predictable.
Applying workload profiles
After creating our workload profiles, we can typically use them to validate storage solutions with a number of testing.
An example of which is limits finding, which entails determining the workload conditions that drive performance below minimal thresholds, and the documenting of storage behaviour at failure point. Another is functional testing, this is the investigation under simulated load of various functions of the storage system like backup.
One can also try error injection, which requires an analysis of the solution under simulated load of specific failure scenarios (e.g., fail-over when a drive or controller fails). Lastly, users can also try soak testing, this include conducting an observation of the storage system under load sustained over extended periods of time potentially around three days to one week, or sometimes more.
Performance and load testing with workload profiles can also be used to tune and validate flash and hybrid storage infrastructure in critical areas.
Why is workload profiling important?
Workload profiling can offer vital insights into the existing or planned SSD infrastructure of an organisation, empowering storage professionals to optimise cost while assuring performance and reliability goals are met.
With a robust storage performance validation process in place, engineers and architects can optimally select and configure networked storage systems for their workloads by aligning performance requirements to purchase and deployment decisions. With such insight, application performance can be predictably assured and storage costs can be significantly reduced for production storage systems. It simply enables storage architects and engineers to make better, more informed decisions by eliminating the guesswork related to storage performance.