What About the Other Storage Challenge: Velocity?
Posted on Wednesday, June 12, 2019
If current trends continue, data storage will be a problem forever: digital transformation, IoT and new computing models are generating vast amounts of data every day.
We’ve looked at the topic of ‘volume’ several times on the CDS blog, as scientists and engineers grapple with the issue of capacity. DNA, salt crystals, powder and walls have all been proposed as high-capacity storage media of the future. In each case, these media offer far greater potential storage density than current magnetic disks and solid-state drives (SSD).
Big data just keeps getting bigger
Capacity is undeniably important. Major projects like the Large Hadron Collider will increase demand from 10 petabytes per year, to 150PB in the next five years. US Department of Energy experiments into plasma physics are creating 100 PB every run. These are vast data sets at the upper end of demand; but as IoT becomes commonplace, even medium-class businesses will see exponential growth in demand.
But this data is useless if it cannot be accessed and analyzed relatively quickly. This is particularly true of applications that power real-time decision making. Any delay is unacceptable and could even be fatal in deployments like autonomous vehicles.
But what about speed?
SSD storage has gone some way to increasing velocity– the speed of I/O operations. But this, too, is limited.
Several vendors have begun work on flash RAM-based servers to support in-memory databases and applications like Apache Ignite and Oracle RDBMS. Fast and efficient, this hardware remains extremely expensive, and well below the capacity capabilities of big data applications.
As with all prior storage technologies, in-memory systems will become cheaper over time. In the meantime, OEMs will need to address the remaining data throughput bottlenecks, such as the reduction in speed when reading and writing data to the secondary and tertiary storage layers.
Current efforts are focused on increasing the speed of RAM, and capacity of each chip. It remains to be seen where the next major innovation in data storage speeds comes from.
Do Magnetic Disks Really Last Longer Than SSDs?
Because SSDs degrade over time, there’s a perception that hard drives last longer, but is that true?
Is Storage Design Undermining Your AI Efforts?
Huge storage capacity is vital – but it isn’t enough for artificial intelligence applications.