Fog Computing and IoT – the solution to data storage issues?
Posted on Tuesday, March 27, 2018
Perhaps the biggest future challenge facing data-driven businesses is how to capture, analyze and store the information being created by IoT smart sensors. As discussed on the CDS blog before, simply off-loading that data to the cloud will not work for real-time systems because of the latencies involved.
However new technologies are coming that may address this issue – even if that’s not what they were designed for.
Latency is not restricted to IoT and the edge. The bridge between CPU and disk array is fast becoming a problem for system designers, creating latency even on “local” systems. At the data center level, most CTOs are deploying hyper-converged systems to reduce the bottleneck between storage and CPU – but there are new technologies waiting to emerge.
The Register recently profiled a number of technologies that are undergoing testing with a view to uniting processing and storage into a single device. According to the report, it will take 4.5 hours to read a 0.5Pb dataset into a 2-CPU server fitted with 8TB SSD drives. Using a Catalina 2 card containing 8TB SSD drives fitted with integrated CPUs however, the same process can be completed 8x faster, in just 15 minutes.
NGD, who produce the Catalina card, believe that the technology will help to further reduce bottlenecks between processor and data. In order to take full advantage of the cards however, CTOs will need to re-engineer their applications, adding significantly to the cost of deploying on-drive processing.
On-drive processing – in the fog
On-disk processing does have potential in the data center, but its natural home may actually be in IoT systems. On-drive applications could be used to sift, sort and process data generated by IoT sensors, before uploading anything that will be of long-term value to the data center.
Take CCTV for instance. A smart camera could record data directly to a Catalina 2 card, where the incoming stream is run through facial recognition software running natively on the local storage. Where a match is detected, the relevant information is uploaded to the cloud, while the remainder is kept locally, or discarded entirely.
In this way, remote IoT sensors become like “fog” – they are cloud-enabled, but also capable of operating autonomously. They will also be much quicker to act on data than a network of remote sensors configured to upload data for remote processing.
As with all new technologies, time will tell whether they can deliver on expectations. But it is good to know that there are some genuinely innovative ideas being tested to overcome the challenges created by IoT.
To learn more about fog computing and how we can assist your business, contact us.
The SSD Capacity Record Has Been Broken Again
A new 100TB drive means that Samsung’s claim to the largest SSD lasted less than 30 days.
Can You Really Afford To Store Data “Offline” Any More?
Tape archive storage has been an important part of data strategy for years – but it is time to finally kill offline archives