Duplicate Data – still a threat to your business?

Image of lots of squares to indicate duplicate data

Posted on Thursday, January 21, 2016

Historically the large cost of storage meant that businesses had to be proactive in managing data to ensure capacity was not wasted. Purchasing additional capacity was a major capital investment that needed to be balanced against corporate finances as a whole to prevent over-extension.

Proactive data management was vital to reduce the amount of duplicated data stored, both in the live systems and in the tape archives. End users and IT analysts alike were charged with managing inboxes and file stores. Obviously this added to the administrative burden of keeping systems running, but it also helped to improve the quality of the data stored.

The modern IT environment is somewhat less disciplined thanks to three key factors:

1. Cheaper storage

The cost of storage continues to fall, making the purchase of capacity an almost annual exercise. The ability to add capacity means that many businesses have relaxed their data management regimes, creating a self-perpetuating cycle of consumption by duplicate data.

As systems approach capacity, most businesses simply purchase more disk space, delaying data de-duplication indefinitely. 

2. Data retention legislation

Many national governments now require businesses to retain certain information to assist with criminal investigations or other compliancy demands. Confusion about what is required, particularly for multinational enterprises that operate in multiple jurisdictions, have led some to simply retain everything, rather than risk losing a key item that later results in a lawsuit.

This uncertainty may lead to data being further duplicated to ensure copies are available elsewhere in the organisation to maintain compliance in each market.

3. Big Data hype

Big Data techniques are built on the principle of having huge amounts of data available for processing and analysis. However some organisations take data retention to the extreme, keeping everything, whether it will be part of their Big Data strategy or not.

As well as collecting a lot of irrelevant information, the “keep everything” approach means that duplicate data is spreading through live systems and archives consuming resources unnecessarily – even if businesses use the Cloud for its “infinite capacity” capabilities.

Duplicate data won’t kill your enterprise, but it will eat your profits

The reality is that wasted storage capacity is an unnecessary running cost that directly affects profitability. Your business is unlikely to go bust because of duplicate data, but reduced margins are unlikely to impress shareholders. 

CDS will be publishing a whitepaper looking at the on-going challenges of data duplication in the next few weeks. In the meantime, please get in touch with the CDS team to discuss how we can help you better manage your use of storage and reduce duplicate data.