IT Brief New Zealand logo
Technology news for New Zealand's largest enterprises
Story image

Alleviating the data boom

Mon 1 Nov 2010
FYI, this story is more than a year old

Unstructured data is found in very different formats such as email attachments, documents, spreadsheets, and video images. The total volume of unstructured data stored is forecast to grow at a compound annual growth rate in excess of 60%. Total digital information stored is estimated to double every 18 months. However, only an estimated 5% of that data will be traditional structured data.  Structured data, which is typically generated by a financial transaction, is stored in a relational database. This makes it easy to understand both the content and the context of the data. This is not the case with unstructured data. This makes it difficult to know what backup and retention policies should apply and how to get the best return from that information.Why doesn’t existing technology work?Unstructured data is usually stored outside of a database, which makes it difficult to manage. Usually, the context of the file is lost or separated from the data. In addition, the content within unstructured data is not easily understood. This makes storage decisions difficult because, for example, different spreadsheets or documents will have very different priorities, depending on their content. Storage policies based simply on file type will not suffice. Current data storage technologies such as Network Attached Storage (NAS) were originally designed to allow concurrent access to data from multiple users. File systems store data in hierarchical systems, or "trees”, of directories, folders, subfolders and files. This allows rapid retrieval of data without much metadata, or information associated with the underlying data, being required. In contrast, the lack of content information in the metadata for unstructured data means that files cannot be differentiated for appropriate storage policies, so generally a blanket approach is taken, such as storing a particular file type indefinitely. As the volume of unstructured data grows, the number of nested folders grows which makes retrieval challenging, particularly if the metadata is limited.  As the tree structure grows, the performance of the file system deteriorates and backups become more problematic.In essence, conventional file systems such as NAS are not well suited to manage the explosive growth in unstructured data. High costs, unnecessary complexity and poor performance have led to a new solution for unstructured data.What is the solution?Object-based storage addresses the challenges of unstructured data storage by retaining rich metadata about a data object. This allows both the context and the content relating to the underlying data to be preserved. Whereas a conventional file system may store basic metadata relating to file name, originator, the date of file creation or modification and file type, the object-based storage approach retains multiple elements that enable both the identification of the content as well as providing information to help determine how the data should be treated over its lifecycle.The benefit of object storage is that retrieval of information may be made without knowing the file name, dates or file designations.  In addition, the rich metadata may be employed to route the file through a policy-based process to maximise its economic benefit to the firm, as well as minimising the cost of retaining it.In addition, object-based storage makes e-discovery of retained data and its exposure to business intelligence processes so much easier. The rich metadata in object-based storage also makes it easy to store related information, which may be stored as disparate files, in groups. For example, an engineer may have text files, video files, spreadsheets, engineering drawing images associated with a particular project all together, in a similar way that paper-based records are kept.A further benefit of object-based storage is that, unlike a conventional file, each object is assigned a unique ID. This is generated using a 128-bit random number generator, thus ensuring uniqueness and allowing an internet-type access system. This also means that the object does not need a specified location for retrieval and millions of objects can be stored in a flat address space without a complex file system or "trees”, thus dramatically reducing complexity and costs. The migration of data from one system to another system is made much simpler and less disruptive.Furthermore, the presence of a hash signature, together with the object’s ID, allows easy deduplication purges and also helps to prove the integrity of the data where compliance is important.  That is, the hash signature is generated according to the data within the object. If the hash signature remains the same, it can provide proof that the data is unmodified. This provides an effective means of archiving data that has high legal and compliance properties.Finally, object-based storage lowers storage costs and complexity. NAS overheads such as concurrency, permissions, file locks are not needed to the same extent, so object storage can be affordably scaled to petabytes of data. Expensive primary storage can be released for its correct purpose.The routing of files using rich metadata in object-based storage, according to policies, allows specific and appropriate treatment of objects for use and archiving rather than treating all data as equal.Why use cloud storage?Conventional file system storage will co-exist with object-based storage for quite some time. While on-premise storage costs continue to decline with improving technology, the avalanche of unstructured data points to   cloud storage for a cost-effective solution. Object storage is rapidly emerging as a standard for cloud storage. A number of cloud storage service providers employing object-based storage have emerged in the past few years, including Cleversafe, Mezeo and Iron Mountain. Amazon has had an offering for some time.Cloud storage allows the firm’s unstructured data to be cost-effectively stored while relatively scarce and expensive on-premise primary storage is dedicated to its best use. What should you be careful of in the cloud?As with all cloud computing services, caveat emptor applies. Each storage cloud is unique and appropriate due diligence must be performed prior to any commitment. The service level agreement (SLA) must make clear the terms of engagement, who owns the data, how it will be extracted and what standards are employed while the data is in custody of the service provider.The standards for object-based storage have been established by a group within the Storage Networking Industry Association (SNIA) for the T10 committee of the International Committee for Information Technology Standards (INCITS). Current object-based systems use standard APIs such as Representational State Transfer (REST) and Simple Object Access Protocol (SOAP) or proprietary APIs to tell applications how to store and retrieve object IDs. Any proprietary, non-standard based protocols are a warning sign for further due diligence testing. Object-based storage offers a powerful solution to the problem of exponential growth in the quantity of unstructured data. Cloud storage service providers offer a cost-effective alternative to an on-premise solution. Due diligence is paramount to ensure future data mobility is preserved by seeking vendors who comply with industry standards. 

Related stories
Top stories
Story image
Tech job moves
Tech job moves - ARMA International, Avec, Komo & YouGov
We round up all job appointments from August 15-18, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
CCL launches Managed M365 offering for NZ work environments
CCL, the cloud specialist within Spark Business Group, has recently launched its Managed M365 offering.
Story image
InterSystems releases updates to its IRIS data platform
Provider of next-generation solutions InterSystems has announced a series of new releases to its award-winning InterSystems IRIS data platform.
Story image
Cloud Security
Organisations struggling to secure new cloud environments
“In the wake of COVID-19, organisations substantially accelerated their digital transformation initiatives to accommodate a remote workforce."
Story image
Microsoft accelerating NZ tech sector with new data centres
Microsoft’s forthcoming New Zealand data centres are introducing hyperscale public cloud capabilities on a scale previously unseen in the country.
Story image
Exclusive: LawVu streamlines the in-house legal struggle
LawVu is a New Zealand-born company successfully transforming the in-house legal industry with its category-defining workspace.
Story image
Micro-mobility companies caught at a crossroads - study
Reformulating models, operations and goals, shared micro-mobility companies can tap into a market worth US$9 billion in ride revenues in 2030.
Story image
Cloudera launches all-in-one data lakehouse cloud service
CDP One makes it faster, easier and less risky for businesses to move to the cloud and migrate existing workloads to a modern data architecture.
Story image
On-premise micro data centres an emerging option for storage
Digital transformation is occurring in all industries, resulting in unprecedented amounts of data. Explore the option of storing your information on-site using micro data centres.
Story image
Unified Communications
MAXHUB reveals latest innovations for ANZ at Integrate 2022 in Sydney
Some of the most exciting and innovative updates from this year's Integrate 2022 event have come from the leaders at MAXHUB.
Story image
CDC hyperscale data centres now open in New Zealand
CDC Data Centres (CDC) says two new, state-of-the-art hyperscale data centres are now open for business in Auckland.
Story image
Verizon supports Fujifilm's cybersecurity advancements
Verizon Business is supporting Japan’s Fujifilm Holdings to strengthen its global cybersecurity monitoring and cyber intelligence capabilities.
Story image
Data Protection
Safeguarding your financial data
As the digital revolution marches on, managing data security has never been more important. Here are five important steps to take toward better financial data security.
Story image
Collaboration app market revenue grows 28.4% year-over-year
IDC has found that global revenues in the collaboration applications market grew 28.4% year-over-year in 2021 to $29.1 billion.
Story image
Workday research finds A/NZ organisations becoming leaders in digital agility
New research from Workday and IDC has revealed that Australia and New Zealand are becoming leaders in digital agility.
Story image
Business Intelligence / BI
TIBCO ranks highly in 2022 business intelligence study
Dresner Advisory Services has recognised TIBCO as a Service Leader and a Credibility Leader in Business Intelligence in the 2022 Wisdom of Crowds Business Intelligence Market Study.
Story image
CM Group recognised as Best Overall MarTech Company
MarTech Breakthrough has ranked CM Group as the Best Overall MarTech Company for the third year running and Cheetah Digital as Best Overall Marketing Campaign Management Solution.
Story image
Schneider Electric launches Connected Services with Claroty
Schneider Electric is teaming up with Claroty to assist in maintaining cybersecurity for physical assets - a critical issue.
Story image
Arcion extends data replication platform with new capabilities
"Arcion is committed to investing in continuous innovation. Our goal is to be the first choice in whole-product solutions for large-scale data replication."
Story image
Worldwide 5G mobile data traffic exploding - report
"With 5G, there is a wider range of deployment scenarios, forcing vendors to provide comprehensive solutions to support every need."
Story image
Whispir reports significant revenue growth as CaaS market expands
Australian cloud platform Whispir has announced its financial results, reporting significant growth as it continues to introduce new Communications-as-a Service (CaaS) offerings to the market.
Story image
Hybrid Cloud
The essential guide to digital transformation by SolarWinds
Digital transformation is a buzzword thrown around all the time by companies, but what does it actually mean and why is it important? SolarWinds breaks it down.
AWS Marketplace
Learn how security orchestration, automation, and response (SOAR) enhances your security strategy.
Link image
Story image
Gartner Magic Quadrant
Spryker named Gartner Visionary in 2022 Magic Quadrant for Digital Commerce
Spyker has announced it has been recognised by Gartner as a Visionary in the 2022 Magic Quadrant for Digital Commerce.
Story image
Lenovo launches CO2 Offset Service for SMBs across A/NZ
Lenovo has announced the rollout of a new, first-of-its-kind CO2 Offset Service for SMBs across Australia and New Zealand. 
Story image
Artificial Intelligence
Concentric AI protects sensitive data with new capabilities
The new capability reveals sensitive data shared across email and business messaging platforms and highlights who has inappropriate access to content.
Story image
8x more users attacked via old Microsoft Office vulnerability in Q2
"Criminals craft malicious documents and convince their victims to open them through social engineering techniques."
Story image
Digital Transformation
Federated change is the best path to digital evolution
Businesses that can successfully manage the exponentially expanding masses of data produced by modern consumers will be the businesses that survive and prosper.
Story image
CISOs need to consider a risk-based cybersecurity strategy
Rather than talking in terms of attack vectors and vulnerabilities, CISOs and security decision-makers must look at actual business risk.
Story image
Cradlepoint launches architectural extension of NetCloud solution
With this latest release Cradlepoint is focused on helping lean IT organisations with advanced SD-WAN and zero trust capabilities.
Story image
Customer experience
Exclusive: How Accenture is changing the customer experience game
Creating highly personalised real-world customer experiences using API, near-field communication, and spatial technology is about taking a traditional experience and elevating the customer journey into a digital world.
Story image
Facial recognition
Benefits vs risks of facial recognition technology
Once a distant, futuristic concept, facial recognition technology is now found in many technological applications with a variety of different functions. 
Story image
‘Windows shops’ target admin rights to de-risk their environments
New data shows up to 75% of critical vulnerabilities could be mitigated through a rights and privileges crackdown
Story image
Ivanti reveals potential ROI of 261% for Endpoint Management Solution
Ivanti has announced the results of a commissioned Total Economic Impact (TEI) study conducted by Forrester Consulting.
Story image
Virgin Money works with 9Spokes to create custom offering
Virgin Money has worked with Kiwi SaaS 9Spokes to build a custom version of its business dashboard offering, M-Track, to generate stronger insights for the bank and its customers.
Story image
Demand grows for future-proof mobile access solutions
HID Global, in partnership with IFSEC Global, recently released the 2022 State of Physical Access Control report for the fourth year running.
Story image
Australian Grand Prix uses CENNZnet blockchain for fan experience
CENNZnet was employed by Power'd Digital to deliver the Formula 1 Heineken Australian Grand Prix's 2022 AusGP Access program.
Story image
Cyber resilience
NZ’s Cyber Resilience Framework to be evolving and potentially automated
The government's already putting $2.4 million into the Cyber Resilience Framework in its initial stages, what is it and why is it important?
Story image
Crypto crime: Illicit activity falls with rest of market
Cryptocurrency scams, which typically present themselves as passive crypto investing opportunities, are less enticing to potential victims.
Story image
Contact Centre
Treasure Data launches new customer experience functionality
Treasure Data has introduced a new strategic vision and product positioning for Customer Data Cloud, reflecting the company's focus.
Story image
Email scams
HelpSystems shines light on impact of response-based threats
Response-based attacks targeting corporate inboxes have climbed to their highest volume since 2020, representing 41% of all email-based scams.
Story image
DDoS activity rises dramatically - Radware report
The first six months of 2022 were marked by a significant increase in DDoS activity across the globe, according to a new report.