itb-nz logo
Story image

Unstructured data: Making sense of the mass and the mess

24 Jun 2020

Article by Pure Storage Asia Pacific and Japan's FlashBlade vice president, Sunil Chavan.

Enormous data growth is coming. In fact, market intelligence firm IDC predicts data will increase 61% to 175 zettabytes by 2025. Organisations today are amassing vast amounts of data; everything from customer information to the burgeoning mountain of sensor and machine data from the adoption of the Internet of Things. 

While a lot has been said about the potential of data, unstructured data accounts for 80 - 90% of the overall digital data footprint. This unstructured data, which exists in forms such as images, search data, videos and sensor data, is largely unsearchable and difficult to analyse--posing a significant challenge for businesses. 

What’s more, much of this data exists in complex infrastructure and silos, such as data lakes, data warehouses, storage area networks and backup systems. At one point in time, these silos served a purpose. Now, they create massive headaches for IT teams, who have to manage these complex systems, and end users, by making it harder to pull information from across the organisation to extract insights.

The good news is, organisations at the cutting edge of their industries have grasped that what was once “cold” or “dark” data has an increasingly important role to play in helping them to maintain their market-leading positions. Examples of these workloads can be found in areas such as genomics, semiconductor design, data analytics, media and entertainment, oil and gas exploration and computer-aided engineering (CAE), among others. Clearly, being able to gain useful insights from unstructured data is crucial to producing an overall competitive advantage for organisations in a variety of sectors.

Producing accurate analytics and training data-hungry artificial intelligence and machine learning algorithms requires an intensity, speed and volume that can often cripple legacy storage systems and media. Therefore, finding needles in the haystacks of unstructured data requires a long hard look at the existing storage foundation. 

A modern data experience requires infrastructure that can bridge silos and meet demands for performance, agility and simplicity, without complexity or compromise. This calls for a data hub approach. 

At the core of data hubs is a data-centric architecture designed for the very purpose of data sharing across the four many silo groups: data warehouses, data lakes, streaming analytics and AI clusters. This approach integrates their strengths onto a single, unified platform, eliminating bottlenecks across the applications that need data for better insight. Additionally, data hubs are simple and elastic, allowing applications to spin up and down resources as needed.

Pure customer Liquidnet built a data hub based on FlashBlade and FlashArray to perform timely trades and to generate real-time analytics that its institutional clients rely on. Delivering real-time analytics previously was not feasible because legacy storage systems lacked the I/O and parallelism needed to move very large data sets fast enough. FlashBlade provided these vital capabilities: the ability to process large volumes of streaming data in real time and the ability to make data sets available to multiple workloads simultaneously across multiple applications like ElasticSearch, Spark and Kafka.

The tremendous growth of unstructured data is creating huge opportunities for organisations. Until recently, the potential to maximise unstructured data had been restricted by the limitations of legacy storage systems. Fortunately, with a modern data hub architecture built on a next-generation storage infrastructure, such as Pure FlashBlade, organisations can extract real-time insights at the speed of business while enjoying cloud scalability and operational simplicity.

If you would like to learn more about how FlashBlade can help your organisation, visit purestorage.com/flashblade.

Story image
AR and VR presents huge potential for construction industry, but businesses slow to adopt
According to GlobalData, the construction industry is slowly shifting from years of the wait-and-watch stance to adopting digital technologies to improve the overall project lifecycle from conceptual design to construction.More
Story image
Lenovo launches two new ThinkSystem computing solutions
The company has launched its Lenovo ThinkSystem SD650-N V2 and ThinkSystem SR670 V2 solutions, which uses Lenovo's Neptune liquid cooling technology and NVIDIA GPUs.More
Story image
Pure Storage to offer validation for integrated partner solutions
Pure Validated Designs from Commvault and Vertica deliver blueprints for data protection and analytics on Pure architecture with more to come.More
Story image
COVID-19 creating increased global demand for NZ’s digital businesses
Activity is picking up for many Kiwi software and digital businesses as demand increases for digital and cloud-based service offerings.More
Story image
DevSecOps increasingly important, but APAC organisations lagging behind
The rise of DevSecOps comes at a time when IT leaders are faced with an increasingly active cyber threat landscape, coupled with higher consumer expectations of digital offerings and application usage due to a sharp increase in online activities.More
Story image
Businesses struggling to achieve cloud migration in wake of COVID-19
Cloud adoption has increased due to the COVID-19 pandemic, but businesses are struggling to meet their cost and performance needs due to migration challenges, new research finds.More