IT Brief New Zealand logo
Technology news for New Zealand's largest enterprises
Story image

Data evolution: Past, present and future

By Contributor
Wed 1 May 2019
FYI, this story is more than a year old

Article by Denodo Technologies CEO and founder Angel Viña

As the needs of enterprises have shifted and come to rely on data as possibly the single biggest asset driving business outcomes, the data landscape has shifted as well. Tracing back through the evolution of data proves to be incisive – it has become evident that our attitude to data, and how it serves us, has changed dramatically.

In the 1980s, when all data was stored in a single database, our concept of data and its uses was based on the idea of a central repository. From there, the single database evolved into data warehouses, which allowed data to be ingested and stored according to its source and category, but in doing so it became siloed, and ultimately proved to be inefficient.

Enterprises later began to realise the power of analytics, and early-generation analytics programs were written to sit above the data layer and derive business value from it. The data warehouse evolved into mini data warehouses to avoid bottlenecks created from having one mass repository, and then evolved again into the enterprise data warehouse, with integrations and layers of analytic software built into them. By the mid-2000s, Hadoop changed things up once more, building code into the data itself, so that it became more fluid and easily accessible.

It was also in this period where we witnessed a surge in the number of disparate data sources that an enterprise had at its disposal, with the rise of social media and mobile applications. This brought a whole lot more data into the business, much of which was unstructured rather than structured.

However, companies required an unobstructed view of both data sets to maximise the benefits and effectiveness of the intelligence within their enterprise. Storing all data in one place was no longer an effective strategy, as it became hard to access and even harder to analyse effectively. Deriving true business value from both structured and unstructured data meant it was not stored in a single place, introducing a host of other inefficiencies.

One source of data for finance, for example, would need its own data warehouse, as would sales, as would marketing. Much of that data would be common across several departments, so the governance of data became much more complicated as well.

Throughout this brief history of big data storage, a common theme remains constant — the need to access data quickly and effectively. This has always been the central concern for an enterprise needing fast, efficient access to data for generating reports, predicting consumer behaviours, employing predictive maintenance on key assets, and enabling strategic decision-making.

The next phase of data access in the enterprise is to move to a scenario where data resides in source systems, but is accessed virtually from one single platform. Just as virtualisation was a highly effective model for moving information and apps into the cloud, data virtualisation will be crucial in enabling the next stage of the data evolution.

The logical approach to accessing different types of data across disparate sources and systems has been to bring it together into one single pool rather than store it in one repository. Yet data virtualisation can allow enterprises to connect to this pool wherever it happens to be, even across distributed systems, without having to physically bring it together into the central repository.

With data divided into structured and unstructured forms and streaming in from across disparate systems, it would be almost impossible to process without automation. Currently, enterprises generally process data at great speed, even as it is collected.

However more than raw data, people require information in their hands in real time, or at least as close to real time as is possible. To get from data to information, there is a significant need to process data and convert it into intelligence and business insights even as more data is streaming onto the business.

Now that artificial intelligence and machine learning are becoming fundamental parts of business functionality, automating the value derived from data makes a lot of sense to speed up processes and remove inefficiencies.

Beyond this, data virtualisation creates intelligence through the metadata layer, which allows businesses to understand and relate to changes occurring in source systems, and have that aggregated data available to end users. Virtualisation can also provide information on the lineage of data, indicating how it has transformed over time, where it has come from, how it has associated with other data sources over the life of its journey.

This in turn can provide deep insights into core business functions such as sales and marketing — mapping and preventing customer churn, indicating what products they may have purchased in the past, whether they have warranties for that product, or if they have upgraded services at any stage as a few use cases.

Automated, virtualised data also catalogues this information across all systems, documenting changes so that the organisation does not have to manually reconcile it. This can provide users insights into data literally as events occur to all areas of the business. Hot ticket items suddenly trending online? Retailers can set an alert, mark up prices, or order more stock in real time, reacting to what business intelligence reveals in real time.

The data landscape has changed and it is clear that the virtualisation of data is the next stage of its evolutionary journey. We’ve moved beyond the physical movement of data, reducing the costs associated with that process, of storing it in one large repository. This equates to reduced overheads, coupled with the savings in time afforded by a more efficient system.

Combine that with a better, more holistic view of data, and the ability to assimilate information in real time as it streams into the organisation, and the business benefits become obvious. Virtualisation of data is the next vital step towards running a faster, more efficient and cost-effective enterprise.

Related stories
Top stories
Story image
ChildFund
ChildFund launches new campaign to protect children online
ChildFund says WEB Safe & Wise aims to protect children from sexual exploitation and abuse online while also empowering them to become digitally savvy. 
Story image
Data Protection
Barracuda launches new capabilities for API Protection
"Every business needs this type of critical protection against API vulnerabilities and automated bot attacks," Barracuda says.
Story image
Infrastructure
Nutanix study reveals financial services sector lagging with multicloud adoption
Nutanix has released new research that reveals the financial services sector is lagging behind when it comes to multicloud adoption.
Story image
Digital Transformation
The Huawei APAC conference kicks off with digital transformation
More than 1500 people from across APAC have gathered for the Huawei APAC Digital Innovation Congress to explore the future of digital innovation.
Story image
Remote Working
How zero trust and SD-WANs can support productive remote working
The way people connect with applications and data has changed, users are remotely accessing resources that could be stored anywhere from a corporate data center to the cloud.
Story image
Fonterra
Fonterra to use automated guided vehicles by Dematic
New Zealand's dairy giant Fonterra is upgrading its manufacturing facility in Edendale with new automated guided vehicles (AGVs) from Dematic.
Story image
Digital Transformation
Pluralsight and Ingram Micro Cloud team up on cloud initiative
Pluralsight has teamed with Ingram Micro Cloud to build upon cloud competence and maturity internally, and externally support partners’ capabilities.
Story image
Artificial Intelligence
CFOs using digital workers and AI to prevent unnecessary loss
New technology is now allowing CFOs to use digital workers to automate their accounting processes, making it easier for them to avoid unnecessary losses.
Story image
Collaboration
Meadows exits Ingram Micro New Zealand on a good note
We look at the resignation of Danny Meadows from Ingram Micro's New Zealand Unified Communications & Collaboration business unit.
Story image
Cybersecurity
NCSC advisory highlights poor security configurations
The GCSB's National Cyber Security Centre (NCSC) has released a cyber security advisory identifying commonly exploited controls and practices.
Story image
Cybersecurity
The 'A-B-C' of effective application security
Software applications have been a key tool for businesses for decades, but the way they are designed and operated has changed during the past few years.
Story image
Application Security
What are the DDoS attack trend predictions for 2022?
Mitigation and recovery are vital to ensuring brand reputation remains solid in the face of a Distributed Denial of Service (DDoS) attack and that business growth and innovation can continue.
Story image
Cybersecurity
A10 Networks finds over 15 million DDoS weapons in 2021
A10 Networks notes that in the 2H 2021 reporting period, its security research team tracked more than 15.4 million Distributed Denial-of-Service (DDoS) weapons.
Story image
Cybersecurity
Managed service providers: effective scoping to avoid costly vendor pitfalls
Managed security services are outsourced services focusing on the security and resilience of business networks.
Story image
Sisense
Data and analytics could be key to higher selling prices in APAC
Sisense's latest report has found that almost half of data professionals in APAC think customised data and analytics can create better selling prices for their products.
Story image
SaaS
Rubrik Security Cloud marks 'next frontier' in cybersecurity
"The next frontier in cybersecurity pairs the investments in infrastructure security with data security giving companies security from the point of data."
Story image
Wireless Nation
Wireless Nation, N4L provide 4G network to remote NZ schools
Wireless Nation and Network for Learning (N4L) have rolled out the Rural Connectivity Group’s (RCG) new 4G network to better connect three Chatham Islands schools.
Story image
Apricorn
Data backup plans inadequate, data still at risk - study
The Apricorn 2022 Global IT Security Survey revealed that while the majority organisations have data backup plans in place, data for many are at risk.
Story image
Ransomware
A third of companies paying ransom don’t recover data - report
Veeam's report finds 76% of businesses who are victims of cyberattacks paid the ransom to recover data, but a third were still unable to get their information back.
Story image
Sift
Sift shares crucial advice for preventing serious ATO breaches
Are you or your business struggling with Account Takeover Fraud (ATO)? One of the latest ebooks from Sift can provide readers with the tools and expertise to help launch them into the new era of account security.
Story image
Artificial Intelligence
How to ensure ethical deployment of AI implementations
The increase in automation and machine technology such as AI and machine learning has unlocked a whole new level of scale and service to organisations. 
Story image
Lightspeed
Lightspeed launches all-in-one marketing platform in A/NZ
ECommerce provider, Lightspeed has launched a new all-in-one marketing solution, Lightspeed Marketing & Loyalty in Australia and New Zealand.
Story image
Cloud Security
Aqua Security createa unified scanner for cloud native security
“By integrating more cloud native scanning targets into Trivy, such as Kubernetes, we are simplifying cloud native security."
Darktrace
Threat actors are exploiting weaknesses in interconnected IT/OT ecosystems. Darktrace illuminates your entire business and takes targeted action to stop emerging attacks.
Link image
Story image
Ransomware
Cybersecurity starts with education
In 2021, 80% of Australian organisations responding to the Sophos State of Ransomware study reported being hit by ransomware. 
Story image
Attain
Revenue operations is taking centre stage
As the business world continues to evolve, new demands need to be met to keep up with the ever-changing landscape. 
Story image
Cybersecurity
Hard numbers: Why ambiguity in cybersecurity no longer adds up
As cybersecurity costs and risks continue to escalate, CEOs continue to struggle with what their investment in cyber protection buys. Getting rid of ambiguity becomes necessary.
Story image
Commerce Commission
ComCom appeals $2.25 million fine in Vodafone FibreX case
The Commerce Commission has filed an appeal in the High Court against a $2.25 million fine imposed on Vodafone NZ for its offending under the Fair Trading Act.
Story image
Digital Transformation
How to modernise legacy apps without compromising security
At a time when digital transformation has become central to business, even the most important applications come with a ‘use-by’ date.
Exabeam
Find out how a behavioural analytics-driven approach can transform security operations with the new Exabeam commissioned Forrester study.
Link image
Story image
Data Center
Preventing downtime costs and damage with Distributed Infrastructure Management
Distributed Infrastructure Management (DIM) can often be a lifeline for many enterprises that work with highly critical ICT infrastructure and power sources.
Story image
Sustainability
Grasping the opportunity to rethink the metrics of a sustainable data centre
A data centre traditionally has two distinct operations teams: the Facility Operations team, and the IT Operations team. Collaboration between them is the key to defining, measuring, and delivering long-term efficiency and sustainability improvements.
Booster
Booster Innovation Fund. A fund of Kiwi ingenuity – for Kiwi investors.
Link image
Story image
Vodafone
Vodafone NZ buys remaining stake in retail joint venture
Vodafone New Zealand has purchased the remaining 50% stake in the specialist joint venture (JV) with private equity company Millennium Corp.
Story image
Excel
Could your Excel practices be harming your business?
While Excel has been the de-facto standard for budgeting, planning, and forecasting, is it alone, enough to support organisations in the global marketplace that’s facing rapid changes due to digital transformation?
Story image
CRM
Zendesk announces new conversational CRM solutions
“The last few years have made it obvious that digital is the front door, convenience is paramount and relationships are anchored in conversations."
Story image
Amazon Web Services / AWS
RedShield leverages AWS to scale cybersecurity services
"Working with AWS gives RedShield the ability to mitigate significant application layer DDoS attacks, helping leaders adopt best practices and security architectures."
Story image
Digital Transformation
Trading up: It's time to swap core systems for flexible digital applications
This year will see more oranisations planning and commencing high tech renovations that will shake up the way they operate.
Softiron
For every 10PB of storage run on HyperDrive vs. comparable alternatives, an estimated 6,656 tonnes of CO₂ are saved by reduced energy consumption alone over its lifespan. That’s the equivalent of taking nearly 1,500 cars off the road for a year.
Link image
Story image
Apple
Apple previews new features for users with disabilities
Apple says new software features that offer users with disabilities new tools for navigation, health and communication, are set to come out later this year.
Story image
Workato
Workato unveils enhancements to enterprise automation platform
"The extra layer of protection with EKM, zero-logging, and hourly key rotation gives customers a lot more visibility and control over more sensitive data."
Story image
Wireless
Cradlepoint expands its Cellular Intelligence capabilities
Cradlepoint has announced additional Cellular Intelligence capabilities with its NetCloud service.
Story image
Digital Transformation
The impact of COVID-19 on healthcare environments and care delivery
The COVID-19 pandemic has revolutionised the healthcare industry while overcoming staff shortages, social distancing requirements, and lockdowns.
Story image
Sustainability
Power at the edge: the role of data centers in sustainability
The Singaporean moratorium on new data center projects was recently lifted, with one of the conditions being an increased focus on power efficiency and sustainability.