IT Brief New Zealand logo
Technology news for New Zealand's largest enterprises
Story image

How to avoid your data lake becoming a data ‘swamp’

By Julia Gabel
Fri 25 May 2018
FYI, this story is more than a year old

Article by Steve Singer, A/NZ country manager at Talend

In today’s highly competitive, always-on business world, organisations are seeking to gain an edge through digital transformation.

As a result, many feel a sense of urgency to change everything they do - from manufacturing to customer service - in a constant pursuit of continuous innovation and process efficiency.

Data sits at the heart of all these digital transformation projects. It’s the critical component that helps generate smarter, improved decision making by empowering business users to eliminate gut feelings, unclear hypotheses, and false assumptions.

For this reason, many organisations believe building a massive data lake is the ‘silver bullet’ for attaining real-time business insights.

In fact, according to a study by CIO from IDG, 75% of business leaders believe their future success will be driven by their organisation’s ability to make the most of their information assets.

However, only 4% of these organisations say they have a data-driven approach that will allow them to benefit from their data.

Growing complexity

The reality is that the new initiatives and technologies being rolled out come with a unique set of generated data which, in turn, creates additional complexity in the decision-making process.

To cope, a growing number of organisations are migrating to the cloud. However, this ends up creating other issues. For example, once data is made more broadly available via the cloud, more employees want access to it.

They’re keen to extract value from increasingly diverse data sets, faster than ever before.

This desire puts pressure on IT teams to deliver real-time, data access that serves the diverse needs of users looking to apply real-time analytics to support their roles.

Those users are also looking for ways to prepare, share, and manage data more easily.

The rise of the data lake

To achieve this, many organisations decided to move raw data to one place where everybody can access it, thus creating a data lake.

In essence, a data lake is a large body of raw data held in a natural state where different users can examine it, delve into it, or extract samples from it.

However, organisations are beginning to realise that all the time and effort spent building massive data lakes has frequently made things worse due to poor data governance and management. All too often the result can become a ‘data swamp’.

Management is key

In the same way data warehouses failed to manage data analytics a decade ago, data lakes will undoubtedly become data swamps if not correctly managed. Indeed, putting all data in a single place won’t by itself solve the broader data access problem.

Leaving data uncontrolled, un-enriched, unqualified and unmanaged will dramatically hamper the benefits of a data lake as it will still have the ability to only be utilised properly by a limited number of experts with a unique set of skills.

The long-term value of an enterprise data lake depends on the level of trust employees have in the data it contains.

Failing to control data accuracy and quality will create mistrust amongst employees, seed doubt about the competency of IT, and jeopardise the whole data value chain.

The benefits of a cloud data warehouse

Increasing numbers of firms are deciding that a governed cloud data lake represent an adequate solution to overcoming some of the traditional stumbling blocks.

The four key steps to follow to adopt this approach are:

Unite all data sources

Ensure your organisation has the capacity to integrate a wide array of data sources, formats and sizes.

Storing a wide variety of data in one place is the first step, but it’s not enough. Bridging data pipelines and reconciling them is another way to gain the capacity to manage insights.

Accelerate trusted insights 

Efficiently manage data with cloud data integration solutions that help prepare, profile, cleanse, and mask data while monitoring data quality over time regardless of file format and size. 

When coupled with cloud data warehouse capabilities, data integration can enable companies to create trusted data for access, reporting, and analytics in a fraction of the time and cost of traditional data warehouses. 

Embrace collaborative data governance

The old schema of a data value chain, where data is produced solely by IT in data warehouses and consumed by business users, is no longer valid. 

Now everyone wants to create content, add context, enrich data, and share it with others.

Companies should encourage a collaborative governance by delegating appropriate role-based, authority or access rights to citizen data scientists, line-of-business experts, and data analysts.

Democratise data access

Without making people accountable for what they’re doing, analysing, and operating, there is little chance organisations will succeed in implementing the right data strategy across business lines.

Thus, you need to build a continuous ‘data value chain’ where business users contribute, share, and enrich the data flow in combination with a cloud architecture that will accelerate data usage by load balancing data processing across diverse audiences.

Data has very much become a strategic asset, however, it’s all too often more like a hidden treasure at the bottom of many companies.

Once modernised, shared and processed using AI and the cloud, data will reveal its true value and deliver better and faster insights to help companies get ahead of the competition.

Related stories
Top stories
Story image
Tech job moves - Forcepoint, Malwarebytes, SolarWinds & VMware
We round up all job appointments from May 13-20, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Let’s clear the cloud visibility haze with app awareness
Increasingly, organisations are heading for the cloud, initiating new born-in-the-cloud architectures and migrating existing applications via ‘lift and shift’ or refactoring.
Story image
Remote Working
Successful digital transformation in the hybrid work era is about embracing shifting goalposts
As organisations embraced remote working, many discovered they lacked the infrastructure needed to support history’s first global load test of remote work capabilities.
Story image
Vectra AI
Understanding the weight on security leader’s shoulders, and how to shift it
Millions of dollars of government funding and internal budgets are being funnelled into cybersecurity to build resilience against sophisticated threats, indicating how serious this issue has become.
Story image
Data Protection
Barracuda launches new capabilities for API Protection
"Every business needs this type of critical protection against API vulnerabilities and automated bot attacks," Barracuda says.
Story image
Supply chain
Jetstack promotes better security with supply chain toolkit
The web-based resource is designed to help organisations evaluate and plan the crucial steps they need to establish effective software supply chain security.
Story image
Cloud Security
Aqua Security createa unified scanner for cloud native security
“By integrating more cloud native scanning targets into Trivy, such as Kubernetes, we are simplifying cloud native security."
Story image
Digital Transformation
The Huawei APAC conference kicks off with digital transformation
More than 1500 people from across APAC have gathered for the Huawei APAC Digital Innovation Congress to explore the future of digital innovation.
Story image
Sift shares crucial advice for preventing serious ATO breaches
Are you or your business struggling with Account Takeover Fraud (ATO)? One of the latest ebooks from Sift can provide readers with the tools and expertise to help launch them into the new era of account security.
Story image
Cybersecurity starts with education
In 2021, 80% of Australian organisations responding to the Sophos State of Ransomware study reported being hit by ransomware. 
Find out how a behavioural analytics-driven approach can transform security operations with the new Exabeam commissioned Forrester study.
Link image
Booster Innovation Fund. A fund of Kiwi ingenuity – for Kiwi investors.
Link image
Story image
Artificial Intelligence
CFOs using digital workers and AI to prevent unnecessary loss
New technology is now allowing CFOs to use digital workers to automate their accounting processes, making it easier for them to avoid unnecessary losses.
Story image
Artificial Intelligence
SAS announces new products amid cloud portfolio success
Analytics and AI company SAS is deepening its broad industry portfolio with offerings that support life sciences, energy, and martech.
For every 10PB of storage run on HyperDrive vs. comparable alternatives, an estimated 6,656 tonnes of CO₂ are saved by reduced energy consumption alone over its lifespan. That’s the equivalent of taking nearly 1,500 cars off the road for a year.
Link image
Story image
Apple previews new features for users with disabilities
Apple says new software features that offer users with disabilities new tools for navigation, health and communication, are set to come out later this year.
Story image
Wireless Nation
Wireless Nation, N4L provide 4G network to remote NZ schools
Wireless Nation and Network for Learning (N4L) have rolled out the Rural Connectivity Group’s (RCG) new 4G network to better connect three Chatham Islands schools.
Story image
Artificial Intelligence
AI-based email security platform Abnormal Security valued at $4B
"A new breed of cybersecurity solutions that leverage AI is required to change the game and stop the rising threat of sophisticated and targeted email attacks."
Story image
Meadows exits Ingram Micro New Zealand on a good note
We look at the resignation of Danny Meadows from Ingram Micro's New Zealand Unified Communications & Collaboration business unit.
Story image
More than 40% of banks worried about cloud security - report
Publicis Sapient's new report finds security and the lack of cloud skills and internal understanding of business benefits are big obstacles for banks moving to the cloud.
Threat actors are exploiting weaknesses in interconnected IT/OT ecosystems. Darktrace illuminates your entire business and takes targeted action to stop emerging attacks.
Link image
Story image
Digital Transformation
How to modernise legacy apps without compromising security
At a time when digital transformation has become central to business, even the most important applications come with a ‘use-by’ date.
Story image
Could your Excel practices be harming your business?
While Excel has been the de-facto standard for budgeting, planning, and forecasting, is it alone, enough to support organisations in the global marketplace that’s facing rapid changes due to digital transformation?
Story image
Digital Transformation
Pluralsight and Ingram Micro Cloud team up on cloud initiative
Pluralsight has teamed with Ingram Micro Cloud to build upon cloud competence and maturity internally, and externally support partners’ capabilities.
Story image
Application Security
What are the DDoS attack trend predictions for 2022?
Mitigation and recovery are vital to ensuring brand reputation remains solid in the face of a Distributed Denial of Service (DDoS) attack and that business growth and innovation can continue.
Story image
A10 Networks finds over 15 million DDoS weapons in 2021
A10 Networks notes that in the 2H 2021 reporting period, its security research team tracked more than 15.4 million Distributed Denial-of-Service (DDoS) weapons.
Story image
Maintaining secure systems with expectations of flexible work
Most office workers feel they've proved they can work successfully from home, and as much as employers try, things aren't going back to the way they were anytime soon.
Story image
Alarming surge in Conti Ransomware Group activity - report
A new report has identified a 7.6 per cent increase in the number of vulnerabilities tied to ransomware in Q1 2022.
Story image
Digital Transformation
The impact of COVID-19 on healthcare environments and care delivery
The COVID-19 pandemic has revolutionised the healthcare industry while overcoming staff shortages, social distancing requirements, and lockdowns.
Story image
Digital Transformation
Trading up: It's time to swap core systems for flexible digital applications
This year will see more oranisations planning and commencing high tech renovations that will shake up the way they operate.
Story image
Customer experience
Gartner recognises Okta for abilities in Access Management
Okta has announced it has been recognised as a Customers' Choice for the fourth time in a row in the Gartner Peer Insights "Voice of the Customer" report.
Story image
Grasping the opportunity to rethink the metrics of a sustainable data centre
A data centre traditionally has two distinct operations teams: the Facility Operations team, and the IT Operations team. Collaboration between them is the key to defining, measuring, and delivering long-term efficiency and sustainability improvements.
Story image
Revenue operations is taking centre stage
As the business world continues to evolve, new demands need to be met to keep up with the ever-changing landscape. 
Story image
Lightspeed launches all-in-one marketing platform in A/NZ
ECommerce provider, Lightspeed has launched a new all-in-one marketing solution, Lightspeed Marketing & Loyalty in Australia and New Zealand.
Story image
Amazon Web Services / AWS
RedShield leverages AWS to scale cybersecurity services
"Working with AWS gives RedShield the ability to mitigate significant application layer DDoS attacks, helping leaders adopt best practices and security architectures."
Story image
ChildFund launches new campaign to protect children online
ChildFund says WEB Safe & Wise aims to protect children from sexual exploitation and abuse online while also empowering them to become digitally savvy. 
Story image
Fonterra to use automated guided vehicles by Dematic
New Zealand's dairy giant Fonterra is upgrading its manufacturing facility in Edendale with new automated guided vehicles (AGVs) from Dematic.
Story image
Data and analytics could be key to higher selling prices in APAC
Sisense's latest report has found that almost half of data professionals in APAC think customised data and analytics can create better selling prices for their products.
Story image
Artificial Intelligence
How to ensure ethical deployment of AI implementations
The increase in automation and machine technology such as AI and machine learning has unlocked a whole new level of scale and service to organisations. 
Story image
Power at the edge: the role of data centers in sustainability
The Singaporean moratorium on new data center projects was recently lifted, with one of the conditions being an increased focus on power efficiency and sustainability.
Story image
A third of companies paying ransom don’t recover data - report
Veeam's report finds 76% of businesses who are victims of cyberattacks paid the ransom to recover data, but a third were still unable to get their information back.
Story image
Data backup plans inadequate, data still at risk - study
The Apricorn 2022 Global IT Security Survey revealed that while the majority organisations have data backup plans in place, data for many are at risk.
Story image
Vodafone NZ buys remaining stake in retail joint venture
Vodafone New Zealand has purchased the remaining 50% stake in the specialist joint venture (JV) with private equity company Millennium Corp.
Story image
Data Center
Preventing downtime costs and damage with Distributed Infrastructure Management
Distributed Infrastructure Management (DIM) can often be a lifeline for many enterprises that work with highly critical ICT infrastructure and power sources.