Story image

Cloud data warehouse trends and best practices

21 Feb 2019

According to a new survey conducted by TDWI on behalf of Talend, new cloud data warehouses (CDW) offer broader data capabilities, stronger performance, and greater flexibility than traditional on-premise databases. 

However, the survey found that while CDWs are often an important first step in digital transformation, enterprises need to follow some best practices to overcome implementation challenges and increase investment return.

TDWI senior research director Philip Russom says, “TDWI sees a wide range of data-driven IT systems moving to the cloud aggressively, and this includes the data warehouse. 

“Cloud gives the data warehouse the elastic scale, agnostic storage, multi-tenant access, and controlled cost it needs for modern requirements.”

“However, cloud data warehouses should be complemented with substantial data integration infrastructure to unify the many pieces of the warehouse with all the data sources and targets available.”   Decision Resources Group (DRG) is one example of this. As a company that manages comprehensive data repositories covering 90% of the U.S. healthcare system, DRG was struggling to combine and organise their disparate data sources. 

Healthcare data is stored in a structured way, creating the need for DRG to clean and normalise millions of records and group data to assess patient needs and market conditions. 

DRG was successful in switching to a cloud-first strategy by implementing Talend and the Snowflake cloud data warehouse as the foundation of its new Real World Data Platform. Because of this, DRG became 150% more productive without increasing costs, and have since onboarded 100 terabytes of data in just three months. 

The organisation can now supply more meaningful data enabling physicians to understand different patient populations and provider markets to interact with them in a more optimised fashion.   While survey respondents noted that adopting CDWs was critical to helping them achieve faster performance and lower costs, and take advantage of cloud features, there were a number of challenges associated with CDWs as well. 

Over 50% of respondents indicated data governance as a top challenge, closely followed by integrating data across multiple sources at over 40% and getting data into the warehouse at about 38%. 

Organisations data analytics needs in a CDW are becoming increasingly complex. Over 35% of respondents expressed the need for in-memory processing, supporting structured and unstructured data, and integration with third-party analytics tools. 

As a result, CDWs need to accommodate a wide variety of data and serve a broad range of technical use cases.      Interestingly, 62% of respondents in the process of implementing CDWs want them to complement a data lake for analytics. 

All survey respondents were interested in features such as data quality, metadata management, processing and transforming data both before and after data is loaded to a CDW. 

As these requirements cannot be met solely by CDW technologies, the response suggests a need for integration solutions to complement the infrastructure. 

CDWs have to be enabled to accommodate a range of use cases, from business to technical, and support increases in speed and scale, while handling both current and future needs. 

What the future of fibre looks like in NZ
The Commerce Commission has released its emerging views paper on the rules, requirements and processes which will underpin the new regulatory regime for New Zealand’s fibre networks.
Gen Z confidence in the economy is on the decline
Businesses need to work hard to improve their reputations.
Why NZ businesses have less than two years to adopt digital before disruption hits
Research found that digital disruption is already impacting two-thirds of New Zealand organisations.
Dell EMC launches interactive AI Experience Zones
The AI Experience Zones are designed to educate visitors about how to start, identify, and implement an AI project.
What NZ can learn from the Baltimore cyberattack
“Businesses must control physical access to their computers and secure their networks."
Infratil seeks clearance to acquire up to 50% stake in Vodafone NZ
The commission will give clearance to a proposed merger if they are satisfied that the merger is unlikely to have the effect of substantially lessening competition in a market.
Hands-on review: MiniTool Power Data Recovery Software
I came across a wee gem of advice when researching the world of data recovery. As soon as you get that sinking feeling and realise you’ve lost a file, stop using your computer.
Deepfakes the 'next wave of concern' - but can law really stomp it out?
Enforcing the existing law will be difficult enough, and it is not clear that any new law would be able to do better. Overseas attempts to draft law for deepfakes have been seriously criticised.