IT Brief New Zealand logo
Technology news for New Zealand's largest enterprises
Story image

New AI capabilities enabled with LiveData now automating Apache metadata into Databricks

By Ryan Morris-Reade
Tue 25 May 2021
FYI, this story is more than a year old

WANdisco, a LiveData company, has announced its LiveData Migrator platform can now automate the migration of Apache Hive metadata directly into Databricks, enabling users to quickly enable new AI and machine learning capabilities.

The LiveData Migrator platform, which automates the migration and replication of Hadoop data from on-premises to the cloud, can help users save time and reduce costs. WANdisco says for the first time, enterprises wanting to migrate on-premises Hadoop and Spark content from Hive to Databricks, can do so at scale and efficiently, while managing the risks associated with large-scale cloud migrations.

It says datasets don’t need to be fully migrated before they are converted into the Delta format. LiveData Migrator automates an incremental transformation to Delta Lake.

Ongoing changes to source metadata are instantly reflected in the Lakehouse platform, and on-premises data formats used in Hadoop and Hive are automatically made available in Delta Lake on Databricks. 

By combining data and metadata and making on-premises content immediately usable in Databricks, users can eliminate migration tasks that previously required constructing data pipelines to transform, filter, and adjust data, along with up-front planning and staging. This means work otherwise required for setting up auto-load pipelines to identify newly-landed data then and converting it into its final form is no longer required.

“This new feature brings together the power of Databricks and WANdisco LiveData Migrator,” says WANdisco CTO, Paul Scott-Murphy. 

“Data and metadata are migrated automatically without any disruption or change to existing systems. Teams can implement their cloud modernisation strategies without risk, immediately employing workloads and data that were locked up on-premises and are now in the cloud, by using the Lakehouse platform offered by Databricks.”

Databricks VP of product partnerships, Pankaj Dugar, says enterprises often want to break silos and bring all their data into a lakehouse for analytics and AI, but they've been constrained by their on-premises infrastructure. He says with the new Hive metadata capabilities in WANdiscos LiveData Migrator, it will now be easier to take advantage of the Databricks Lakehouse platform.

LiveData Migrator automates cloud data migration at any scale by enabling companies to migrate data from on-premises Hadoop-oriented data lakes to any cloud within minutes, even while the source data sets are under active change. 

Businesses can migrate their data without the need of engineers or other consultants to enable their digital transformation. LiveData says its Migrator works without any production downtime or business disruption while ensuring the migration is complete and continuous, and any ongoing data changes are copied to the target cloud environment.

Users can choose to convert content to the Delta Lake format when they create the Databricks metadata target. The data they choose to migrate is then set by defining a migration rule, and then selecting the Hive databases and tables that require migration.

Related stories
Top stories
Story image
Amazon Web Services / AWS
Zscaler, AWS accelerate onramp to the cloud with zero trust
Zscaler has announced an extension to its relationship with Amazon Web Services, as well as innovations built on Zscaler's Zero Trust architecture.
Story image
Cybersecurity
Aqua Security, CIS create software supply chain security guide
Aqua Securityand the Center for Internet Security have together released the industry’s first formal guidelines for software supply chain security.
Story image
Cybersecurity
Threat actors ramp up their social engineering attacks
As people get better at identifying potential threats in their inbox, threat actors must evolve their methods. Their new M.O? Social engineering.
Story image
Commerce Commission
ComCom puts electronics sector on notice over resale price maintenance
The Commerce Commission has concluded an investigation into allegations that television manufacturers were engaging in illegal resale price maintenance.
Story image
DNS
DigiCert acquires DNS Made Easy and affiliated brands
Greg Clark comments, says, "This combination enhances the security of certificate validation and enables the automation of future validations."
Story image
Tech job moves
Tech job moves - Boomi, Limepay, Thales, VMware & Zoom
We round up all job appointments from June 6-16, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Apple
Jamf updates healthcare IT to protect data on Apple devices
Jamf has rolled out new functionality to help healthcare and IT teams protect patient data and streamline clinical access for their Apple fleet.
Story image
Sustainability
Schneider Electric ups the ante on sustainability strategy
"We've made some progress but to avoid a major energy challenge, all data centres - including distributed edge data centres - must be more sustainable."
Story image
Robotic Process Automation / RPA
rapidMATION helps Coates achieve success with landmark RPA solution
A strong Robotic Process Automation solution (RPA) can help solve many complex issues that businesses face daily. 
Story image
Government
Cyclone selected as NZ MOE software licensing partner
Following a recent Request for Proposal (RFP), Christchurch-based company Cyclone Computer Company Ltd (Cyclone) has been selected as The Ministry of Education’s software licensing partner.
Story image
Artificial Intelligence
Salesforce harnesses automated solutions with new developments
Salesforce has launched Sales Cloud Unlimited, a new feature to help accelerate productivity with AI and automation.
Story image
Microsoft
Volpara, Microsoft project to detect cardiovascular issues
Volpara Health Technologies is working with Microsoft on a research and development project to speed up creating a product that detects and quantifies breast arterial calcifications (BACs).
Story image
Digital Transformation
Apptio adds portfolio enhancements to promote digital strategy
"While digitalisation creates opportunities, it also makes budgeting far more complex, leading many companies to waste substantial funds."
Story image
API
Industry-first comprehensive risk-based API security enhances protection
Application Programming Interfaces (APIs) have become a crucial part of operating web and mobile application businesses and are causing significant economic growth in the digital sector.
Story image
Ransomware
Rapid7 report examines use of double extortion ransomware attacks
New insight into how attackers think when carrying out cyber attacks, along with further analysis of the disclosure layer of double extortion ransomware attacks, has come to light.
Story image
F5 Networks
Telstra, F5 team up to bolster services and solutions
“This partnership demonstrates our ongoing investment into APAC as we continue delivering high value services and solutions to our partners and customers."
Story image
Network Security
Netskope announces zero trust network access updates
Customers can now apply zero trust principles across a range of hybrid work security needs, including SaaS, IaaS, private applications, and endpoint devices.
Story image
Infrastructure
VMware wins Google Cloud partner award for infrastructure modernisation
The cloud computing and virtualisation company was recognised for its achievements as part of the Google Cloud ecosystem.
Story image
SaaS
Varonis strengthens security capabilities for AWS and S3
Varonis has strengthened and expanded its cloud and security capabilities, with a critical aim of improving safety and boosting data visibility in Amazon Simple Storage Service (S3).
Story image
B2B
Corpay partners with supply chain platform PracBiz Exchange
Corpay's new partnership with PracBiz’s allows more than 4000 B2B suppliers on the latter's platform to use Corpay's global payments services.
Story image
Cloud
Cloudflare outage in 19 data centers worldwide due to own error
Cloudflare says its outage for 19 of its data centers yesterday was because of a change in a long-running project to increase resilience in its busiest locations.
Story image
Healthcare
Workday winning on culture and family focus
This family-first approach sees all employees receive access to family-wide private healthcare cover, as well as income protection and life insurance policies.
Story image
CRM
Microsoft launches app for modern selling experience
The new release is designed to enhance CRM systems with customer engagement data from Microsoft 365 and Microsoft Teams.
Story image
Orbital Insight
Orbital Insight solution set to drive better data-driven decisions
The company says the new Site Intelligence solution will provide granular visibility, behaviour analytics and deep insights about customers and competitors at any location. 
Story image
Dark web
Cybercrime in Aotearoa: How does New Zealand law define it?
‘Cybercrime’ is a term we hear all the time, but what exactly is it, and how does New Zealand define it in legal terms?
Story image
Citrix
The best ways to attract young talent during labour shortages
New research from Citrix reveals hybrid working and ventures into the metaverse are top of mind for Gen Z workers.
Story image
Cybersecurity
Greater API usage raises concerns for protection - report
Radware has released its 2022 State of API Security report, which shows a rise in APIs, with 92% of the organisations surveyed significantly or somewhat increasing their usage.
Story image
Collaboration
IT and security team collaboration crucial to data security
Many IT and security decision makers are not collaborating as effectively as possible to address growing cyber threats.
Story image
Entelar
How TruSens air purifiers can create healthier workspaces
The pandemic has heightened our awareness of our own and others’ health, and made us all much more conscious of the environments we work in.
Story image
Infrastructure
Lenovo announces launch of Retail Solutions portfolio in A/NZ
Lenovo has announced the launch of its Lenovo Retail Solutions portfolio across the A/NZ region, which they say will help retailers transform their business and face future retail demands.
Story image
Contact Centre
Customer service agents don't want to return to contact centres
A new report has revealed that 85% of customer service agents want to work full-time at home and not return to contact centre offices.
Story image
Shouta
The rise of digital gifting in the workplace
The name itself does most of the explaining; it’s a gift you receive virtually. But a misconception about digital gifts is that they need to be redeemed virtually as well. 
Story image
Data
How a single mandate changed software development forever
There’s conjecture about exactly when it was issued and by whom, but a mandate made twenty years ago is continuing to shape the software development process today.
Story image
PagerDuty
Ready for anything with the PagerDuty Operations Cloud
In a world of digital everything, teams face increasing complexity. Ever-growing dependencies across systems and processes put customer and employee experience, not to mention revenue, at risk.
Story image
Electricity
Canstar finds Flick Electric NZ’s favourite provider
Canstar’s annual research to find New Zealand’s favourite electricity provider reveals Flick Electric has come out on top.
Story image
eInvoicing
Airwallex, Xero extend partnership with easier invoice payments
Airwallex has extended its long-term partnership with Xero by releasing a new payment link integration for Xero invoices that will make receiving them easier and faster for Australian businesses.
Story image
Disaster Recovery
Eaton provides business continuity with disaster avoidance application
Many SMBs often struggle with problems relating to UPS and IT assets due to the sheer amount of time and resources required to run effectively. 
Story image
Hybrid workforce
Why hybrid working is here to stay and how to ace it
Citrix's new report reveals hybrid workers are more productive and engaged at work than their office and completely remote counterparts.
Story image
10 Minute IT Jams
Video: 10 Minute IT Jams - An update from Rimini Street
Today we welcome back Daniel Benad, who is the GVP & regional GM for Oceania at Rimini Street.
Story image
trust
Consumers want personalisation, but don't trust brands with their data
Customers expect personalisation during every brand interaction but they don't trust brands to keep their personal data secure and to use it responsibly. 
Story image
Cybersecurity
Why is NZ lagging behind the world in cybersecurity?
A recent report by TUANZ has revealed that we are ranked 56th in the world when it comes to cybersecurity - a look into why we're so behind and what needs to be done.
Story image
Market growth
Salesforce unveils new offerings for consumer goods companies
Salesforce has announced new products for consumer goods companies to help brands navigate increasing market complexity more easily.
Story image
Open source
DataStax secures US$115 million to fund database expansion
DataStax has secured US$115 million in funding, which it will use to develop and expand its Astra DB multi-cloud database and Astra Streaming service globally.
Story image
Manufacturing
Sternum joins NXP, collaborates on IoT security and observability
Sternum has announced it has joined the software partner community of NXP Semiconductors, a manufacturer of and large marketplace for embedded controllers.