IT Brief New Zealand logo
Technology news for New Zealand's largest enterprises
Story image

Databricks launches new machine learning platform

By Ryan Morris-Reade
Fri 28 May 2021
FYI, this story is more than a year old

Databricks has announced the latest generation of its machine learning offering, with the launch of Databricks Machine learning, a new data-native platform built on top of an open lakehouse architecture. 

According to the company, the new platform enables new and existing ML capabilities on the lakehouse platform to be integrated into a collaborative, purpose-built experience that provides ML engineers with everything they need to build, train, deploy, and manage ML models from experimentation to production, combining data, and the full ML lifecycle. 

The platform also includes two new capabilities: 

  • Databricks AutoML to augment the machine learning process by automating all the steps that data scientists have to manually do, while still exposing enough control and transparency. 
  • Databricks Feature Store to improve discoverability, reuse, and governance of model features in a system integrated in the enterprise's data engineering platform.

"Many ML platforms fall short because they ignore key challenges in machine learning; they assume that data is available at high quality and ready for training," Databricks says. 

"That requires data teams to stitch together solutions that are good at data but not AI, with others that are good at AI but not data. 

"To complicate things further, the people responsible for data platforms and pipelines (data engineers) are different from those that train ML models (data scientists), which are different from those who deploy product applications (engineering teams who own business applications)," the company explains.

"As a result, solutions for ML need to bridge gaps between data and AI, the tooling required, and the people involved."

Built on an open lakehouse foundation, the platform ensures customers can easily work with any type of data, at any scale, from machine learning across traditional structured tables, to unstructured data like videos and images, to streaming data from real-time applications and IoT sensors, and quickly move through the ML workflow to get more models to production faster.

AutoML has the potential to allow data teams to more quickly build ML models by automating a lot of heavy lifting involved in the experimentation and training phases. 

But, enterprises who use AutoML tools today often struggle with getting AutoML models to production. This happens because the tools provide no visibility into how they arrive at their final model, which makes it impossible to modify its performance or troubleshoot it when edge cases in data lead to low confidence predictions. 

According to Databricks, the introduction of the AutoML capabilities within the platform takes a unique 'glass box' approach instead. It allows data teams to not only quickly produce trained models either through a UI or API, but also auto-generates underlying experiments and notebooks with code, so data scientists can easily validate an unfamiliar data set or modify the generated ML project. 

Data scientists have full transparency into how a model operates and can take control at any time. This transparency is critical in highly regulated environments and for collaboration with expert data scientists.

Machine learning models are built using features, which are the attributes used by a model to make a prediction. To work most efficiently, data scientists need to be able to discover what features exist within their organisation, how they are built, and where they are used, rather than wasting significant time repeatedly reinventing features. 

Additionally, feature code needs to be kept consistent across several teams that participate in the ML workflow, otherwise, model performance will drift apart between real-time and batch use cases - a problem called online/offline skew.

According to Databricks, its Feature Store is the first of its kind that is co-designed with a data and MLOps platform. It says tight integration with the popular open source frameworks Delta Lake and MLflow guarantees that data stored in the Feature Store is open and that models trained with any ML framework can benefit from the integration of the Feature Store with the MLflow model format. 

According to the company, the Feature Store eliminates online/offline skew by packaging feature store references with the model, so that the model itself can look up features from the Feature Store instead of requiring a client application to do so. As a result, features can be updated without any changes to the client application that sends requests to the model. 

Databricks say the Feature Store knows exactly which models and endpoints consume any given feature, facilitating end-to-end lineage as well as safe decision-making on whether a feature can be updated or deleted.


 

Related stories
Top stories
Story image
Amazon Web Services / AWS
Zscaler, AWS accelerate onramp to the cloud with zero trust
Zscaler has announced an extension to its relationship with Amazon Web Services, as well as innovations built on Zscaler's Zero Trust architecture.
Story image
Cybersecurity
Aqua Security, CIS create software supply chain security guide
Aqua Securityand the Center for Internet Security have together released the industry’s first formal guidelines for software supply chain security.
Story image
Cybersecurity
Threat actors ramp up their social engineering attacks
As people get better at identifying potential threats in their inbox, threat actors must evolve their methods. Their new M.O? Social engineering.
Story image
Commerce Commission
ComCom puts electronics sector on notice over resale price maintenance
The Commerce Commission has concluded an investigation into allegations that television manufacturers were engaging in illegal resale price maintenance.
Story image
Cybersecurity
Significant security concerns resulting from open source software ubiquity
"The risk is real, and the industry must work closely together in order to move away from poor open source or software supply chain security practices."
Story image
DNS
DigiCert acquires DNS Made Easy and affiliated brands
Greg Clark comments, says, "This combination enhances the security of certificate validation and enables the automation of future validations."
Story image
10 Minute IT Jams
Video: 10 Minute IT Jams - An update from Rimini Street
Today we welcome back Daniel Benad, who is the GVP & regional GM for Oceania at Rimini Street.
Story image
Manufacturing
Sternum joins NXP, collaborates on IoT security and observability
Sternum has announced it has joined the software partner community of NXP Semiconductors, a manufacturer of and large marketplace for embedded controllers.
Story image
F5 Networks
Telstra, F5 team up to bolster services and solutions
“This partnership demonstrates our ongoing investment into APAC as we continue delivering high value services and solutions to our partners and customers."
Story image
Testing
Video: 10 Minute IT Jams - An update from Tricentis
Tricentis provides software testing automation, and software quality assurance products for enterprise software.
Story image
Secure access service edge / SASE
Cloudflare adds new capabilities to zero trust SASE platform
New features for Cloudflare One include email security protection, data loss prevention tools, cloud access security broker, and private network discovery.
Story image
Open source
DataStax secures US$115 million to fund database expansion
DataStax has secured US$115 million in funding, which it will use to develop and expand its Astra DB multi-cloud database and Astra Streaming service globally.
Story image
Network Security
Netskope announces zero trust network access updates
Customers can now apply zero trust principles across a range of hybrid work security needs, including SaaS, IaaS, private applications, and endpoint devices.
Story image
Contact Centre
Customer service agents don't want to return to contact centres
A new report has revealed that 85% of customer service agents want to work full-time at home and not return to contact centre offices.
Story image
Dark web
Cybercrime in Aotearoa: How does New Zealand law define it?
‘Cybercrime’ is a term we hear all the time, but what exactly is it, and how does New Zealand define it in legal terms?
Story image
Artificial Intelligence
Salesforce harnesses automated solutions with new developments
Salesforce has launched Sales Cloud Unlimited, a new feature to help accelerate productivity with AI and automation.
Story image
Microsoft
Volpara, Microsoft project to detect cardiovascular issues
Volpara Health Technologies is working with Microsoft on a research and development project to speed up creating a product that detects and quantifies breast arterial calcifications (BACs).
Story image
Infrastructure
VMware wins Google Cloud partner award for infrastructure modernisation
The cloud computing and virtualisation company was recognised for its achievements as part of the Google Cloud ecosystem.
Story image
API
Industry-first comprehensive risk-based API security enhances protection
Application Programming Interfaces (APIs) have become a crucial part of operating web and mobile application businesses and are causing significant economic growth in the digital sector.
Story image
trust
Consumers want personalisation, but don't trust brands with their data
Customers expect personalisation during every brand interaction but they don't trust brands to keep their personal data secure and to use it responsibly. 
Story image
Orbital Insight
Orbital Insight solution set to drive better data-driven decisions
The company says the new Site Intelligence solution will provide granular visibility, behaviour analytics and deep insights about customers and competitors at any location. 
Story image
PagerDuty
Ready for anything with the PagerDuty Operations Cloud
In a world of digital everything, teams face increasing complexity. Ever-growing dependencies across systems and processes put customer and employee experience, not to mention revenue, at risk.
Story image
Entelar
How TruSens air purifiers can create healthier workspaces
The pandemic has heightened our awareness of our own and others’ health, and made us all much more conscious of the environments we work in.
Story image
SaaS
Sealord partners with Infor to improve sustainability
Sealord has chosen Infor as a strategic partner to implement an operational cloud-based platform that provides day-one functionality and sustainability gains.
Story image
Digital Transformation
Apptio adds portfolio enhancements to promote digital strategy
"While digitalisation creates opportunities, it also makes budgeting far more complex, leading many companies to waste substantial funds."
Story image
Market growth
Salesforce unveils new offerings for consumer goods companies
Salesforce has announced new products for consumer goods companies to help brands navigate increasing market complexity more easily.
Story image
Electricity
Canstar finds Flick Electric NZ’s favourite provider
Canstar’s annual research to find New Zealand’s favourite electricity provider reveals Flick Electric has come out on top.
Story image
Disaster Recovery
Eaton provides business continuity with disaster avoidance application
Many SMBs often struggle with problems relating to UPS and IT assets due to the sheer amount of time and resources required to run effectively. 
Story image
Collaboration
IT and security team collaboration crucial to data security
Many IT and security decision makers are not collaborating as effectively as possible to address growing cyber threats.
Story image
eInvoicing
Airwallex, Xero extend partnership with easier invoice payments
Airwallex has extended its long-term partnership with Xero by releasing a new payment link integration for Xero invoices that will make receiving them easier and faster for Australian businesses.
Story image
SaaS
Varonis strengthens security capabilities for AWS and S3
Varonis has strengthened and expanded its cloud and security capabilities, with a critical aim of improving safety and boosting data visibility in Amazon Simple Storage Service (S3).
Story image
Digital Transformation
Cybersecurity priorities for digital leaders navigating digital transformation
In recent years, Asia-Pacific has especially been a hotspot for cyberattacks, and as we continue into 2022, it’s evident that the problem is becoming more significant.
Story image
Healthcare
Workday winning on culture and family focus
This family-first approach sees all employees receive access to family-wide private healthcare cover, as well as income protection and life insurance policies.
Story image
Cybersecurity
Why is NZ lagging behind the world in cybersecurity?
A recent report by TUANZ has revealed that we are ranked 56th in the world when it comes to cybersecurity - a look into why we're so behind and what needs to be done.
Story image
Partnerships
Microsoft expands APAC Enabler Mentorship Program
"Mentors are the key to success for every professional. A good mentor is a coach, a guide, as well as a vocal advocate."
Story image
Citrix
The best ways to attract young talent during labour shortages
New research from Citrix reveals hybrid working and ventures into the metaverse are top of mind for Gen Z workers.
Story image
Infrastructure
Lenovo announces launch of Retail Solutions portfolio in A/NZ
Lenovo has announced the launch of its Lenovo Retail Solutions portfolio across the A/NZ region, which they say will help retailers transform their business and face future retail demands.
Story image
Government
Cyclone selected as NZ MOE software licensing partner
Following a recent Request for Proposal (RFP), Christchurch-based company Cyclone Computer Company Ltd (Cyclone) has been selected as The Ministry of Education’s software licensing partner.
Story image
Apple
Jamf updates healthcare IT to protect data on Apple devices
Jamf has rolled out new functionality to help healthcare and IT teams protect patient data and streamline clinical access for their Apple fleet.
Story image
Robotic Process Automation / RPA
rapidMATION helps Coates achieve success with landmark RPA solution
A strong Robotic Process Automation solution (RPA) can help solve many complex issues that businesses face daily. 
Story image
Cloud
Cloudflare outage in 19 data centers worldwide due to own error
Cloudflare says its outage for 19 of its data centers yesterday was because of a change in a long-running project to increase resilience in its busiest locations.
Story image
Shouta
The rise of digital gifting in the workplace
The name itself does most of the explaining; it’s a gift you receive virtually. But a misconception about digital gifts is that they need to be redeemed virtually as well. 
Story image
Training
Employers look to hire inexperienced coders due to skills shortage
"Even inexperienced workers without prior qualifications or experience had managed to pivot to new roles in coding as long as they are willing to upskill."
Story image
Hybrid workforce
Why hybrid working is here to stay and how to ace it
Citrix's new report reveals hybrid workers are more productive and engaged at work than their office and completely remote counterparts.