itb-nz logo
Story image

Google Cloud rolls out Cloud Dataproc on Kubernetes

11 Sep 2019

Google Cloud is trialling alpha availability of a new platform for data scientists and engineers through Kubernetes.

Cloud Dataproc on Kubernetes combines open source, machine learning and cloud to help modernise big data resource management.

The alpha availability will first start with workloads on Apache Spark, with more environments to come.

According to Google Cloud product managers Christopher Crosbie and James Malone, Google Cloud Dataproc can provide open source data analytic processing for those who need to process data and train models at scale, faster.

However, as enterprise infrastructure becomes increasingly hybrid in nature, machines can sit idle, single workload clusters continue to sprawl, and open source software and libraries continue to become outdated and incompatible with your stack,” they explain.

“It’s critical that Cloud Dataproc continues to empower data professionals to focus more on workloads than infrastructure by combining the best of cloud and open source.”

The platform will include key benefits such as faster workloads, unified resource management, job isolation, collaboration, and expertise sharing.

Unified resource management will allow data scientists to work with a central view that spans both Kubernetes and YARN cluster management systems.

“Kubernetes has flipped the big data and machine learning open source software (OSS) world on its head, since it gives data scientists and data engineers a way to unify resource management, isolate jobs, and build resilient infrastructures across any environment.”

More resilient infrastructure: A self-healing GKE environment can support the smooth operation of mission critical ETL and machine learning jobs on Spark.

“Data scientists and data engineers don’t have to worry about sizing and building clusters, manipulating Docker files, or messing around with Kubernetes networking configurations. It just works. With leading support from the team that built Kubernetes, enterprises have access to the skills they need to close any Kubernetes skills gap on their team.”

Less time and resource on infrastructure, more on workloads – the development of new applications and models faster at scale

Isolate jobs to accelerate analytics life cycles – users can package up entire jobs in standalone containers to allow for testing, upgrading and patching without breaking underlying cluster.

Collaboration and expertise sharing to close the Kubernetes skills gap – new capabilities, bugs and security issues can be discussed and resolved by open source community

This is the first step in a larger journey to a container-first world. While Apache Spark is the first open source processing engine we will bring to Cloud Dataproc on Kubernetes, it won’t be the last,” comment Crosbie and Malone.

They add that Google Cloud’s data and analytics strategy has always involved open source as a core pillar.

“This alpha announcement of bringing enterprise-grade support, management, and security to Apache Spark jobs on Kubernetes is the first of many as we aim to simplify infrastructure complexities for data scientists and data engineers around the world.”

Story image
Vodafone NZ completes international optical network upgrade
The project enables a dedicated, resilient and diverse set of internet pathways upon which international data can be transferred at millisecond speed.More
Story image
New Relic recruits former Qlik, Delphix execs to boost A/NZ team
"We are absolutely thrilled to have two very seasoned executives join us at New Relic as we help customers chart and execute on their plans for digital delivery and cloud services."More
Link image
<span class="coloured">Unleash the intelligent way to automate at Pega Discover – Intelligent Automation</span>
Find out how the world’s largest brands are accelerating business and simplifying systems in this two-hour, interactive virtual event. By the end you’ll be primed to start getting business done smarter and faster while scaling toward your biggest business goals. Register Now.More
Story image
IBM, Alphabet and well-funded startups in the race for quantum supremacy
"It may not come as a surprise that quantum computing one day replaces artificial intelligence as the mainstream technology to help industries tackle problems they never would have attempted to solve before.”More
Story image
DevOps & IT teams feel pressure to keep digital services online during pandemic
Many DevOps and IT teams spend more than 10 extra hours per week dealing with a 47% rise in incidents.More
Link image
Why video-streaming companies should consider a multi-CDN strategy
Video streaming continues to grow each year, and in order to ensure quality of experience, new strategies must be leveraged.More