Ignite 2018 - Making AI real for your business with Azure Data

At Microsoft Ignite in Orlando, I shared how the confluence of Cloud, Data, and Artificial Intelligence (AI) is driving unprecedented change and is rapidly becoming foundational to innovation in every industry.

The driving force behind powerful AI applications is data – and getting the most out of AI requires a modern data estate. Unlocking the potential of data has become an imperative for organisations that want to leverage AI to uncover new insights and create new business opportunities.

Today, we shared a number of exciting announcements that will enable organisations to build data and AI solutions that transform their businesses.

For over two decades, organisations have relied on SQL Server to manage all facets of their relational data. Last year at Ignite, we announced the general availability of SQL Server 2017, which brought the much beloved SQL Server engine to Linux. With over 5.6 million downloads in just the first 6 months, it rapidly became our most popular SQL Server version yet.

Today, we are building on that momentum with the announcement of the SQL Server 2019 preview. With SQL Server 2019, organisations can now seamlessly manage their relational and non-relational data in a single, integrated solution.

It comes with big data capabilities built-in, including support for Apache SparkTM and Hadoop Distributed File System (HDFS). And with SQL Server Machine learning Services and Spark Machine learning (ML), organisations can get AI-driven insights from their entire data estate, regardless of data type.

We also enhanced PolyBase with additional connectors to data sources, such as Azure SQL Data Warehouse, Azure Cosmos DB, Mongo DB, Oracle, and Teradata, further helping organisations break down their data silos. With this new level of data virtualisation, organisations can now apply the same security and access control policies and run T-SQL queries on all their data – regardless of its source.

SQL Server 2019 also runs anywhere in Kubernetes – on-premises, on Azure Stack, or in the cloud. With the preview of Azure Data Studio's tooling for SQL Server 2019, organisations can now not only manage their big data clusters, they can also take advantage of a new analytics notebook experience and a PolyBase wizard for fast and easy access to data stored in other database management systems.

We are excited to see customers, like Johns Hopkins University, already running demanding big data workloads on it to unlock analytics and AI.

"From its inception the Sloan Digital Sky Survey database has run on SQL Server, and SQL Server also stores object catalogs from large cosmological simulations. We are delighted with the promise of SQL Server 2019 big data clusters, which will allow us to enhance our databases to include all our big data sets.

The distributed nature of SQL Server 2019 allows us to expand our efforts to new types of simulations and to the next generation of astronomical surveys with datasets up to 10 PB or more, well beyond the limits of our current database solutions," - Dr. Gerard Lemson, Institute for Data Intensive Engineering and Science, Johns Hopkins University

We also made a number of exciting announcements that make Azure the best destination for an organisation's data. For SQL workloads, Azure SQL Database Managed Instance offers an easy path to the cloud and incredible value. Generally available today, Azure SQL Database Managed Instance enables organisations to migrate their SQL Server workloads to Azure with zero code changes. It offers native VNET support and is compatible with the full SQL Server feature set.

Thanks to offers like Azure Hybrid Benefit and SQL Database reserved capacity, organisations that migrate to Azure can save up to 80 percent compared to other cloud offerings. In addition, organisations can now use the fully managed Azure Database Migration Service to migrate at scale from SQL Server to SQL Database Managed Instance. It is inspiring to see the strong demand for Azure SQL Database Managed Instance and the value it is already delivering to organisations.

"We were able to deploy our TimeXtender solution into production on Azure SQL Database Managed Instance in a matter of weeks. We immediately realised a 49% cost savings and a 25-30 percent performance improvement, and the promise of applying artificial intelligence through machine learning to our data is an exciting opportunity for us," - John Steele, GM of Business Technology - Systems, Komatsu Australia.

Azure SQL Database Managed Instance is also catching the attention of analysts. Forrester recently interviewed several customers who have migrated their SQL databases onto a managed instance and found that on average they saw a migration payback period of less than 6 months and a 40 percent improvement in DBA productivity.

We also announced today the preview of Azure SQL Database Hyperscale, a highly scalable service tier that adapts on-demand to workload needs. With Azure SQL Database Hyperscale, organisations can grow a single database up to 100 TB, significantly expanding the potential for application growth. Innovations like these make Azure SQL Database the best relational database in the cloud.

As a further testament to Azure's commitment to openness, we also made a number of exciting announcements for our open source databases. Azure Database for MariaDB is now in preview, bringing a fully managed, open source MariaDB database-as-a-service to the Azure platform. MariaDB on Azure offers developers built-in high availability, dynamic scaling, flexible pricing, and world-class security and infrastructure.

We also announced the preview of Advanced Threat Protection for Azure database services for MySQL and PostgreSQL, enabling organisations to detect and respond to potential threats as they occur. Azure Database for PostgreSQL now provides preview support for Query Store, Query Performance Insights, and Performance Recommendations as part of its Intelligent Performance offering. Using this combination of features, organisations can inspect their databases to understand their workloads, identify bottlenecks, and detect changes in query performance.

Azure Cosmos DB is Microsoft's globally distributed, multi-model database service for NoSQL workloads. Azure Cosmos DB provides turnkey global distribution, elastic scaling, single-digit millisecond latency, five well-defined consistency models, and guaranteed high availability. Azure Cosmos DB automatically indexes all data without requiring developers to deal with schema or index management. It is a multi-model service, which natively supports document, key-value, graph, and column-family data models.

Today, we are happy to announce the general availability of multi-master at global scale in Azure Cosmos DB. This enables unprecedented write scalability and availability with multi-master replication. With Azure Cosmos DB's multi-master replication every region becomes writable (in addition to being readable) which enables single digit millisecond write latency and 99.999% write availability around the world. To learn more, please read our latest SLA page.

We are also excited to announce general availability of Azure Cosmos DB Cassandra API. By providing wire protocol level compatibility with the Apache Cassandra, Azure Cosmos DB ensures you can continue using your existing application and favorite open source software (OSS) tools with virtually no code changes and gives you the flexibility to run your Cassandra apps fully managed with no vendor lock-in. It is exciting to see the amazing momentum for Azure Cosmos DB in the market and to see how customers are adopting it to drive their business forward.

"Cosmos DB has delivered a high performance, worldwide platform for our global reputation system. With a click of a button we can rapidly deploy regions worldwide to deliver excellent response time to our customers and partners. The Cosmos DB Cassandra API eliminated major engineering efforts during the migration of our services as we were able to integrate with very minimal changes. Our largest service is slated to migrate with the launch of Cassandra API support," - Michael Shavell, Technical Director / Architect at Symantec

Azure Cosmos DB not only delivers industry-leading performance, but it also offers superior value. Organisations that migrate their existing NoSQL applications to Azure Cosmos DB can save up to six times compared to enterprise versions of OSS NoSQL databases. Today, we are happy to announce general availability of Azure Cosmos DB Reserved Capacity, enabling organisations to save up to 65 percent off the list price of Azure Cosmos DB.

In addition, today we announced that a database can be provisioned for as little as 10,000 RU/s (lowering the entry point by 5x) and can grow incrementally to fit an organisation's needs. We are also happy to announce that we are extending the length of the popular Try Cosmos DB for Free trial to 1 month. In total, Azure Cosmos DB's capabilities and value make it the best destination for all of your NoSQL workloads.

When it comes to analytics and AI, Azure provides the most compelling set of services to help organisations quickly gain important insights to their businesses. Azure Data Factory enables organisations to effortlessly ingest data at multi-TB scale into Azure from over 70 data sources, including AWS S3, AWS Redshift, Google Big Query, Oracle, Teradata, Netezza and SAP HANA for deeper analytics.

Azure Databricks provides a fast, easy and collaborative Apache Spark-based analytics platform for data preparation and machine learning (and deep learning) model development and training. It offers the best of OSS analytics frameworks and tight integration with the Azure ecosystem. Azure SQL Data Warehouse is our enterprise-class, fully integrated data warehouse and is the first of its kind to offer independent scaling for compute and storage. These three services taken together form our 'modern data warehouse' and 'machine learning on big data' patterns and offer organisations incredible value.

"Our modern data warehouse gives us increased agility which was missing from our datacenter. By focusing less on the operational complexity and more on the business logic, we are increasing our agility and reaping business value faster," - Chetan Kundavaram, Global Director, Anheuser Busch Inbev

Today, we announced that Azure SQL Data Warehouse will now be accessible to more customers through a smaller service level (DW500c). With this release, organisations can get started with a powerful cloud data warehouse for as low as $6.049/hour. This performance-oriented tier is now available across 26 Azure global regions, including Azure Government. Azure SQL Data Warehouse now has even better integration with Azure Databricks allowing organisations to easily stream, train, and publish streaming data to build real-time analytics solutions.To learn more, please check out our Azure SQL Data Warehouse announcement blog.

Also new is the preview of Azure Databricks Delta. Azure Databricks Delta enables organisations to increase data reliability, improve performance, and simplify their data pipelines. Building high performance analytics solutions at scale using Azure Databricks is now simpler than ever for organisations. With new metadata, indexing and caching capabilities over Spark tables, organisations can get better reliability and performance on Spark jobs and queries in Azure Databricks.

As part of our continued commitment to supporting open source analytics, I also announced today that HDInsight will now support Hadoop 3.0—making Azure the first cloud provider to offer this capability. HDInsight also offers enhanced security with the general availability of Enterprise Security Package, which enables organisations to join their HDInsight clusters to a domain, authenticate users with Kerberos, and authorize and audit their access using Apache Ranger. Developers will also find it easy to debug and diagnose large jobs using playback, automatic data skew detection, and other innovative enhancements. To learn more, please check out our HDInsight announcement blog.

We also announced today the preview of Azure Data Explorer—a lightning fast service optimised for data exploration. The service helps organisations quickly discover insights from petabytes of structured and unstructured data. Azure Data Explorer can be used to query streaming data to drive instant insights for analyzing performance, identifying trends and anomalies, and diagnosing problems.

Data engineers can also use it to explore data in the early stages of data analytics to better understand what's in their data. Azure Data Explorer enhances the modern data warehouse by providing instant visibility and analytics over data spanning minutes ago to weeks or months. To learn more, please check out our Azure Data Explorer announcement blog.

Finally, I'd like to close with an example of a truly transformational customer solution. Shell, a global oil and gas company, has been on a journey to leverage data and AI to digitally transform their business. One amazing example of this is how they are using the power of their data to improve the safety and effectiveness of their retail stores. With Azure Databricks, Shell can train algorithms to analyse thousands of hours of closed-circuit television (CCTV) footage to identify potential safety concerns in near real-time.

"We have developed a collection of machine learning models that operate both at the edge and in the cloud. We can identify events of interest at the edge in real-time and pass those to the cloud for tight processing and we can use the cloud for near real-time alerting. The Azure cloud is powering this transformation, bringing together the best of platform-as-a-service offerings, like Azure Databricks, and combine them with open source technologies like TensorFlow and Kafka," - Daniel Jeavons, GM of Data Science, Shell.