Organisations' struggle to achieve data-driven growth

Only 13% of organisations excel at delivering on their data strategy, according to a new report.

The report, "Building a high-performance data and AI organisation," was released by MIT Technology Review Insights and produced in association with Databricks.

It focuses on a survey of 351 senior data officers, and in-depth interviews with data and analytics leaders at global organisations that earn $1 billion or more annual revenue across 14 sectors, which include: Total, The Este Lauder Companies, McDonald's, L'Oral, CVS Health and Northwestern Mutual.

The report highlights how data management and supporting technologies are evolving, and looks at some of the current challenges that hold organisations back from achieving data-driven growth. Some of these challenges include poor data management, largely a lack of centralised data platforms for analytics and machine learning, and inadequate collaboration among data teams. This leads to a low return on investments for data analytics and business intelligence initiatives.

According to the report, only 13% of organisations excel at delivering on their data strategy. This group of high-achievers deliver measurable business results across the enterprise. Their success could be due to a sound foundation of data management and architecture, which enables them to democratise data and derive value from machine learning.

The chief data officers interviewed for the study ascribe great importance to democratising analytics and machine learning capabilities. The impact on business by machine learning is limited by difficulties in managing its end-to-end lifecycle. Scaling machine learning use cases is exceedingly complex for many organisations. The most significant challenge, according to 55% of respondents, is the lack of a central location for storing and discovering new machine learning models.

The report finds that technology-enabled collaboration is creating a working data culture and that pushing these to the edge with advanced data technologies will help end-users to make more informed business decisions which are hallmarks of a strong data culture.

"Managing data is highly complex and can be a real challenge for organisations. But creating the right architecture is the first step in a huge business transformation," says the editor of the report, Francesca Fanshawe.

"There are many models an enterprise can adopt, but ultimately the aim should be to create a data architecture that's simple, flexible, and well-governed.

It finds that organisations' top data priorities over the next two years fall into three areas, all of which support the wider adoption of cloud platforms: improving data management, enhancing data analytics and machine learning, and expanding the use of all types of enterprise data, including streaming and unstructured data.

"The past year has been an accelerant of change as data-driven organisations look to adapt, innovate, and future proof their technology and architecture investments," says global principal technologist at Databricks, Chris D'Agostino.

"Now more than ever, enterprises need a modern data analytics strategy that is open, flexible, and empowers everyone across the organisation to make faster, more informed decisions with a unified view of all their data, whether that's using machine learning and AI algorithms or straightforward SQL and BI reporting

Key findings in the report include:

• Just 13% of the large organisations in the study excel at delivering on their data strategy, despite the explosion of data that most industries have experienced recently.
• 43% of respondents said increasing the adoption of cloud platforms is one of the most important enterprise wide-data strategy initiatives over the next two years.
• Only 12% of respondents said they've achieved optimal price/performance for their analytics workloads, meaning organisations have yet to see the return on investment for their analytics and BI initiatives.
• The pandemic has accelerated digital transformation, which includes shifts in IT infrastructure strategies, with 50% of executives citing they are currently evaluating or implementing a new data platform to address their current data challenges.
• Open standards are a top requirement of future data architecture strategies. If respondents could build a new data architecture for their business, the most critical advantage over the existing architecture would be a greater embrace of open source standards and open data formats.