IT Brief New Zealand logo
Technology news for New Zealand's largest enterprises
Story image

To work like a disrupter, change the way you work with data and analytics

By Contributor
Fri 15 May 2020
FYI, this story is more than a year old

Article by SAS vice president of product strategy, Shadi Shahin.

What do companies like Netflix, Airbnb, Spotify and Lyft know that you don’t know? What do these companies have that you might be lacking?

Sure, they had an initial idea. But we’ve all had good ideas. How did each of these new market heroes move from the seed of an idea to the disruption of an entire industry?

For each of them, it took a combination of strong technological and analytical skills to develop a digital platform that turned data into a fundamental asset for the company.

Now you might be thinking: But we do analytics, too. We have a smart team of data scientists. We know the latest techniques for machine learning and deep learning, and we even attend the best analytics conferences.

But there’s a difference between analytical maturity and technological maturity. Disrupters are both analytically and technologically mature.

They don’t just understand how to build models and use algorithms. They also know how to design systems that incorporate algorithmic thinking into their core. They know how to replicate and tweak models by the thousands, how to deploy models at scale and how to test, iterate, improve and retire models systematically.

Compare your analytic maturity to peers in your industry, and determine where you have room to grow. Many traditional companies are analytically mature but not technologically mature.

If you’ve done a great job of capturing and analysing data but come up short on moving those results into action on a large scale, you’re in luck.

We’re seeing three clear shifts in the industry that are making it easier for even the most traditional companies to work like disrupters and change the way they deploy analytics. They are:

  1. Intelligent data preparation.
  2. Containers for analytics.
  3. ModelOps for machine learning.

 The convergence of these three technologies has the potential to change the way you work with data and analytics – so you, too, can think like a disrupter.

Intelligent data preparation

Data has always been a problem. Data scientists still spend a lot of time manipulating data. And now that you have more – and more complex – data, how will you deal with it? What if we recast data management as a problem with an AI solution?  

Intelligent data preparation uses AI algorithms to recognise patterns in data, understand what data belongs together and provide context around data.

Right now, training a data set is usually a manual process that requires a lot of labelling. You take a piece of data or an image and add metadata. You start by saying, this is a cat, this is a dog, this is a barn. You label all that data and then feed it to a training set. In a three-dimensional world of video, you might label plywood, a corner of plywood, the length of the plywood.

Instead of requiring humans to tag and label the data, intelligent data preparation offers auto tagging and auto labelling. It uses reinforcement learning to create reason-based algorithms. It can even teach models to learn from other models and then to retrain the model itself. It’s continuously learning based on the decision it’s making.

Intelligent data preparation isn’t limited to image data. You can also use it across the board on streaming data, static data, master customer data – and more. It can join columns, clean fields and even combine warehoused data with streaming data – all automatically.

Disrupters have these capabilities built into their systems already. They are working with traffic data, hospitality data, streaming data – and more – and processing it intelligently for thousands of business decisions every day.  

Containers for analytics

In the world of IT, containers are the latest essential for deploying software in cloud environments. They fit nicely into the way IT likes to test and run software because they deploy software faster, manage upgrades simply and make it easy to combine different software packages. You can try things out without forcing a hard upgrade.

For software providers, it also makes it easier to build packaged deployments, integrate with other packages and add more automation into a system to self-optimise for customer workloads or other requirements.

Ultimately, containers will help to democratise the use of advanced analytics and lower the barrier to entry for trying new software products. They make it very easy for you to get software at the pace you want to consume it. And you can experiment with new algorithms or new techniques without a lot of upfront risk or expenses.

Imagine, for example, that you’ve deployed a machine learning algorithm for a next-best offer. When a new algorithm becomes available for that same task, you can automatically download that and funnel five per cent of your traffic through this algorithm. Then you let the machine tell you if it’s performing better than the previous algorithm.

If it is, the machine starts funnelling 10 per cent of visitors through the new algorithm, then 25 per cent and so on until you’ve replaced the algorithm altogether. If, on the other hand, the results are not improved, it returns the algorithm and goes back to how it was doing it before.

The disrupters are constantly testing new algorithms and running new programs in this manner. It’s one way they keep improving with small increments at scale. With containers, you’ll be able to do the same.

ModelOps for machine learning

How do you cycle machine learning models from the data science team to the IT production team? Do you do regular deployments and updates? Do you watch for models to degrade and take action? Do you put all of your best models into production?

ModelOps is a process you can use to move models from the lab to validation, testing and production as quickly as possible while ensuring quality results. It helps you manage and scale models to meet demand and continuously monitor them to spot and fix early signs of degradation.

If you use this ModelOps process, you will put more models into production and see continued results:

  • Data: Explore and access data from a trusted and secure source.
  • Develop new models: Create models with deployment and monitoring in mind.
  • Register models: Preserve data lineage and track-back information.
  • Deploy models: Improve deployment speeds with close collaboration between data scientists and IT.
  • Monitor models: Consistently track performance, and then retrain or replace models as needed.  

 This process is designed to be sensitive to data fluctuations, model bias and model degradation. It improves time to deployment for new models and ensures regular updates.

A solution for ModelOps can help you compete with the disrupters who have perfected this process while deploying and monitoring models by the thousands.

Work like a disrupter

The three technology developments discussed above are changing the way we manage data and roll out analytics projects. If you’re already analytically mature, you can use these new technologies to become technologically mature as well. Use these tips as first steps on your journey:

  1. Pick an area where you have ongoing data management issues and try to tackle them with intelligent data preparation.
  2. Look at where you have complex software and code footprints and see if containers can help simplify your infrastructure for analytics.
  3. Examine your existing data science work and see where you can benefit from a governed process for managing, deploying and updating models at scale.

If, on the other hand, you are technologically mature with a strong IT infrastructure, but not analytically mature, you can benefit from these technologies too:

  1. Consider what outside data sources you can bring in if you used intelligent data preparation, or what advanced algorithms you might be able to develop if your data was more intelligent.
  2. Look into using containers for analytics, not just for your operational systems. The same best practices you’ve learned for automation and portability in those systems will help your analytics efforts too.
  3. Determine where you could make the biggest impact with machine learning models that are managed as a corporate asset and start to experiment with advanced analytics while keeping your software development strengths intact.

No matter where you are on the continuum from traditional to disrupter, you can benefit from exploring the technologies described here.

It can be a challenge to advance on the scale of technological maturity, but I’ve seen it happen. I’ve watched markets shift and opportunities open up as traditional companies took their analytical skills and paired them with new technological skills. It could be you next.

Related stories
Top stories
Story image
Could New Zealanders initiate a cyber attack from within?
The threat landscape is significantly increasing worldwide, and the opportunities it presents are a growing concern in Aotearoa.
Story image
The path to bolstering supply chain security in New Zealand
A significant amount of today's business and leisure activity relies on IT supply chains. From complex international freight trades to local small business distribution channels, any supply chain that involves IT infrastructure serves as a crucial tool in our daily lives. 
Story image
Ingram Micro
Ingram Micro NZ bolsters MSI product range with new offerings
The inclusion of MSI Mobile Workstation and Business & Productivity laptops rounds out the MSI product portfolio.
Story image
Digital Transformation
SAP partners with New Zealand Rugby for digital transformation
The multi-year partnership will see SAP advance NZR with its organisational operations, team performance, fan experience and sustainability goals.
Story image
Symbio consolidates TNZI business to support APAC expansion
Symbio has recently announced the consolidation of its international business (TNZI) under the Symbio brand to support its Asia Pacific expansion strategy.
Story image
The paradox of change, and how to get around it
The business decision to technologically transform should not come from an IT department but from the business itself.
Story image
Red Sift
Entrust expands strategic partnership with Red Sift
Entrust has expanded its strategic partnership with Red Sift to make it easier for businesses to adopt Brand Indicators for Message Identification (BIMI) standards for email identification and security.
Story image
Third-party automotive apps bear significant privacy risks
Mobile applications for connected cars provide various features to make life easier for motorists, but they can also be a source of risk.
Story image
MEC spending for private cellular networks to reach $5.8b
A new ABI Research report finds the revenue for Multi Access Edge Compute (MEC) deployments for private cellular networks will reach USD$5.8 billion by 2030.
Threat actors are exploiting weaknesses in interconnected IT/OT ecosystems. Darktrace illuminates your entire business and takes targeted action to stop emerging attacks.
Link image
Story image
Identity and Access Management
The post-pandemic workforce requires secure IAM capabilities
HID Global discusses what identity and access management means for organisations in today's convoluted digital world.
Story image
Contact Centre
Leveraging technology in contact centres to reduce attrition rates
Many organisations worldwide have accelerated DX to better respond to changing market drivers and business environments after the disruption of the pandemic.
Story image
GapMaps Live to improve brand decisions on physical locations
GapMaps has released its latest service GapMaps Live, giving more insights and features to help brands make better decisions about physical locations.
Story image
Data Protection
Information management capabilities to meet privacy requirements
Organisations with customers or operations across more than one country face a spate of new and proposed privacy and data protection laws.
Story image
Silver Peak
The path to an adaptive, modern network
Managing and securing the network looks different than it did just two years ago—especially given that most of these networks are made up of multi-generations of infrastructure stitched together over time.
Story image
Accenture - a collective security approach a driving factor for cyber resilience
With the approaching Davos World Economic Forum upon us, it is even more imperative to discuss the impact of cybersecurity on business operations leading into the future.
Story image
SoftIron named global leader for efficient DC infrastructure solutions
SoftIron has been named a global leader for supplying energy-efficient data infrastructure solutions for core-to-edge data centers after an assessment by Earth Capital Ltd.
Story image
Cyber attacks
Devastating cyber attacks expected to hit energy sector
Energy executives anticipate life, property, and environment-compromising cyber attacks on the sector within the next two years.
Story image
Microsoft previews Power Platform website design offering
Microsoft has announced the preview of Power Pages, the fifth product in its Power Platform family, designed for low-code makers and professional developers.
Story image
Ponemon Institute
Email revealed to be riskiest channel for data loss
More than half (60%) of organisations experienced data loss or exfiltration caused by an employee mistake on email in the last 12 months.
Story image
Data Center
Preventing downtime costs and damage with Distributed Infrastructure Management
Distributed Infrastructure Management (DIM) can often be a lifeline for many enterprises that work with highly critical ICT infrastructure and power sources.
Story image
Managed service provider
Barracuda MSP Day 2022 highlights MSP opportunities
Barracuda Networks has released a report showing global services-related MSP revenue is set to increase by more than a third in 2022 compared to 2021.
Story image
Customer experience
The importance of service level management to customer experience
Staffing shortages have impacted site reliability engineers in particular since they are under extreme pressure to ensure that digital assets perform at optimum levels 24/7.
Story image
Digital Signage
MAXHUB's Digital Signage range to bolster boardroom productivity
The new MAXHUB Digital Signage technology is purpose-built to make every kind of team meeting more effective.
Find out how a behavioural analytics-driven approach can transform security operations with the new Exabeam commissioned Forrester study.
Link image
Story image
Artificial Intelligence
Gartner reveals top three tech trends for banks this year
Gartner says generative artificial intelligence, autonomic systems and privacy-enhancing computation are gaining traction in banking and investment services.
Story image
Sift shares crucial advice for preventing serious ATO breaches
Are you or your business struggling with Account Takeover Fraud (ATO)? One of the latest ebooks from Sift can provide readers with the tools and expertise to help launch them into the new era of account security.
Story image
The ups and downs and runarounds of catching cybercriminals in NZ
We're becoming more and more aware of cybercrimes but how many criminals actually get caught? The New Zealand police explain why the answer is complicated.
Story image
NVIDIA announces a spate of new innovations at Computex 2022
NVIDIA has announced its latest innovations in data center, robotics, content creation, and gaming in a virtual keynote address on the opening day of Computex 2022 in Taipei.
Story image
What every CISO must answer to enable a best-in-class security operations program
It has been widely reported recently that South Australian government employees have been the victims of a cyberattack.
Story image
Sysdig unveils new Kubernetes troubleshooting and cloud innovations
Sysdig has introduced two new innovations that look to help bolster cloud services and simplify Kubernetes troubleshooting.
Story image
Informatica, Oracle enter strategic global cloud partnership
Oracle named Informatica as a preferred partner for enterprise cloud data integration and data governance for data warehouse and lakehouse solutions on OCI. 
Story image
Digital Transformation
Digital transformation increasing business complexities
A new survey suggests businesses must re-examine their digital transformation approach to better help employees adapt to change.
Story image
Top data and tech challenges for APAC banks - report
InterSystems’ new report finds that 87% of A/NZ banks experience frustrations and concerns in using their data to drive decision-making.
Story image
Microsoft NZ and TupuToa to boost diversity in cybersecurity sector
Microsoft NZ has teamed up with TupuToa to co-develop a cyber security employment programme specifically aimed at creating more diversity in Aotearoa's cybersecurity sector.
Story image
Fortinet introduces self-learning AI in latest offering
Fortinet is introducing self-learning AI capabilities in its new network detection and response offering, FortiNDR.
Story image
Digital Transformation
Harnessing digital innovations to maximise loyalty programmes and improve CX
When it comes to the retail sector, merchants have never had access to so many different client touchpoints and data to understand their customers better.
Story image
Microsoft NZ announces initiative for local SaaS businesses
Microsoft has announced a new initiative, bringing together all of its tools to help local SaaS businesses build, sell and grow their companies at scale.
Story image
Infosec unveils role-guided cybersecurity training roadmaps
Infosec Skills Roles maps hands-on training and certifications to the 12 most in-demand cybersecurity roles to maximise training efficiency.
Story image
Deloitte expands cloud observability practice with Dynatrace
Deloitte is expanding its cloud observability practice, including DevOps principles, AI/ML, cloud complexity management and software engineering.
Story image
Elevation of Privilege the top 2021 Microsoft vulnerability
BeyondTrust has released its 2022 Microsoft Vulnerabilities Report, finding that Elevation of Privilege is the top vulnerability category for the second consecutive year.
Story image
Chorus and Nokia launches first trial of 25G PON broadband
Chorus and Nokia have announced the successful demonstration of 25 gigabit per second fibre (Gbps) broadband technology at the Chorus Fibre Lab in Auckland. 
Story image
Aligned Data Centers increases sustainability-linked loan
Aligned Data Centers has increased its sustainability-linked loan from $375 million to $1.75 billion to speed up the next phase of its strategic growth.
Story image
TUANZ to address rural connectivity at 2022 symposium
TUANZ is hosting the Rural Connectivity Symposium for the first time in person since 2019, providing a forum to discuss the state of rural connectivity.