IT Brief New Zealand logo
Technology news for New Zealand's largest enterprises
Partner content
Story image

The inherent flaws in big data analysis

By Sara Barker
Mon 21 Oct 2019
FYI, this story is more than a year old

Business intelligence and big data programs are being hailed as a fast way to provide the insights that underpin business decisions, as well as planning and performance.

Many organisations will be aware that enterprise data warehouses or big data platforms such as Hadoop and DFS hold all the data. Intelligence tools then draw on that data to create visualisations, dashboards, reports, and drilldowns.

Data integration technologies like ETL also move organisations’ systems or record into a data warehouse or big data platform.

But there is a major flaw inherent in this approach, according to analytics software provider TIBCO: A few people neglect the rationale behind managing enterprise dimensions, attributes and hierarchies – if ignored, all of these result in erroneous analysis.

Take enterprise dimension management, for example. One of the most common challenges is how to deal with dimensions that don’t conform, generally because underlying systems of record define or use dimensions in different ways; or the common dimensions are missing necessary classifications and attributes.

Say that sales and marketing want to understand how effective an organisation’s industry-focused marketing programs have been at generating new opportunities. Analysts need to be able to connect the industry-coded events to your prospects and closed deals.

All those dimensions may have an industry attribute, but marketing is using a public standard such as standard industry codes (SICs), while sales is using an internally created industry classification scheme.

Data governance enthusiasts may advocate for standardisation but it’s not always so clear-cut.

TIBCO explains that the marketing team may be using SIC because that’s how all their vendors segment audiences. While the sales team may have decided to define their own classification because SIC was too complex.

Those kinds of issues create uncertainties about systems of record and whether they provide all of the attributes an organisation requires.

“It may seem like the simplest solution is to extend and manage attributes in the primary system of record, but that won’t work for your users,” explains TIBCO. “Depending on the context, they may need different attribute values altogether.”

Attributes such as markets, segments, and sub-segments reflect a product’s current segmentation – at least in a performance context. But if brand management is involved in planning and analysis, those attributes may hold different values.

“This illustrates the need for a separate system that business users can utilise to enrich dimensions before they’re loaded into the EDW or big data store. One key requirement will be a mechanism to support the planning and analysis contexts with not just new attributes, but multiple alternate attribute value sets and temporary extensions.”

To manage dimension and hierarchy management, TIBCO recommends that organisations look for following features in a solution:

  • Tools to define dimensions, attributes and hierarchies
  • Support for all types: derived, explicit, balanced, and unbalanced
  • Services to balance and link hierarchies
  • Versioning to manage past, present, and future dimensions and hierarchies
  •  Inheritance to manage alternate hierarchies
  • User interfaces to manage dimension, attributes, and hierarchies
  • No-code, browser-based UIs
  • Built-in full text and fuzzy searching and filtering
  • Perspectives and custom layout to configure role-specific UIs
  • Services for integration and distribution to downstream systems, ad-hoc analysis tools, data warehouses, and big data platforms

Software such as TIBCO’s EBX is able to help organisations manage dimensions and hierarchies, without the headaches of erroneous analytics.

TIBCO EBX also offers more capabilities, including collaborative workflow to support government and change management; a data quality engine; fine-grained security and permissions; and business rules to enforce controls and validation.

To learn more about enterprise dimension and hierarchy challenges, TIBCO offers a free whitepaper that explains common challenges such as those outlined above, as well as possible ways to solve them. 

Click here to access the whitepaper.

Related stories
Top stories
Story image
DevOps
Deloitte expands cloud observability practice with Dynatrace
Deloitte is expanding its cloud observability practice, including DevOps principles, AI/ML, cloud complexity management and software engineering.
Story image
Contact Centre
Leveraging technology in contact centres to reduce attrition rates
Many organisations worldwide have accelerated DX to better respond to changing market drivers and business environments after the disruption of the pandemic.
Story image
Symbio
Symbio consolidates TNZI business to support APAC expansion
Symbio has recently announced the consolidation of its international business (TNZI) under the Symbio brand to support its Asia Pacific expansion strategy.
Story image
APAC
Top data and tech challenges for APAC banks - report
InterSystems’ new report finds that 87% of A/NZ banks experience frustrations and concerns in using their data to drive decision-making.
Story image
Alteryx
Alteryx releases updates, empowers data insights for enterprise
Alteryx has released new advancements designed to aid enterprises with cloud analytics, democratise insights and ensure data governance.
Story image
Microsoft
Microsoft previews Power Platform website design offering
Microsoft has announced the preview of Power Pages, the fifth product in its Power Platform family, designed for low-code makers and professional developers.
Story image
DaaS
NetApp launches Spot PC, a new Desktop-as-a-Service solution
This is a new managed cloud DaaS solution with security, automation, observability and optimisation capabilities, designed for the needs of today.
Story image
Sustainability
Aligned Data Centers increases sustainability-linked loan
Aligned Data Centers has increased its sustainability-linked loan from $375 million to $1.75 billion to speed up the next phase of its strategic growth.
Story image
Microsoft
Microsoft, Cloudian partnership offers data center flexibility
Cloudian’s HyperStore object storage platform is now integrated and validated to work with Microsoft SQ Server 2022, offering more flexible and scalable data centers.
Story image
BYOD / Bring Your Own Device
How zero trust can lead the battle against ransomware
SecOps teams champion a zero trust strategy to support the fight against the escalating risk of cybercrime and help monitor threat actors across a network.
Story image
Informatica
Informatica, Oracle enter strategic global cloud partnership
Oracle named Informatica as a preferred partner for enterprise cloud data integration and data governance for data warehouse and lakehouse solutions on OCI. 
Story image
Data Protection
Information management capabilities to meet privacy requirements
Organisations with customers or operations across more than one country face a spate of new and proposed privacy and data protection laws.
Story image
Microsoft
Elevation of Privilege the top 2021 Microsoft vulnerability
BeyondTrust has released its 2022 Microsoft Vulnerabilities Report, finding that Elevation of Privilege is the top vulnerability category for the second consecutive year.
Story image
Check Point
Check Point and CCTV expert join forces to boost protection
The partnership will involve Check Point Quantum IoT Protect Nano Agent being embedded in Provision-ISR’s CCTV cameras for on-device runtime protection.
Story image
Cyber attacks
Devastating cyber attacks expected to hit energy sector
Energy executives anticipate life, property, and environment-compromising cyber attacks on the sector within the next two years.
Story image
Silver Peak
The path to an adaptive, modern network
Managing and securing the network looks different than it did just two years ago—especially given that most of these networks are made up of multi-generations of infrastructure stitched together over time.
Story image
Infosys
Consumer relationships with digital services continues to change
Two years of pandemic-induced reliance on technology for work has altered our relationship with digital apps and services, new research has found.
Story image
Ransomware
CERT NZ releases first Cyber Security Insights for 2022
CERT NZ has released Quarter One: Cyber Security Insights 2022, which offers an overview of reports about cybersecurity incidents affecting New Zealanders.
Story image
Malware
Fortinet introduces self-learning AI in latest offering
Fortinet is introducing self-learning AI capabilities in its new network detection and response offering, FortiNDR.
Story image
Sift
Sift shares crucial advice for preventing serious ATO breaches
Are you or your business struggling with Account Takeover Fraud (ATO)? One of the latest ebooks from Sift can provide readers with the tools and expertise to help launch them into the new era of account security.
Story image
Artificial Intelligence
Frost & Sullivan recognises Genesys as leader in new reports
Frost & Sullivan has recognised Genesys as a leader in the cloud contact centre market for its robust cloud and digital capabilities.
Story image
SaaS
The paradox of change, and how to get around it
The business decision to technologically transform should not come from an IT department but from the business itself.
Story image
Digital Signage
MAXHUB's Digital Signage range to bolster boardroom productivity
The new MAXHUB Digital Signage technology is purpose-built to make every kind of team meeting more effective.
Story image
Red Sift
Entrust expands strategic partnership with Red Sift
Entrust has expanded its strategic partnership with Red Sift to make it easier for businesses to adopt Brand Indicators for Message Identification (BIMI) standards for email identification and security.
Story image
Identity and Access Management
The post-pandemic workforce requires secure IAM capabilities
HID Global discusses what identity and access management means for organisations in today's convoluted digital world.
Story image
Hybrid Cloud
Barracuda expands cloud-native SASE platform
"The expansion of Barracuda's cloud-native SASE platform for hybrid deployment models and IIoT environments solves a number of challenges."
Exabeam
Find out how a behavioural analytics-driven approach can transform security operations with the new Exabeam commissioned Forrester study.
Link image
Story image
Cybersecurity
What every CISO must answer to enable a best-in-class security operations program
It has been widely reported recently that South Australian government employees have been the victims of a cyberattack.
Story image
TUANZ
TUANZ to address rural connectivity at 2022 symposium
TUANZ is hosting the Rural Connectivity Symposium for the first time in person since 2019, providing a forum to discuss the state of rural connectivity.
Story image
Digital Transformation
Harnessing digital innovations to maximise loyalty programmes and improve CX
When it comes to the retail sector, merchants have never had access to so many different client touchpoints and data to understand their customers better.
Story image
Multi Cloud
Red Hat updates dev tools, empowers use of hybrid and multicloud
Red Hat has unveiled updates across its portfolio of developer tools, designed to help organisations build and deliver applications faster.
Story image
Chorus
Chorus and Nokia launches first trial of 25G PON broadband
Chorus and Nokia have announced the successful demonstration of 25 gigabit per second fibre (Gbps) broadband technology at the Chorus Fibre Lab in Auckland. 
Story image
Ransomware
APAC organisations fail to disclose ransomware breaches
85% of organisations in APAC were breached by ransomware at least once in the past five years, but only 28% publicly disclosed the incident.
Story image
Microsoft
Microsoft NZ announces initiative for local SaaS businesses
Microsoft has announced a new initiative, bringing together all of its tools to help local SaaS businesses build, sell and grow their companies at scale.
Story image
Transport
Third-party automotive apps bear significant privacy risks
Mobile applications for connected cars provide various features to make life easier for motorists, but they can also be a source of risk.
Darktrace
Threat actors are exploiting weaknesses in interconnected IT/OT ecosystems. Darktrace illuminates your entire business and takes targeted action to stop emerging attacks.
Link image
Story image
MEC
MEC spending for private cellular networks to reach $5.8b
A new ABI Research report finds the revenue for Multi Access Edge Compute (MEC) deployments for private cellular networks will reach USD$5.8 billion by 2030.
Story image
NVIDIA
NVIDIA announces a spate of new innovations at Computex 2022
NVIDIA has announced its latest innovations in data center, robotics, content creation, and gaming in a virtual keynote address on the opening day of Computex 2022 in Taipei.
Story image
Kubernetes
Sysdig unveils new Kubernetes troubleshooting and cloud innovations
Sysdig has introduced two new innovations that look to help bolster cloud services and simplify Kubernetes troubleshooting.
Story image
Sustainability
SoftIron named global leader for efficient DC infrastructure solutions
SoftIron has been named a global leader for supplying energy-efficient data infrastructure solutions for core-to-edge data centers after an assessment by Earth Capital Ltd.
Story image
Artificial Intelligence
Gartner reveals top three tech trends for banks this year
Gartner says generative artificial intelligence, autonomic systems and privacy-enhancing computation are gaining traction in banking and investment services.
Story image
Data Center
Preventing downtime costs and damage with Distributed Infrastructure Management
Distributed Infrastructure Management (DIM) can often be a lifeline for many enterprises that work with highly critical ICT infrastructure and power sources.
Story image
Cybercrime
The ups and downs and runarounds of catching cybercriminals in NZ
We're becoming more and more aware of cybercrimes but how many criminals actually get caught? The New Zealand police explain why the answer is complicated.
Story image
Cybersecurity
Accenture - a collective security approach a driving factor for cyber resilience
With the approaching Davos World Economic Forum upon us, it is even more imperative to discuss the impact of cybersecurity on business operations leading into the future.