Story image

Fast data analytics requires even faster governance

17 Oct 16

The Apache Hadoop framework, with its open source methodology and thriving ecosystem of supported projects, is often the preferred platform for analysing large volumes of diverse data for business insight.

The recent Big Data Everywhere San Francisco conference underscored two of Hadoop’s most prominent virtues: its flexibility and speed. While the theme of the conference was analytics that can be performed using Hadoop, the concerns over real-time analytics pertain to any fast data platform or target capable of ingesting, transforming, and analysing data in real time.

Organisations within regulated industries, for example, need to exercise caution with applications or use cases that leverage nearly instantaneous insight. For fast data analytics and insight, an updated and flexible policy framework is needed to accommodate the discovery of potentially noncompliant trends and information.

Real-time analytics needs supervision

Today, the need for compliance-focused supervision over fast data analytics projects is emerging as a new business concern not only for regulated organisations, but also organisations that harbour concerns about inappropriate access to data or misuse of analytics.

Data can be ingested, analysed, and visualised in near-real time – leaving little chance for compliance practitioners to interpret or react to trends revealed by analysis. Instead of increasing dependence on compliance tools and technologies, regulated firms would be best advised to focus on developing a compliance policy framework that can scale with data.

Realistically, this means establishing channels of communication and routine opportunities for compliance teams to review analytics initiatives, the data they are based on, and the insights they produce.

This is relevant today due to a timely convergence of factors: propagation of unstructured data sources, sophistication of natural language processing, consumerisation of analytics tools, and the legal expectation that businesses should be held accountable for their data, regardless of where it resides.

This means that compliance “red flags” might go completely unseen by compliance personnel. For instance, a marketing department using a Hadoop-based system to analyse social media may uncover a trend of public complaints regarding a dangerous product failure.

If the compliance team is not made aware of this and appropriate actions are not taken in a timely manner, the business may be exposed to expensive civil litigation and/or settlements.

The modern technology landscape, and particularly open source technology platforms like Hadoop, makes it possible to analyse data and implement automated decision-making at a speed that far outpaces the human ability to interpret the information.

Therein lies the problem for risk management and compliance teams, whose authority lies in their ability to override and enforce the policy settings which are often carried out with automated technology. Traditionally, unstructured data types subject to compliance monitoring could be captured and handled by point tools, such as those designed for email.

But with the proliferation of data types and the push to cross-analyse content from different sources, there is now the risk that “red flag” patterns – of particular interest to compliance managers – may now exist outside of the systems originally designed to enforce regulatory requirements.

Article by Paige Bartley, senior analyst, Information Management, Ovum.

Dimension Data nabs three Cisco partner awards
Cisco announced the awards, including APJ Partner of the Year, at a global awards reception during its annual partner conference.
WatchGuard’s eight (terrifying) 2019 security predictions
The next evolution of ransomware, escalating nation-state attacks, biometric hacking, Wi-Fi protocol security, and Die Hard fiction becomes reality.
Rimini Street hits NZ shores with new subsidiary
The third-party support provider for Oracle and SAP has opened a new Auckland-based office and appointed Sean Jones as NZ senior account executive.
Why the adoption of SAP is growing among SMEs
Small and medium scale enterprises are emerging as lucrative end users for SAP.
Exclusive: How the separation of Amazon and AWS could affect the cloud market
"Amazon Web Services is one of the rare companies that can be a market leader but remain ruthlessly innovative and agile."
HPE extends cloud-based AI tool InfoSight to servers
HPE asserts it is a big deal as the system can drive down operating costs, plug disruptive performance gaps, and free up time to allow IT staff to innovate.
Digital Realty opens new AU data centre – and announces another one
On the day that Digital Realty cut the ribbon for its new Sydney data centre, it revealed that it will soon begin developing another one.
A roadmap to AI project success
Five keys preparation tasks, and eight implementation elements to keep in mind when developing and implementing an AI service.