Implementing a big data programme in the cloud

Enterprises with limited resources may face technical and infrastructure challenges when implementing big data analytics solutions, according to CenturyLink.

Big data projects don't exist in a vacuum, the company says.

CenturyLink says a company's big data infrastructure elements should have the flexibility to support multiple approaches to integration with one another, as well as with external elements.

Martin Hooper, head of business development, Australia, at CenturyLink Cognilytics, says cloud infrastructure provides the resources needed for big data strategies.

"The data in your Hadoop application comes from many sources and is fed into a variety of applications in order to maximise their value for different constituencies," Hooper explains.

"In addition, the constant shifting of big data workloads requires your big data infrastructure to scale elastically," he says.

"Elastic infrastructure means being able to scale up and down at will. Cloud infrastructure is the best option for elasticity.

"The cloud typically gives you this ability, making it relatively simple to add or remove data analytics capacity on demand," says Hooper.

Hooper says there are four key steps to take before implementing a big data project in the cloud:  1. Plan for the big picture  "Like most IT initiatives, it usually saves time to plan a cloud-based big data program up front," Hooper says.

He says architectural decisions made early on in the project will influence the success of future phases.

"In planning, it is important to first identify potential specific business use cases, such as customer loyalty, product development, or risk management," explains Hooper. 2. Overcome cultural obstacles  According to Hooper, one of the greatest barriers to becoming a data-driven organisation is overcoming a legacy decision-making culture.

"It takes experience and credibility to make a convincing argument for a cloud-based big data strategy, and an effective change management program to succeed in its implementation," he explains.  3. Consider compliance and data sovereignty  Organisations in every industry need to deal with data privacy and security management, while many also have to maintain certain practices to meet regulatory compliance standards and avoid penalties.

Hooper says establishing an effective data governance strategy is an essential step to developing a successful cloud-based big data program.  4. Meet infrastructure requirements  "For a big data application to grow in size and capability, it needs to be developed on an architecture that is designed to support its specific needs," says Hooper.

"This is why it is important to identify the baseline infrastructure and future requirements of a big data project.

"The scalability, performance, reliability, and security of the cloud infrastructure involved in the project should be taken into consideration," he says.