Story image

Interview: Why log management is crucial to modern business

08 Mar 2019
Sponsored

Recently I had the chance to sit down for an interview with Balázs Scheidler, an industry veteran in IT security and software engineering.

In 2000 he co-founded Balabit because he was tired of using the old syslog protocol and not getting valuable data fast enough from log messages for security analytics and IT operations.

In just a few years his syslog-ng log management solution became the de facto industry standard for log collection in Linux-based installations, used by millions of corporations around the world. Having established a name for itself, the company then delved into the Privileged Access Management (PAM) solution space.

Last year Balabit was acquired by One Identity, with Scheidler’s role now focused on shaping the future strategy for its PAM and log management solutions.

Scheidler says syslog was originally created in the early 1980s to debug software as part of the Sendmail project.

"Basically, it was designed to be a fire and forget approach to log management and logging in general. It was very simple as they didn't care too much about how reliable it was or even security in general,” says Scheidler.

Their point was to make it possible to get activity data from the mail servers off those hosts and to their separate repository where they could run their own statistics and reporting.

"Vendors of network gear, applications and OSes started to adapt syslog for their own use, which expanded the scope of data considerably. It was very difficult to extract value from log messages because of the huge amount of unstructured data, making it a bit like finding a needle in a haystack," says Scheidler.

Despite this, businesses still saw the huge value in log messages as if there was an incident or some kind of outage, the only data source was this data. Even if an incident was recognised years after it happened, you could still go back and find the details and root causes.

However, Scheidler says there are still organisations in this age relying on syslog over UDP, which doesn’t guarantee delivery and doesn’t use encryption in the logging, which means they’re being deprived of the many benefits of the latest technology to support their log management

“As I mentioned, the adoption of syslog-ng was significant early on, but today still more and more devices are becoming capable and the demand to integrate them into logging infrastructure is always increasing. Furthermore, log data is needed by more and more downstream systems. A SIEM solution is a natural consumer of log messages, but these days things like UBA or business intelligence systems also tap into logs. Using something that can connect log-producing devices with downstream processing systems is needed and syslog-ng is one of the biggest implementations of this idea,” says Scheidler.

“Fixing the reliability and security issues while adopting the growing set of devices that do logging and be able to feed more downstream systems - those are the main reasons for improving log management practices today.”

In terms of how syslog-ng differs from other log management solutions, Scheidler says the original reason he started the project was to improve reliability.

“Originally it was using UDP only, and then I introduced the TCP based-logging system and we have constantly improved that ever since. So now in a distributed enterprise-wide and global log management system we are capable of never losing messages within the log management infrastructure. We are also very good at security - when it comes to data at rest and data in transit we solve both of these issues,” Scheidler says.

“And we do this at a high scale. So I mentioned the relationship between log-producing devices and SIEMS, and feeding multiple teams which naturally means that we have terabytes of data to transport daily, and scale is a key feature of a system like that. To put it into perspective, syslog-ng can process hundreds of thousands of messages a second, and it does that efficiently. What other systems would do on 10 computers, it can do on one.”

Scheidler stressed that while there is an overlap, syslog-ng isn’t a SIEM solution. Often they can meet all the requirements for SMEs that are unable to afford a complete SIEM system or deployment (because they can tend to be quite expensive), but he generally positions syslog-ng as the ‘glue’ between log-producing and log-consuming devices.

“Effectively it can run on end systems and ensure that the collection, centralising, storage, archival, indexing, and searching functions are all performed on our side,” says Scheidler.

“You don't have to send everything into the SIEM, only those things that you really need to, which enhances the performance of your existing SIEM.”

Looking to the future, Scheidler says they’re moving with the demands of their customers.

“Our priorities now lie in the cloud space. Centralised system functions or log management functions are usually an on-premise system because you have to collect everything that you already have on-premise. Customers now expect us to fold logs coming from Cloud systems into the same repository. For example, they want to see Office 365 logs in their Splunk installation, or they want to see AWS access logs in their SIEM, to talk about it broadly,” says Scheidler.

“And that means we have to integrate with those systems as well and provide solutions for customers to solve this hybrid issue. ‘I have a lot of on-premise systems that produce logs; I have a lot of cloud-based systems that produce logs; I want to see them in the same place be that either in the cloud or on-premise’ - that is the opportunity we’re looking to capitalise on.”

In terms of implementation of the log management solution, Scheidler says it is a very simple process.

The syslog-ng Store Box (SSB) for example can be deployed in just a few days, offering a high-performance, high-reliability log management appliance that enables you to search logs, secure sensitive information with granular access policies, generate reports to demonstrate compliance and forward log data to third-party analysis tools.

Click here to learn how to adopt log management best practices.

How Cognata and NVIDIA enable autonomous vehicle simulation
“Cognata and NVIDIA are creating a robust solution that will efficiently and safely accelerate autonomous vehicles’ market entry."
Kinetica launches a new active analytics platform
"With the platform now powered by NVIDIA DGX-2, customers can build smart analytical applications that combine historical data analytics and ML-powered analytics."
Avi Networks: Using visibility to build trust
Visibility, also referred to as observability, is a core tenet of modern application architectures for basic operation, not just for security.
Zoom’s new Rooms and Meetings features
Zoom has released information about the upcoming releases for its Rooms and Meeting offerings for 2019.
Aussie company set to democratise direct-to-orbit IoT access
Adelaide-based Myriota has released a developer toolkit that has been trialled and tested by a smart waste management platform.
Dynatrace takes pole position in APM Magic Quadrant
It placed highest on Ability to Execute and furthest on Completeness of Vision in the 2019 Quadrant for Application Performance Monitoring (APM).
HCL and Xerox expand strategic partnership
Under the terms of the agreement, HCL will manage portions of Xerox’s shared services, including global administrative and support functions.
Inspur announces AI edge computing server with NVIDIA GPUs
“The dynamic nature and rapid expansion of AI workloads require an adaptive and optimised set of hardware, software and services for developers to utilise as they build their own solutions."