Interview: SolarWinds explores databases & DataOps in the cloud-first era

Database management has progressed in leaps and bounds as organisations look to cloud migration, fuelling the birth of DataOps and automation that have drastically changed the way organisations collect, use, and manage seemingly endless tides of data every day.

We spoke to SolarWinds head geek Thomas LaRock to discuss the world of database administrators (DBAs), the possibilities within DataOps, and how SolarWinds is helping organisations navigate challenges and opportunities in their journeys.

While the definition of DBA can mean many things to many organisations, the general overview is the DBA helps organisations create or maintain databases according to their data collection and management requirements.

But as LaRock points out, those who work across DBA roles can have vastly different responsibilities depending on the nature of their organisation.

"Some may focus on maintenance tasks, others do more development work, and there are also site engineers who used to fall under old preconceptions of a DBA. These different roles can create challenges."

The challenges lie in two areas: the level of expectations from end users, and what role a DBA has in their organisation. DBAs are also using technologies that change rapidly from day to day. On top of that, DBAs and database developers are also navigating a global pandemic where priorities are constantly changing.

LaRock likens the concept of a database to a black box—often, organisations don't know what's going on inside it.

"You know when something's gone wrong because there's a feeling that something should be faster than it is. The feeling is usually vague and often undefined, so a DBA needs to figure out why something doesn't work to a user's expectations. Somebody will just call you and say, this needs to be faster, and you have to start extracting information to find out exactly what the problem is."

Issues within databases are inevitable, particularly in today's world where organisations are creating, collecting, and processing millions of data pieces every day. From this inevitable rise of data came DataOps, which provides a framework for data management in databases and other places.

"DataOps is not just automation, but there is also a feedback loop. Automation, in this sense, is more of a data engineering function and a pipeline. Data needs to move in and out of systems, but it also needs to be automated."

Imagine a system in which a database fails on one particular row or piece of data. That failure stops the entire system, and it requires a manual process to sort out what went wrong.

"For example, a DBA in a financial services organisation might receive files from banks, overnight. These files will then get input into a system, but if there is one little garbage character in there, the whole thing might be abandoned. Then it's back to square one as the DBA would show up in the morning, rerun the log, and figure out where the data problem is."

The goal of automation is to minimise human involvement. Instead, human interaction should focus on observability—how the system is doing, what kind of error rates there are, if the system is flowing, and anything that needs manual reprocessing. That's essentially the automation part of DataOps.

"DataOps is a way of automating all of that because there's no real reason the human has to get that involved, especially for one record—toss it to the side, for now, let everything else process. Let's come back and figure out why that one record is wrong. Everything else continues in the background. DataOps provides a constant feedback loop about things that go wrong, if users have problems, or if a pipeline has a higher error rate than normal. So DataOps must have a feedback loop built-in."

While DataOps and DBA vary depending on individual business needs, there is one thing every organisation requires: a data catalogue. This is essentially an inventory of all datasets that enables organisations to find out, describe, and organise their data. There are different data catalogues for various business purposes.

"I think companies would be surprised to learn they're collecting and likely paying for data feeds that contain a lot of the same information," says LaRock.

Think of financial services, for example. The word 'cash' can mean different things to different parts of the business. If there are 10 different definitions, creating 10 columns in a table based on each definition would be wasteful.

"You have to build this proper data catalogue to get started because before you start building data pipelines, you want to know more about the data you're even working with, what's required, and what you want compared to what you need."

"The first step to DataOps is building out that data catalogue and understanding that you might have a massive amount of data to deal with. If there are thousands of systems in a business, pick one that's mission-critical or perhaps a couple that are linked. Once you're done, figure out where the next system is and if there are any overlap points. Is there something you can streamline between those two feeds?"

The SolarWinds portfolio of database solutions includes Database Performance Analyzer, SQL Sentry, Database Performance Monitor, and many other solutions.

"Essentially, these solutions cater for everything from high-level to deep-dive, either as-a-service or self-hosted by our customers."

Database Performance Analyzer is designed for database query performance monitoring, analysis, and tuning across platforms like Oracle, Azure SQL Database, AWS RDS, Microsoft SQL Server, PostgreSQL, and MySQL.

DBA xPress is a free tool that can help organisations compare and synchronise data across databases, analyse dependencies and connections, browse SQL Server database content, and much more.

"DBA xPress enables users to find out where a piece of data is across their systems. It can also show different systems related to each other, which is essential when building a data catalogue and mapping data around their enterprise database," says LaRock.

"Of course, when you start building that catalogue out, you start with understanding where your data is and where the overlaps are with, that's going to help you when you start thinking about how you need to migrate your systems to the cloud.

"Migration could be as simple as a shift to cloud, or you could talk about a re-engineering of your system as you migrate. Both of these require an understanding of the data being shifted, and that's where a tool like DBA xPress comes in.

"On top of that, you could add performance tools that compare performance before and after migrations. Many companies stumble on this point when they're migrating to the cloud or starting DataOps, so it's important to have the right products and solutions to help data professionals get the job done."

Learn more about SolarWinds here.