Why Database Patching Best Practice Just Doesn’t Work and How to Fix It

Patching really, really matters – patching is what keeps
technology solutions from becoming like big blocks of Swiss cheese,
with endless security vulnerabilities punching hole after hole into
critical solutions.

But anyone who’s spent any amount of time maintaining systems
will know that patching is often easier said than done.

Yes, in some instances, you can just run a command line to
install that patch, and that’s it. These instances are increasingly
rare though – given the complexity of the technology environment,
you’re more likely faced with a complex process to achieve patching
best practice.

In this article, we’ll outline why database patching matters
(yes, databases are vulnerable too!), explain what the problem is
with patching databases, and point to a novel solution that takes
the pain out of database patching.

Watch out – your database services are vulnerable too

We know that database services are critical – databases underpin
IT operations in countless ways, working away in the background.
Yet databases just aren’t the most interesting parts of the
technology stack, which is one of the reasons database patching can
get neglected. In a recent survey by Imperva^[1], the company found that
nearly 50% of on-premise databases were vulnerable to a known
exploit.

Cybercriminals, however, are not ignoring databases. Just like
any other element of the technology stack, databases are full of
vulnerabilities. Just one database service alone has over a thousand related
vulnerabilities^[2].

Consider a few examples. In September 2016, CVE-2016-6662^[3]
was reported, a vulnerability that allows attackers to inject
malicious MySQL configuration settings into a victim’s database
service. It affected MySQL clones too – including MariaDB, which
was forced to publish detailed mitigating steps
here^[4].

Another example: in 2020, a database vulnerability was
identified where^[5]
attackers could mount a privilege escalation attack because of how
certain versions of MariaDB handled “setuid” at the installation
stage.

In both our examples, patching – or upgrading to a newer version
of the database service – would close the vulnerability. But herein
lies the problem: patching doesn’t happen as consistently as it
should, and not just because tech teams are lazy – or because
database services are forgotten about.

Just get on patch management, right…?

Not quite. There’s a second reason why database patching gets
neglected – patching a database can be incredibly hard, with
conflicting and ambiguous instructions. This problem is
particularly prevalent where database implementations are quite
complex.

Take MySQL clusters, for example. The open-source database MySQL
has an official article outlining how users need to
patch a MySQL cluster^[6]
– but the instructions are intricate, and it only considers one
particular setup of MySQL cluster, InnoDB, when there are other
MySQL cluster strategies.

The above MySQL instructions also leave out a few important
aspects of patching. It doesn’t cover how the patching process may
affect other applications – or how it may affect other systems in
your technology solution. It can’t offer this advice, of course,
because every environment is different, and the writers don’t know
what your environment looks like.

And therein lies a major issue with patching best practice, and
database best practice in general: it’s almost impossible to
account for the infinite practical variations – from differences in
database configuration to different levels of technical
knowledge.

Patching best practice just fit for purpose

The net result can be that implementing published patching best
practice is a very ambiguous and uncertain exercise. Sysadmins can
easily decide the risks and implications of patching going wrong is
much more significant than the risk of a cyberattack on the
database. So, while in theory, it’s easy to just “get on with
patching”, the reality is very different.

Even where teams have the technical knowledge and the practical
certainty to make a success of database patching, there is still
the reality that a database service must go offline for some time
to perform the patching.

Without high availability, downtime is the most disruptive side
effect as tech services go offline, disrupting work.

High availability configurations^[7] can ensure that there’s
no downtime, but even these are likely to experience service
degradation as some servers in a cluster are offline and unable to
support demand or provide adequate protection while some nodes are
down for maintenance.

Complex patching procedures also consume a significant amount of
time which takes resources away from other important tasks – and in
some cases, the resources might simply not be there to ensure
consistent patching.

Finally, taking databases offline for patching and managing
complex migration processes always carries a risk of something
going wrong. Data corruption could creep in during migration, or
some servers may fail to come back to life after patching. These
risks can’t be ignored – and are intrinsic to current database
patching practices that require restarts.

Live database patching as an alternative

Until recently patching a technology service almost always
required a restart, but live patching is becoming increasingly
prevalent. With live patching, the patching tools perform an
in-place swap of the patched code: the service being patched keeps
running while patching takes place, with no restart required.

That’s exactly the role of DatabaseCare^[8], the new solution from
TuxCare. Thanks to DatabaseCare, you can perform comprehensive
patching as often as you like because DatabaseCare patches your
database while it is actively running and serving data.

How does this work? It’s simple in practice. Your server
connects to the on-premises DatabaseCare ePortal where patches are
securely stored. As soon as a new vulnerability is logged, an agent
communicates with the ePortal, which then pulls the update from
DatabaseCare. The agent then momentarily freezes your database
service in a safe mode, and transparently applies the patch in
memory. This “freeze” is so fast that it doesn’t even disrupt
network connections to the database service or running queries.

The result: your databases are automatically updated with the
latest security patches, without downtime, with minimal disruption
and risk – and with zero ongoing effort from your technical
teams.

How does DatabaseCare benefit you?

Let’s take a clearer look at the benefits of live patching –
compared to patching best practice as it stands.

We’ve already pointed to the complexity of database patching,
particularly for high availability, distributed databases.
DatabaseCare replaces a complex set of steps involving tricky
migration procedures with a single, one-time, simple step – that’s
easily automated too.

It removes the ambiguity from patching your databases. Are you
following the right instructions? Will it work, even if you do it
perfectly? All those questions are now gone – patching happens
automatically and in the background. And so yes, the risk involved
in patching databases is now significantly mitigated, which reduces
the hesitation to patch.

At the same time, automated patching also means that you don’t
need to try and fit patching in amongst another long list of
draining IT tasks. And, when patching doesn’t compete for
resources, it happens more regularly. Other business units inside
the organization will appreciate how you no longer require long
maintenance windows for your patching operations.

We all know what regular patching means: tighter security.
Reduce the window between the release date of a patch, and when
that patch is applied, and you reduce the window of opportunity for
attackers to exploit a vulnerability.

Best practice matters – but DatabaseCare unlocks consistent
security

Patching database services isn’t new – best practice
instructions have been around for some time. But there are
practical difficulties to patching best practices as it stands, and
these practical difficulties leave a window for cyber
attackers.

DatabaseCare plugs the gap – it doesn’t disrupt your operations,
it doesn’t pose a risk of failure, and you don’t need resources to
make it work. In turn, your security becomes much more solid.
Installing DatabaseCare is simple too. To find out more, just
review the DatabaseCare page^[9]
on the TuxCare website.