Data is the new gold. It has (almost) become a cliché by now, but that does not make it any less true. Companies and their services increasingly rely on data for their operations and innovations. However, this also means that the data should be available at all times. If a company’s data infrastructure fails, customers are quick to remember outages and switch to more stable competitors.
One of the best ways to guarantee uptime is automation. Spending less time on installations and configurations frees up valuable resources for everyone in the DevOps cycle. Database engineers in particular will focus their attentions on innovations instead of repetitive manual tasks. In this article, we will look at why automation matters, what it means for databases, and how versioning can simplify the delivery process even more.
In a constantly adapting and increasingly digital world, automation means much-needed flexibility. Today’s IT infrastructures vastly differ from traditional systems based on monolithic applications. With multi- and hybrid cloud infrastructures and frequent fluctuations in traffic, scalability is more important than ever. If you factor in the multitude of microservices in the average system, it’s clear that systems have become an intricate web of mutually dependant components.
Dealing with that complexity is far from easy. As a part of the DevOps philosophy, automation is one of the best ways to handle multi-faceted setups consistently, efficiently, and securely. Innovations and best practices from the world of software often apply to other IT infrastructure components, and as the backbone of IT infrastructures, data platforms are no exception.
Automation in data infrastructures means that the entire system can be set up and configured automatically. You can use it to manage different databases in multiple environments from a single central location or web interface that stores the complete configuration. Variables include its environment, scalability requirements, and other parameters. Each part of the data infrastructure still has its own variables and other specifics, but everything is monitored at a regular interval and changes can be made quickly and consistently.
This saves database administrators (DBAs) significant amounts of time, which they can invest in fine-tuning the configuration. Setups that used to take hours or even days can now be executed consistently in just a few minutes. It avoids errors caused by manual repetition, and setups no longer require an operations workbook. DevOps employees can even integrate testing into this workflow and create a single pipeline for a data platform.
Automation frees up valuable time for innovation and other tasks, but we feel that it’s important to note that it should not lead to carelessness. Having everything run automatically can lead to a loss of familiarity. After all, you can only truly get to know a system when it fails during production or stress testing. DBAs should not be afraid to still dive into the configuration files now and then.
We often mention automation alongside high-available data platforms, since these require the data to be spread around multiple databases, servers, and even data centres. When one of these parts fails, the others take over to prevent outages. Of course, to ensure a seamless experience for users, switching between those components needs to happen automatically.
Building a high-available database solution requires many different interacting components, which is why we prefer the term data platforms when talking about our work. Starting from enterprise-ready platforms like EnterpriseDB, you can add Patroni patterns for quick deployments, and Pgpool and PgBouncer for connection pooling. Tie it all up with Ansible automation and your infrastructure will get to a point where complex changes like zero-downtime rolling updates become easy.
However, we have also noticed that integration of these tools is still limited. Some companies are already using tools like Ansible, Puppet or Chef, but only use the base capabilities. If companies truly want to grow sustainably and ensure the same experience for new customers, they have to take their data infrastructure automation a step further. The central configuration point should not only keep installations consistent; it should also support different versions of the same database.
Most people are familiar with the Git version control system for software. It lets developers manage code by testing and merging its iterations. Since you don’t just keep one version of your code, why not do the same for your data platforms? Data is now just as important as code, and general tools like Git can help with that. There are even data infrastructure-specific tools like Liquibase and Bytebase, whose use is more widespread. DBAs can use these tools to track changes and varieties across their infrastructure’s components.
Administrators can use it to create and monitor separate versions of the same database. For example, they can push a separate version every time they launch a development environment. Each version has its own technical parameters like resource usage, but also specific structures, data sets, and even data models.
In other words: development, acceptance and production environments can use the same base set configuration, but can also vary according to their needs. Having separate, optimised sets of data models is especially interesting, since most companies only consider revising those once a year at best.
According to us, it’s necessary to take that next step. That’s why we set up on internal project on automated databases and versioning. It’s also why we prefer the term data platform: it represents an integrated view where approaches like automation and versioning, but also general DevOps, are used to make data a true part of the business.
Still have some questions? Looking for an expert partner with specialised knowledge in automation and high-available open-source databases? Need someone to manage and maintain your data platform as part of your automation approach? Contact us, and we’re sure Hieda can help you out!
