There is a lot of discussion about DevOps and how it can help optimise the software development process, but what about using it for data platforms? To learn more, we sat down with Luc Robberechts, our co-founder and technical lead, and Kilian Niemegeerts, one of the founders and managing partners of our DevOps-focused parent company FlowFactor. In this blog post, we'll discuss why organisations should use DevOps for their databases, and we'll give you some tips on how to adapt the DevOps culture to your database teams. Stay tuned!
Databases are an important part of any organisation. They store critical data and need to be managed efficiently and effectively. That's exactly why DevOps is such a great fit. As a set of practices that aim to improve the speed and quality of software development and their supporting processes, DevOps emphasizes collaboration between teams and encourages the use of automation.
Both Kilian and Luc are quick to point out that, in many organisations, database code changes are bottlenecks in the delivery process. Changes on the database can severely slow things down, but they have noticed that companies are not likely to look to the database department for optimisations in the first place.
Organisations that adopt DevOps for their databases can therefore achieve a number of benefits, including improved communication and collaboration between the database team and other teams, increased efficiency and productivity, and better quality control. In addition, DevOps can help to reduce the risk of database downtime and improve the speed of recovery in the event of a failure.
If you want to use DevOps for your databases, you will need to adapt the DevOps culture to your database teams first. This means creating a collaborative environment where developers and operations teams can work together to improve the efficiency of the software development process. Unfortunately, company politics and individuals’ egos can make this difficult.
At various points in my career, I’ve had customers where they want me to fix the problem, but be cautious in mentioning where the problem lies. I was told to only mention it to “the right people” instead of directly to teams. This happens in multiple areas, but is especially common in database teams. — Luc
It’s important to make sure that the business allocates sufficient budget for substantial changes to their infrastructure. In many cases, IT uses examples to convince them of the benefits of automation. For example, Kilian recalls a case where a DBA only needed two days to set up a basic script for database maintenance. Those two days of effort saved them several hours on each deploy, and with up to ten deploys per month, the business was quickly convinced.
If you want to implement DevOps for your databases, your DBAs will see their responsibilities shift to that of a Database Reliability Engineer (DBRE). We’ve written a dedicated piece on this shift in a dedicated blogpost, so we highly suggest you give it a read. Both DevOps and automation will replace a lot of their repetitive manual implementation and configuration work. Instead, your database engineers will face interesting challenges. At the company culture level, it will also help to tear down the walls between teams like storage, database maintenance, and operating systems.
When I was a beginning database engineer, I quickly fell into routine once I had to go through the same installation document for the twentieth time. In all my projects, I notice that DBAs are more than happy to change their roles to that of a DBRE. Instead of repetitive tasks, colleagues now call them to solve interesting challenges that actually help build their expertise. — Luc
Kilian envisions a future where developers will only have to click a button to set up database environments themselves. However, both gentlemen are quick to point out that DBREs will still be essential in those cases. They will have to keep the databases, their scripts and themselves up to date on the latest technologies, and solve any interruptions in the automation process. They will also follow up on vulnerabilities and security guidelines, and enforce them through automation and updates.
DBREs can function as gatekeepers and guardians of the database ecosystem. Developers can be quick to try out new database technologies, but this could cause issues. For example, adding a NoSQL technology like MongoDB to a pipeline based on a SQL database would severely complicate the process.
Let’s be honest here: there is no one-size-fits-all roadmap or step-by-step plan to implementing DevOps for your databases. It’s important to have a checklist with the specific requirements before you start. For example, do you need multiple environments, high availability, a connection manager, or a backup system with double or even triple redundancy? It’s difficult for database engineers working in just one data environment to know all the possibilities out there, so it’s better to get external advice. They can help you set up an implementation roadmap, or check your existing one for feasibility.
In most DevOps projects, there is a phased approach. Most customers don’t want to go for the holy grail right away, especially if such a transformation upends their existing processes and pipelines. Going for the big bang right away certainly has its uses, but not always. I like to compare IT infrastructure with physical infrastructure: you need to construct the pipelines before proceeding. — Kilian
Another significant reason to look for outside help is taking a load off of your database team’s shoulders. Both gentlemen see cases where database engineers don’t want to automate, simply because they wouldn’t be able to manage the implementation work on top of their already busy schedules. Both FlowFactor and Hieda have seen the benefits for IT teams of a dedicated consultant that doesn’t have to focus on other day-to-day and operational tasks – whether those are related to databases or not.
You also need to choose the right automation tools to manage database deployments and monitor database performance in real time. In time, these will let you set up end-to-end CI/CD pipelines. Some of the most popular DevOps tools for databases include Jenkins, Puppet, Chef, Ansible, Nagios, Ganglia, and Graphite. Both gentlemen have noticed that Ansible, optionally extended with Terraform, is the most popular choice so far. It can be difficult for regular database engineers to stay on top of all these offerings, so having an expert opinion will definitely help to pick the right fit.
Finally, it’s important to realise that database automation through DevOps is a novel approach. FlowFactor oversees a lot of DevOps projects, but they have noticed that current automation efforts still stop at the intermediate level of an application server. According to Kilian, this is largely because of trends like containerisation that may seem like a one-and-done solution. On the other hand, the larger service providers in their client base are already exploring database refreshes as a first step in database automation.
Even as database experts, we were pleasantly surprised by the amount of time and effort that could be saved with database automation tools and scripts. While DevOps for databases clearly has plenty of potential to reshape the way database teams will work in the future, it still has to achieve widespread adoption. We’re excited to see what this shift means for the database industry as a whole.
Looking for a company that can help you implement DevOps for your databases? Then look no further than Hieda! We're experts in open-source data platforms, so we know how to manage databases as efficiently as possible. Thanks to our close connections with FlowFactor, we have a direct line to the best DevOps experts in the business. Contact us today to learn more about how our services can help you achieve better database management!
