What if you are renovating a house, but forget to look at the outdated foundations? A lot of companies modernise their application landscape, but are not sure what to do with the supporting data infrastructure. In this blog, we will discuss the importance of databases in these kinds of projects, and why you should at least consider updating them as well. We are joined by Bart Callens, one of our co-founders, and Kilian Niemegeerts, co-founder of our parent company FlowFactor.
Application modernisation is a hot topic among most, if not all, organisations nowadays. Instead of large monolithic apps, capabilities are increasingly restructured into microservices, containerised using tools like Docker and Kubernetes, and automated through CI/CD pipelines and other DevOps principles.
Developing and running applications this way may cost some time and effort to set up, but the results speak for themselves. Applications and their development become more portable, modular, flexible, agile, quick, and maintainable. This means that companies can switch providers more easily, send out faster patches and updates, scale their applications automatically based on demand, and continuously improve their development process.
Databases, on the other hand, tend to remain monolithic. So how can companies take up databases in their modernisation journeys? And what are some pitfalls of diving in head-first? Let’s find out!
Application modernisation is all about removing bottlenecks in the development process. Databases tend to become the bottleneck when the other aspects have been improved, so it makes sense to look at them as well. If you have split up your application into microservices, but all of them still rely on one (possibly outdated) database to process every request, you end up with a bottleneck.
The solution to modernising databases is, at least in part, the same as for applications: splitting it up. If your development has become lean and mean thanks to microservices, you need multiple small “microdatabases” to keep up. Using a database per service design will let your infrastructure scale horizontally, not just vertically. Splitting up your data infrastructure into smaller units will avoid a single point of failure for your data model, just as microservices isolate critical components compared to a monolithic application.
The best way of splitting up a data infrastructure into these microdatabases is containerisation. Just like applications, databases can be developed and run in containers using Docker and Kubernetes. This might sound strange at first, since Docker containers are best used for stateless systems, where data is not stored. However, Docker has devised a mechanism to generate and use persisting data: volumes. Using volumes, the file system and the Docker container will be separated.
The result of containerisation is a data platform with tens to hundreds of purpose-built microdatabases, rather than a single database. This approach offers the same benefits as for applications. There will be a notable increase in performance, modularity, flexibility, agility, and maintainability. Thanks to horizontal scaling, increases in load and demand will be dealt with much faster with a significantly reduced overall cost.
However, there are some precautions and prerequisites to take. In the next part, we will look at how tooling and expert advice can help avoid a partial or half-hearted implementation that does not function as expected.
The most important aspect of database modernisation is to avoid diving in head-first. Just like other aspects of application modernisation, rethinking databases requires careful thought and planning. As Kilian notes, changing the way you handle data shouldn’t be done spontaneously:
“Don’t just start chopping up your database into parts without thinking of the consequences. Just like monolithic applications, complicated databases that have been used for years can be difficult to split up functionally. Data is the backbone of your operation, so you should take heed and assess all aspects of your infrastructure before starting.” — Kilian
Because data is that essential, it should be carefully monitored and controlled. Your project should therefore include some kind of version control to fully benefit from database modernisation’s agility. Since application modernisation represents a new way of working, your data platform should be able to handle these functional changes just as well as the technical ones. For example, you can use version control to keep track of the greatly increased number and frequency of releases and rollback where necessary. As Bart mentions, containerisation can help here as well:
“As the technology matures, we’ve noted a trend where developers are increasingly treating data as code in containers. Using tools like Liquibase, you can establish automatic links from a data platform to different versions of an application through YAML files. By running scripts with something like Ansible, versions can be modified and reverted with the click of a button.” — Bart
To deal with this multitude of databases and versions, you will need a way to keep track of everything. That’s where data platform tools come in. These programs provide a unified interface or portal with an overview of your complete data infrastructure. They will also help with another aspect of database modernisation: access control. Since your data will be spread around, it is key to make sure that only the right people have access to the right data.
Last, but certainly not least, you require a specialised technical partner to help guide you through this modernisation journey. An experienced advisor will help you avoid the aforementioned pitfalls, because they already know the most common mistakes in similar projects. Because the transition to a new data structure will have a lot less hiccups, the project will go much faster and therefore be cheaper.
Bringing in an expert advisor can also help to figure out which of the many tools out there is the best fit for your particular situation. Finally, an expert view from outside your organisation can help to get a new perspective. Thanks to their experience, technical partners can evaluate whether further steps are needed to modernise other parts of your infrastructure.
Looking for a technical partner to help you take your databases to the next level? Want to modernise your data infrastructure, but not sure where to start? At Hieda, we specialise in open-source databases and innovative data storage technologies. Thanks to our connection with FlowFactor, we have access to industry-leading DevOps experts who can examine your applications as well. Contact us today to find out more!