Growing adoption of DevOps has been one of the trends of the last 15 years or so in software engineering; in short, DevOps stresses the collaboration between Development (writing the software) and Operations (running the software).
This arose because, historically, these were segregated roles, with those in Development handing over their work to Operations; however, this segregation lead to applications that were built without taking into consideration how they would actually be run in production, leading to all sorts of problems:
applications that needed thousands of configuration settings to run;
applications that did not scale, i.e., could not handle increases in volume;
applications that fail in mysterious ways.
Silent machines
A recurring theme that we see with teams that deploy software is how silent it is, in a bad way: the application could be running for weeks until someone (often a user) reports that something isn’t quite right.
Why does this happen?
The simple reason is these applications don’t provide operators with even simple things like:
logs, textual output describing what the application is doing;
metrics, usually numeric data that measure different aspects of the application’s operation.
In other cases, this information does exist, but needs to be proactively looked at… which means that:
it usually isn’t;
if the app is expected to run 24/7, you need someone to check in around the clock (this is a good way to burn out your engineers).
Don’t call us: we’ll call you
What you want is to have your software make some noise when it’s sick, so you can heal it. While conceptually simple, this can (and often does) require a lot of effort to build in and coordinate between several pieces.
Software architecture
At a minimum, any modern toolchain that you build software on needs to perform logging. This is so errors can be identified and operators notified; if this doesn’t exist, you should look into a different toolchain.
Then, your application needs to be architected in a way that ensures that, by default, all errors are handled centrally; what this means is that, if an error occurs, you guarantee that something specific (usually logging or reporting to an external service like Sentry) happens. If it becomes possible for an error to be silently ignored, you’re relying on your customers to (angrily) report them to you.
Your platform
Even if you’ve carefully built logging into your application and instrumented it to collect metrics, you need to push this data into a place where you can look at it and setup some rules that’ll notify you when something seems suspicious. This is where your choice of platform (the place where your software runs) can make things easier or harder:
a good platform can do most of the work for you; all you have to do is deploy your app and everything will appear in its place, as if by magic;
a bad platform won’t give you this (or it might charge you obscenely for it), leaving you with no option but to run more software in order to make up for it.
Conclusion
While application development tools have come a long way, they cannot, by themselves, replace proper design and implementation: even when the best tools are available, software requires the employment of certain architectural patterns to ensure that it performs its intended business duties and, most importantly, that we know when it’s failing and have the data to learn why.
In this article, we’ve outlined a few of these techniques, which are part of our assessment of client maturity. Do you feel that this is an area that might need improvement? If so, get in touch.