One of the most important practices to arise out of this movement is Continuous Delivery, the automation of the software delivery process; by automating, teams improve the reliability and speed of releases and simultaneously, reduce the overhead involved. This, in turn, enables them to deliver more often, which provides more and better feedback.

Why and how does Continuous Delivery work, though? To understand this, we need to learn about the economics of Software Delivery; in this post, we’ll write about Deployment Overhead.

Deployment Overhead

The process of delivering a piece of software from source code to production (whatever that means) involves a number of activities:

• Packaging the application

• Installing the package in the runtime environment

• Testing

• Configuring the application

• Migrating application data

These operations are not, strictly speaking, adding value to the software: you can repeat them as often as you like but you’re not going to get new features in the software unless you actually change the source code. Moreover, these still take time, which you won’t be ablo to use for value added development tasks; for this reason, we call these activities deployment overhead.

Now, imagine that it takes you an hour to do all of these1: if you want to release every day and you have 8 hours of work in a day, that leaves you with only 7 hours to work on code, i.e., your productivity is

In other words, only 87.5% of your time was spent adding value to your product.

However, what happens if, instead of doing this every day, you do it every 2 days (16 work hours)? Your productivity is now:

Wow, you just got a 6% gain in productivity! Maybe we’re on to something: by releasing less frequently, we can increase our productivity…

Indeed, this is true; plotting deployment overhead as a function of the release cadence (the inverse of frequency) makes it easy to spot the trend:

This seems to suggest that batching up more changes for each deployment is more efficient and thus makes the most economic sense.

Summary

In this article, we have started to describe the economics of software delivery, starting with Deployment Overhead and how, taken in isolation, it suggests that it’s best to release infrequently, in order to be more efficient.

However, deployment overhead is not the only variable that influences the economics of software delivery; as we’ll see in upcoming articles, other concerns will come into play that will push us into shorter deployment cadences.

Engineering, being knowledge work, requires large chunks of distraction-free time, in order to tackle complex problems; on the other hand, it’s also a team sport, and thus requires intense collaboration to build any system of non-trivial scale.

Balancing both of these needs is hard but, in our experience, can be done effectively by resorting to asynchronous communication media.

What is asynchronous communication?

When we say “communication”, we usually mean “conversation”: two (or more) people speaking in turns, exchanging ideas.

In this traditional means of communication, all parties are actively involved at the same time (i.e. synchronized), effectively blocked from doing anything else besides speaking or listening1. Examples of this are talking (whether in-person or on a phone) and chat systems (e.g. Slack).

For communication to be “asynchronous”, i.e., to work in the absence of significant synchronization, it is optional that everyone is available at the same time and there is an understanding that a reply probably won’t arrive immediately.

An example of asynchronous communication media that is easy to understand is conventional mail (you don’t need to be at home to receive your mail), but plain email is asynchronous as well.

For software development specifically, the issue tracker (Gitlab, GitHub, JIRA) is also an asynchronous communication media, as are most other tools that support messaging/commenting.

Why is asynchronous communication better?

The first thing to note here is that asynchronous communication is not better in every context: rather, it is usually better in the specific context of software engineering.

It produces less interruptions

By its nature, asynchronous media separate the production of the message from its consumption: in other words, even if someone sends you a letter every day, you’re free to only check your mail once a week.

Since, as mentioned above, effective engineering work requires focus, this allows engineers to focus on their work without interruptions and only stop every hour or so to check on their messages.

It allows for deeper discussions

Discussions in engineering often require:

• compiling information from various sources

• testing hypotheses

Both of which are awkward to do synchronously because they take time and, meanwhile, the other person would just be standing there “doing nothing”.

With asynchronous messages, the writer can devote themselves to writing something that is well structured and easy to “digest”, while the reader can take their time to make sure they understand the points that were made.

How can I use asynchronous messaging better?

An interesting observation is that, while most people will have experience with email, their usage is strongly influenced by synchronous communication patterns and, to a certain extent, defeat its whole point.

Here, we make some recommendations for making better use of asynchronous communication.

Set expectations for response times and availability

This might be the hardest to pull off: in the end, we all know we have little control over other people’s feelings, and it’s easy to imagine “missing” a message, with severe consequences.

In practice, such incidents are very rare and can be handled by providing an emergency contact (e.g. phone call), with the understanding that it’ll only be used for something that requires immediate action.

In most cases, as long as you still reply in a reasonable time (e.g. a couple of hours), most people won’t care too much or notice.

Disable notifications

Even if you’re not reading the messages right away, if you don’t disable notifications, you’ll be distracted by them.

A good strategy here is not to disable notifications completely but to keep them only for “urgent” messages (e.g. explicit mentions).

Pull all messages into a single inbox

This makes it easier to go through all the messages than having to hop between different systems; in practice, this “main” inbox will almost always be email because it’s what almost every tool supports.

This might seem overwhelming at first (especially with the amount of email that certain tools send) but can be made tractable with effective use of filters.

Something else to keep in mind is that not all messages are created equal: some will require a turn-around of only a few hours (e.g. answering a question), while others may not require one for a few days (e.g. a long discussion on a Request For Comments document), and still others might require no answer (e.g. a notification that someone created a Merge Request).

Assign an “on-call” engineer

An option, within a team, is to assign a single engineer to “on-call” duty, meaning that person will field all synchronous requests, in order to shield the team from the distractions.

This assignment can be rotated on a regular basis, so every engineer has the chance to work distraction-free.

Summary

In this article, we’ve described what asynchronous communication is and how it can be beneficial for teams developing software.

Additionally, we’ve provided some suggestions on how to make asynchronous communication work better.

Do you feel that your work environment is too distracting?

Contact Us!

As code complexity rises quite quickly with the size of the code base, it becomes impossible to check for regressions using manual testing procedures and, in practice, any professional software development effort will employ automated testing; automating tests is of such importance that, in many projects, the ratio of production to test code is almost 1:1.

This makes the use of patterns for managing test code important, and one such pattern that we’ve found useful for testing API (Application Programming Interface) based components is implementing a test client.

What is a test client?

A test client is an entire client for your API that is written for the purpose of implementing tests.

In this context, an API is an interface that is accessed over a network (using a protocol such as HTTP); one of the goals of the test client is to abstract that protocol’s details in the same way they would be if you were writing an entire client application.

What are the benefits of writing a test client?

Improved DX (Developer Experience)

The most common complaint heard by someone consuming an API is that it is difficult to use.

This usually happens because:

• Operations/resources have a plethora of arguments/options;

• There are many seemingly identical operations;

• The sequence of operations to perform a higher level action is complex.

By creating a separate code base, you are, in effect, replicating the DX of whoever needs to integrate with your API; this means that any awkwardness in the API design becomes much more obvious/painful to the team designing and implementing the API.

More readable test code

Unfortunately, while most engineers are motivated to produce Clean Code (meaning readable, intent-revealing code) for production, test code is often exempted from this attention to detail, often resulting in tangled, brittle tests.

The following is a too common example of implementing a test:

By implementing a test client, you would instead write something like:

This makes the code much easier to understand and maintain.

What are the disadvantages of implementing a test client?

Duplicated effort

By design, the test client is an abstraction which requires effort to build and maintain; while this effort will usually be small, it needs to be taken into account when planning.

On the other hand, if the effort required to build the test client is significant, this could indicate a complex API design that should be given more attention.

How should I implement my test client?

Avoid code sharing

Ideally, a test client implementation will use a different programming language than the one used to implement the API; for example, if the API is implemented using Java, the client could be implemented in JavaScript or Python.

The benefit of this is that it avoids too much coupling of the API design to the affordances of a particular programming language or stack.

In particular, it makes it almost impossible to share production code with the test client, forcing API designers to face their design decisions; we’ve seen extreme cases in which, by sharing Data Transfer Objects between the application and test code, developers were oblivious to what the JSON payload really looked at the wire level (and yes, it looked nasty).

Use a code generator, if possible

While we find the usefulness of tools like OpenAPI debatable when used strictly for specification or documentation purposes, their ability to generate client stubs from the spec can yield significant productivity gains.

Since the biggest con of implementing a test client is increased development effort, using a code generator greatly alleviates this and makes the pattern much more worthwhile.

Publish the client

If the API is public (i.e. meant to be consumed by external parties), you can publish the test client as an artifact, to make it easier for others to integrate with your API; some examples of this are the API clients for cloud services like Google Cloud Platform or Amazon Web Services.

Even if the API isn’t public and is only consumed by a single application (usually a Web Frontend), it’s still possible (and beneficial) to publish the client, so it can be used not only by the tests but also the actual client.

Summary

In this article, we’ve described the test client pattern; this pattern makes the friction in DX more evident to API designers, thus incentivizing them to improve the design.

Is this pattern one that would make sense for you? Let us know in the comments!

One of the crucial activities that support planning is that of estimation; while this is a contentious view, in this article, we explain how estimation, when done well enough, can make the difference in the planning of software development.

What is estimation?

In software development, the term “estimation” is commonly used to describe the activity of predicting how long a given task, feature or even entire project will take to complete.

Estimating is then, at its heart, assigning a cost to some action which, in the context of planning, allows that action to be compared with the alternatives1.

Why are estimates useful?

In our experience, estimates are most helpful when combined with the benefit the team expects to derive from completing a particular piece of work.

By plotting a unit of work’s estimated cost against its estimated benefit, we can place it in an Action Priority Matrix2:

1. Fill Ins, which can be completed quickly but don’t add a lot of value;

2. Quick Wins, work that has low-cost but high benefit;

3. Thankless Tasks that, despite their high cost to complete, don’t actually achieve much;

4. Big Bets involving a lot of effort but also with a large payoff.

By classifying tasks into these four categories, it becomes evident that a team will want to spend most of its time working on Quick Wins and avoiding Thankless Tasks3, since these will generate (in theory) the most value in the least amount of time; thus, the use of estimates provides a basis for prioritizing work.

Summary

While the topic of estimation in software development can be a contentious one, we’ve described one way in which it can be used to support planning activities; by analyzing a task’s benefit and cost, it becomes easier to recognize in which category of the Action Priority Matrix it belongs and thus prioritize quickly, even with large backlogs.

This arose because, historically, these were segregated roles, with those in Development handing over their work to Operations; however, this segregation lead to applications that were built without taking into consideration how they would actually be run in production, leading to all sorts of problems:

• applications that needed thousands of configuration settings to run;

• applications that did not scale, i.e., could not handle increases in volume;

• applications that fail in mysterious ways.

Silent machines

A recurring theme that we see with teams that deploy software is how silent it is, in a bad way: the application could be running for weeks until someone (often a user) reports that something isn’t quite right.

Why does this happen?

The simple reason is these applications don’t provide operators with even simple things like:

• logs, textual output describing what the application is doing;

• metrics, usually numeric data that measure different aspects of the application’s operation.

In other cases, this information does exist, but needs to be proactively looked at… which means that:

1. it usually isn’t;

2. if the app is expected to run 24/7, you need someone to check in around the clock (this is a good way to burn out your engineers).

Don’t call us: we’ll call you

What you want is to have your software make some noise when it’s sick, so you can heal it. While conceptually simple, this can (and often does) require a lot of effort to build in and coordinate between several pieces.

Software architecture

At a minimum, any modern toolchain that you build software on needs to perform logging. This is so errors can be identified and operators notified; if this doesn’t exist, you should look into a different toolchain.

Then, your application needs to be architected in a way that ensures that, by default, all errors are handled centrally; what this means is that, if an error occurs, you guarantee that something specific (usually logging or reporting to an external service like Sentry) happens. If it becomes possible for an error to be silently ignored, you’re relying on your customers to (angrily) report them to you.

Your platform

Even if you’ve carefully built logging into your application and instrumented it to collect metrics, you need to push this data into a place where you can look at it and setup some rules that’ll notify you when something seems suspicious. This is where your choice of platform (the place where your software runs) can make things easier or harder:

• a good platform can do most of the work for you; all you have to do is deploy your app and everything will appear in its place, as if by magic;

• a bad platform won’t give you this (or it might charge you obscenely for it), leaving you with no option but to run more software in order to make up for it.

Conclusion

While application development tools have come a long way, they cannot, by themselves, replace proper design and implementation: even when the best tools are available, software requires the employment of certain architectural patterns to ensure that it performs its intended business duties and, most importantly, that we know when it’s failing and have the data to learn why.

In this article, we’ve outlined a few of these techniques, which are part of our assessment of client maturity. Do you feel that this is an area that might need improvement? If so, get in touch.

The OODA loop

Many in business will be familiar with the OODA loop; in summary, it is a model of decision making with 4 steps:

1. Observe, where one gathers information on their environment;

2. Orient, where one uses the information to understand their position in the environment;

3. Decide, where one formulates a hypothesis of what action will lead to a favorable outcome;

4. Act, where one performs the actions planned in the previous step.

A central idea is that, in any enterprise, decision makers go through the loop multiple times and, with each round, learn something about the environment that informs later hypotheses and actions. Therefore, there is a competitive advantage to gain by looping faster because, on the one hand, one gathers more information faster while, on the other hand, one’s actions can disrupt a competitor by changing the landscape on which they based their decisions.

Orient

The work that we do focuses on the Orient stage; the reason is that, in our experience, while many engineering companies are observing the environment, they often fail to take into account their position within it, which leads to plans that ultimately fail to produce their expected outcome.

As an example, it’s very common to see small or midsized companies adopting practices from much larger (and publicly visible) companies without understanding that those practices are only needed or worthwhile beyond a certain scale.

We believe we can change this by providing insight into where your company is right now: after all, you can only get to where you want to be if you know where you’re starting from. And not only do we want to tell you what your strengths and weaknesses are, we also want to give you a boost on your path to improvement by suggesting what you could do; ultimately, we want your team to succeed by itself.

Summary

At keeoon, we want to turn software into something that is, on the one hand, a reliable foundation for many business models and, on the other hand, a safe and satisfying career option for the people that build it.

While software engineering is a relatively young and fast moving field, there is already a body of knowledge that makes it possible to assess a software development team or company’s maturity, and point to resources that will help them improve and succeed.

Do you feel like we could help you? If so, get in touch.

In this post, we aim to describe why the company came into being; we’ll start with a couple of stories.

A startup founder’s story

Imagine you’re a startup founder: you have a business idea that you’re trying to validate.

Your idea needs some software, though, and you don’t have the expertise to build it, so you hire a team to build it for you.

You tell them what you want, work with them for a few months… and you have nothing to show for it; or maybe you have something, but it’s definitely not what you had in mind: the interface is hideous and clunky, and trying to do anything is an exercise in frustration because it takes so long… before failing.

You’ve spent most of your money away and, at this point, it’s very likely that your business is going to fail; there are customers interested in your solution but they don’t want to pay for something that gives them headaches.

A struggling company

In another company, in business for over a decade, there is a growing sense of unrest: their once well established position in the market is increasingly under attack from new entrants and, so far, they’ve been unable to innovate quickly enough to fend off the threat.

The engineering team is hard at work but, for some reason, the results just don’t seem to show up; in fact, productivity is going down because some of the most senior engineers have left due to being unhappy with the how the company works.

A better way?

We’ve seen the scenarios above play out too often and we want to do something about it. We want a world where businesses of all kinds can rely on software development teams to deliver, while promoting the right environment for engineers to be productive.

Our journey is just getting started though; if you’re excited by what you’ve just read, get in touch.