Deadline Driven Development
— Chapter 5 —
Author’s Note: This chapter is now freely available to the public. All paywalls have been removed.
Deadlines seem to be an inevitable way of life for software engineers. When you’re working on a project, there are always other people invested in that project along with you, such as managers, PMs, tech leads, and the customers. These people want to know when you’re going to be done with your project. That is reasonable, to a degree.
I am not making the claim here that there should be no deadlines whatsoever. Deadlines can be necessary to handle the expectations of your stakeholders: your clients, managers, tech leads, etc. Deadlines can also be a good motivator for the team: When they see that the end is in sight, and how close they are to finishing the project, that can give the team a great sense of accomplishment and motivation.
But all this assumes that the deadlines are reasonable, and are set by the team of engineers themselves, not by the managers or the sales folks or anyone else. Unfortunately, this is not always the case.
The real issues show up when the managers, PMs, marketing folk, etc. don’t like your estimations and start pushing for an earlier deadline. Or when they expect you to give them a certain exact date, and expect you to deliver on that date without any leeway. Or when they expect you to work 50+ hours a week so you can deliver on their arbitrarily arranged deadlines. Or when they start micromanaging you and do the equivalent of asking you “are you done yet?” every day.
This is a special kind of hell. This is the hellish world of Deadline Driven Development.
Let’s imagine this purely hypothetical company. In this company, the engineers determine their objectives that they want to achieve on a quarterly basis. These quarterly objectives would emerge from their discussions with the managers, PMs, business people, tech leads, and all other relevant parties. There is an implicit understanding that not everything could be delivered on time, that there could be unexpected surprises that inevitably show up in any sort of software development project. At the end of the quarter, if the engineers have succeeded in completing 70% of their tasks, that would be considered an achievement. The remaining tasks would be rolled over to the next quarter.
These are the good times. Obviously, since this particular story is part of a cautionary tale, the good times wouldn’t last.
After some time, the company eventually starts going into the bloat phase. More managers and PMs are hired over the years in unnecessarily huge numbers. Some of these folks want to justify the existence of their jobs. So they start to implement unnecessary processes that would supposedly extract more work from their employees. The deadlines start to get tracked in a much finer grained fashion. Engineers are expected to state the exact number of days it would take them to complete each of their tasks. A Project Manager might ask an engineer how many days it would take you to finish a particular task. If the engineer answers something like 2 weeks, the PM might then ask the engineer whether they could actually complete it in 2 days.
A Project Manager’s primary duty should be to untangle the web of dependencies between the different teams and groups, coordinating between them, and enabling the engineers to proceed with their work. It should not be asking the engineer “are you done yet?” or “how about doing this faster?” all the time incessantly.
I went over all these issues of company bloat and unnecessary processes in great detail in the previous chapters. I already mentioned how micromanagement can cause demoralization in organizations, with calamitous results. I won’t go over the same stuff again. I will instead try to focus on the other detrimental effects of the deadline driven development processes.
The Race to the Tech Debt
The truth is, it is not just the management; everybody wants the project to be delivered fast, including the engineers themselves. Most engineers genuinely want to see their projects succeed within a reasonable timeline, with satisfying results.
And some engineers really focus on the fact that the faster the project is delivered, the faster they can get their promotions and salary raises.
To achieve this purpose, some engineers go ahead and tell their managers what they want to hear. “Oh, you want this project to be complete in 2 months? Sure, absolutely we can do it.”
Sometimes they even seem to complete the project in that short time that was promised. But then again, is the project really complete? “Complete” is a tricky term to define after all.
When trying to deliver fast, the engineers might take shortcuts and add tech debt to the codebase. Proper frameworks for automated testing may not be developed. Classes and functions could be too long, full of long argument lists, and unintelligible. Bugs might have slipped through unintentionally. When you’re developing in a hard-to-understand code base in a hurry, it is easy to make mistakes after all. And sometimes the mistakes are related to the design itself, not even the code. Maybe when designing the software in a hurry, nobody thought of handling certain user scenarios or certain error cases that arise from interacting with external services. The software design errors are even more costly than the simple bugs in the code. They are usually more difficult to debug, and more difficult to fix in the long term.
The real troubles start when this buggy and untested product is released to the customers. Servers start crashing, usually at highly inconvenient times like 3 am on a Saturday. Engineers spend hours and sometimes days going through the logs, trying to replicate the bugs in their development environment, and attempting to find the root causes of the issues. Long postmortem reports are written documenting what happened and how it could be prevented from happening again, which are promptly ignored, because the engineering team is now busy fighting other fires that have sprung up elsewhere in the system.
If only the team took a little more time and care at the very beginning to create a higher quality software design and implementation, none of this excessive time would be wasted later on. In the long term, time to market would be a lot faster. The push to deliver faster usually ends up having the opposite effect of slowing the development down. In the long term, tech debt makes it more difficult to change the code and add new features. It causes the engineers to get bogged down fighting with never-ending bugs, instead of spending their time developing new features.
The race to meet short deadlines is a race to the tech debt. Quality takes time and care.
As an old saying goes: Slow is smooth, and smooth is fast.
Squeezing Work Out of an Engineer
Some managers really love to extract more work out of their already hard working engineers in any way they can. The more work they can extract, the more successful they themselves would seem to their own higher-ups, who might be as clueless as they are. But sometimes they get too greedy. In the process of squeezing more work out of their reports, those managers end up disrupting the engineers’ work and cause even further delays in the projects.
In order to understand how exactly this ends up happening, one must first understand the mind of an engineer at work. After all, software engineering is the labor of the human mind.
There is never a time when an engineer comes to work, sits down, starts coding, and keeps on coding non-stop until the end of the workday. This just simply doesn’t happen.
An engineer typically spends some amount of time each day checking emails, communicating with teammates, attending meetings, and taking some much needed breaks, besides designing software and/or coding. This is just the way of life for an engineer.
Ideally, the actual work, whether it is coding or designing software, happens in chunks of 2 or 3 contiguous hours. If an engineer could find such chunks of uninterrupted time in a day, that would be the ideal work scenario. Then the engineer can go into a state of energized focus, fully concentrated and immersed in their work. This is when an engineer is most productive. There is a term for this state of mind. Psychologists call it the “flow state”.1
An engineer would be very lucky to go into this flow state twice in a day. So that would be two of those 2-3 hour contiguous chunks. One cannot stay in flow-state for more than 3 hours unfortunately, and on some days not even that long. Still, it’s possible to take a break, move around, respond to some emails, and then go back into another iteration of the flow state. So the peak amount of hardcore coding an engineer could do would be around 4-6 hours per day. That is 4-6 hours of hyper-focused state of mind.
This is also a rare scenario.
Most of the time, an engineer can hardly find 2 hours of contiguous time to work on their software, due to meetings or other endless interruptions.
Disruptive Meetings
This is one of the primary ways that some managers disrupt the productivity of their engineers: By assigning unnecessary meetings dispersed throughout the day. This disrupts the engineers’ focus, making it difficult to concentrate on their development tasks. Their productivity inevitably suffers. The amount of bugs in their code goes up.
Ironically, some of those meetings are those that are set up to track the progress of the engineers and try to push them to develop faster. There might be directors who assign daily meetings for the projects that were not delivered in time, asking the engineers to give them daily reports and presentations. Needless to say, those are the prime examples of the very clueless management that I mentioned elsewhere in this book. And none of those meetings could achieve their intended effect and would end up slowing down all progress even more.
Many engineers try to get their work done on their laptops while sitting in large meetings. Even if it is a meeting where they seldom have to talk, it still seems to be a difficult environment which disrupts their focus. I don’t think anyone can produce any quality work in such an environment. I believe that large meetings with vague agendas should be particularly wiped clean off from everyone’s calendars. If only a couple of people are talking in a meeting and the rest are not participating in any way, then that meeting is a waste of time for the rest. Sharing what was talked about in the meeting and emailing others a summary would be a more beneficial strategy for everyone concerned.
Some meetings are inevitable. When you’re an engineer working on a project that involves multiple cross-functional teams, it is absolutely necessary to meet and coordinate with the other team members. You’re also going to have meetings with your own team members separately, to discuss certain issues privately within your team. It is normal and expected for an engineer to have some meetings throughout the week.
In such cases, the meetings should be ideally concentrated in one contiguous chunk in a day, leaving other contiguous time chunks dedicated to actual work and breaks for the engineers. Let’s say, have all your meetings on Tuesday and Thursday mornings, leaving the rest of your days (and your week) free of meetings. You can all decide on this as a team. Some people are more productive in the mornings, and others are more productive in the afternoons. You should all take a vote and decide when you want to have your meetings as a team, and hope to settle on similar time slots with the other cross-functional teams. Keep in account that the more people are invited to a meeting, the harder it will be to find a common time slot, especially for repeating meetings. If you want to have large meetings with lots of people on a weekly basis, then best of luck to you.
Please keep in mind that any meeting longer than an hour is usually pointless. Day-long meetings are an abomination.
Open Offices
Speaking of disruptions to the engineers’ focus, this is one of the subjects I need to briefly talk about: Open Offices. While they are not directly related to the issue of deadlines, open-space offices are a major source of disruption for the engineers.
Open offices happen when the companies forgo the more private work spaces of offices and cubicles, and instead dedicate a hangar-sized floor of the building to be the work environment. The employees’ desks are all together. If they are lucky, they get very short separators between the desks, usually not longer than one or two feet tall. If they are unlucky, they are cram packed into a building’s floor like sardines in a can, with desks all around them and nothing to separate them at all.
For some reason, some companies love the idea of open offices. They claim it increases collaboration between the team members, enabling them to communicate with each other more freely. At least that is the marketing pitch they make to their employees. The real reason why they love open offices is because it helps them decrease their costs as they don’t have to spend money on cubicles or separators. Also, an office building with an open office space is cheaper in real estate costs than a building with actual office rooms, as the companies can jam-pack more employees in an open office than in separate rooms.
While open offices decrease the real estate/rental costs of corporations, they wreak havoc on the focus and concentration of their employees. It is very hard to focus on your work when a bunch of people are talking together a couple of desks down. Sometimes you have to listen to heated discussions about the design of a component that you are not remotely involved with, and sometimes you get to hear loud laughter about the latest episode of someone’s favorite TV series. A lot of people try to work with noise canceling headphones playing music all day long. However, I have personally found that wearing headphones all day long gets uncomfortable after a point.
The Limits of a Human Mind
The flow state can actually be quite addicting. Focusing on a programming task and seeing it to completion can bring quite a bit of joy and fulfillment to an engineer. It makes one want to do and achieve more.
Yet, the human mind has its limits. I believe that at their peak, an engineer can do 4 to 6 hours of solid programming per day. On some days this will be a lot less than 4 to 6 hours, due to meetings and other interruptions. If they try to push their mind beyond those limits, they will most likely start to produce garbage code, where they will have to spend many hours the next day trying to correct the mistakes of the previous days.
There are those who claim to work for 10 or 12 hours a day. But how many of those hours are really spent on pure quality coding? And how many of them are actually spent on taking a lengthy coffee or lunch break, going to the company gym for hours, or having conversations unrelated to work with their coworkers for hours in the middle of the day? Sure, it is completely fine for team members to have non-work related conversations from time to time. Such conversations even help strengthen the team bonds. And sure, there are other tasks that engineers need to do besides pure coding, such as writing emails and meeting with others. Those tasks are very important too. However, too much of any type of work would also burn out one’s mind, regardless of how little focus they require. I believe that anyone who does real hardcore work for 10 to 12 hours a day on a constant basis would completely burn out after a couple of months, and turn into a shell of their former selves. Burnout is a very real and a very serious issue in our field.
Surely, I have known plenty of engineers addicted to caffeine. I know individuals who have to drink 4-5 cups of coffee to get through the day. Personally, I found that excessive caffeine does nothing to improve my productivity, and in the long term actually damages it. If I drink too many caffeinated beverages, I start getting insomnia. And lack of sleep has much more detrimental effects to one’s productivity than any amount of caffeine can compensate for.
Long before the field of software engineering even existed, our society deemed that 40 hours of work per week is optimal. (8 hours a day, 5 times a week.) There were long struggles of labor strikes and worker uprisings. There was even blood spilled.2 In the end, we all agreed to work for 40 hours a week.
Let’s just stick with that.
There are some managers and executives who try to squeeze more out of their employees under the guise of “crunch time” or “we’re behind schedule”. In such cases, it is almost always the management’s fault for not planning for the project properly, not accounting for the unknowns & surprise blockers, and making too many false promises to the project’s stakeholders. Engineers should not reward such bad behavior by working too hard and burning themselves out. In the end, wellbeing and health should always come first.
There are also cases where the company servers keep going down frequently at inconvenient hours (as already mentioned) and the management expects the engineers to work at those odd hours to resolve these issues, usually by making statements like “This is an emergency, we need to take care of it ASAP!” If this keeps on happening, it is a sign that the software might be full of technical debt. In this case, the team should stop working on any new feature development, and start working on refactoring the code, redesigning & rebuilding the software as necessary, and paying down the technical debt. Most of the team’s quarterly objectives should be dedicated to paying down the technical debt until the server issues are resolved.
Throwing More People at a Project
Some managers and executives eventually come to the realization that they can only squeeze so much work from an already hard working software engineer. Then they might come up with another “solution”: Just hire more software engineers. Sure, that is going to be costly; however they believe that the more engineers they can assign to work on a project, the faster that project is going to finish, and the more successful as managers they would seem.
This also comes with an additional benefit for the manager: The more people are reporting to a manager (whether directly or indirectly through another sub-manager), the more clout and influence that manager has in the company. Therefore, the hiring commences. They hire more software engineers, and the other folk who can supposedly assist the software engineers in their jobs: Product managers, project managers, scrum masters, Agile coaches, etc.
This hiring certainly has one undeniable effect: It increases the company bloat. However, does it really help to finish the project any faster? When you throw more people at a project, does it really speed things up?
In a lot of cases, the answer is actually a big fat no.
Software projects usually have an optimum number of developers that can be assigned to it. This is determined by the architectural composition of the system under development. If the system can be decomposed into subsystems or components with clearly defined interfaces that can be developed independently, then different engineers can be assigned to develop those components or subsystems. In this case, it is indeed beneficial to hire more software engineers to develop these components.
But after some point, you run into diminishing returns. Some subsystems or components do not benefit from multiple people working on it. In this case, just coordinating these engineers to design and develop the software becomes a burdensome task. Within such components, there are usually lots of interdependent pieces: Classes calling other classes, functions calling other functions, etc. Engineers that are assigned to develop the small pieces of the same component need to communicate with each other to determine how their assigned pieces will function together. This means more time spent on meetings. If only a single engineer was working on the component, that engineer would spend the entire time doing software development, instead of meeting and coordinating with another engineer.
Thus, if you assign two engineers on a task, usually the task does not finish in half the time. In some cases the task takes even longer to finish than what it would have taken to finish with only one engineer, because of the coordination and communication overhead.
As a side note, this issue is covered in a lot more depth in the book The Mythical Man-Month by Fred Brooks. The book provides this interesting analogy: “Just because a woman can give birth in 9 months, that doesn’t mean 9 women can give birth in one month.”3
At the root of this problem are some organizations that see engineers as replaceable cogs. Engineering is the product of the labor of the human mind. A lot of engineering tasks require intelligence, creativity, and analytical thinking. Large teams or committees of engineers don’t work for every problem. A lot of times, a small, focused, and dedicated team of engineers is the correct answer.
No Silver Bullets
As I argued in this section, management can throw only so many engineers at a project. And from each engineer, they can only squeeze so much work. So how can the projects be completed any faster?
Here is the true answer to this question no one in this industry wants to hear: No, the projects cannot be completed any faster, especially if you want a quality end product.
Rushing a “meh” product to take over the market share might sound like a good strategy in the short term, but you will most probably end up losing your market share to a better produced quality product from a rival company in the long term. And your reputation will take a permanent dive along the way.
There are no silver bullets in software engineering. A project is going to take however long it’s meant to take from a software development perspective. The best thing managers can do is make sure the engineers have all the resources they need to get their work done, try to remove the roadblocks that engineers occasionally face, and try not to get in their way while they are busy doing their work. Micromanaging the engineers with unnecessary development processes or useless meetings is only going to slow their work down further.
Management should also take into account that in software development there are many unknowns. There are many surprises that come up and derail the schedule. During the course of the development, engineers may realize that there are more changes to be made to the software implementation or design. This is the way of life for software development. As one saying goes: “No battle plan survives contact with the enemy”. Management needs to take this into account and realize that none of the deadlines are written in stone. These deadlines, particularly for the large-scale projects, are really rough estimates. They might be thrown off course by months or even quarters. This could happen in any project, however well-planned it might be.
Managers micromanaging their engineers or pushing them hard is not going to make the projects go any faster. All that is really going to do is demoralize the talented engineers, and drive them away from the team and from the organization.
Engineers Should Set The Deadlines
Engineers themselves should be the ones setting the deadlines, not managers, PMs, or anyone else. Engineers are the ones who are doing the work. They are the ones closest to the project. Therefore, they are the ones in the best position to make an estimate.
As they say, with great power comes great responsibility.
Engineers must be as honest and transparent as possible when dealing with the project stakeholders. Engineers must be able to tell the stakeholders the truth about the course of the project, not what the stakeholders want to hear. This means every engineer needs to have the necessary integrity and backbone, as well as the necessary communication skills. Without these, a person has no business being an engineer.
If an engineer becomes aware that a deadline is going to be missed, they need to communicate with all the stakeholders and clients as soon as possible. They need to explain the tradeoffs to the client as transparently as possible. There is no client on earth who would like to have a poorly-functioning product rather than waiting a bit longer to receive a working, well-functioning, and properly tested version. No one wants a car that crashes occasionally or a home security system that gets hacked easily. Clients might be impatient and demanding; however, it is the PM's job, or whoever is facing the clients, to reason with them and handle their expectations.
Some tasks are easier to estimate than others. Notably, if the team is dealing with a new technology or trying to do something that very few have done before, it would be harder to estimate that task. One can still try to estimate, but the uncertainty will be higher. In these scenarios, the engineers need to do their best to make sure all the stakeholders understand this uncertainty.
The Constant Battle Against Tech Debt
It is not only the unexpected surprises in software development that should be taken into account when trying to arrange project schedules and determine the deadlines.
The project schedules must also address tech debt. Because tech debt happens in a never-ending fashion. It is pretty much inevitable.
This is due to the nature of software development.
When doing software development, no one can foresee which direction the software design and implementation is going to head. A class or component which you thought was going to be not-so-important might end up having to take on more responsibility and grow in size & scope. A piece of software logic which you thought could be implemented with a few if-else statements might turn out to be more complicated due to the evolving business needs. Now it needs to be refactored and decomposed into several methods or even into several classes in order to be more open to further changes.
As you’re implementing your main features, you will frequently notice that certain portions of your code need to be refactored, and certain components need to be redesigned. These are the seeds of the tech debt. Just like credit card debt in real life, it starts small. If you keep ignoring it, in a very small amount of time it accumulates. Just like credit card debt, the tech debt accumulates with interest. In the end, the cost you pay is going to be the maintainability and the changeability of your software. It will be more difficult to add new features to your software, and you will spend most of your time debugging issues instead of doing feature development.
No matter how well designed and implemented your software is, tech debt will happen. As you develop your software, tech debt will happen on a constant ongoing basis. Therefore, it needs to be addressed constantly.
A portion of the development schedule and the quarterly objectives should be dedicated to addressing tech debt. A substantial portion.
In well designed and well managed projects, I would argue that at least 20% - 25% of the development schedule should be dedicated to addressing tech debt continuously. In the projects that are showing clear signs of tech debt, this number needs to be a lot higher. If there are frequent server failures that are plaguing the developers every couple of days or even weeks, this number needs to be very close to 100%. In such occasions, all of the efforts of the software development team needs to be dedicated to redesigning and rebuilding the software to pay down the tech debt, as many months or quarters as necessary. Once the tech debt is substantially decreased to a reasonable amount, then the team can carry on with the regular feature development.
This is the correct way of long term thinking. Everyone needs to think for the long term, not the short term. This is better for the overall health of the entire organization and its reputation.
I believe that following this strategy is going to make the overall feature development velocity even faster in the long term. Software engineers are going to deliver properly functioning features in a codebase that is well taken care of consistently.
Teams that decide to sacrifice the long term for the short term gains, as a result, inevitably end up with neither.
Csikszentmihalyi, Mihaly. Flow: The Psychology of Optimal Experience. HarperCollins, 2008.
“Eight-hour day.” Wikipedia, https://en.wikipedia.org/wiki/Eight-hour_day. Accessed 24 October 2023.
Brooks, Frederick P. The Mythical Man-Month: Essays on Software Engineering. Addison-Wesley Publishing Company, 1975.