Beyond Agile
Bonus Chapter: How can we iterate upon the Agile methodology itself, and improve it?
My chapter about Agile has received a positive response from the majority of readers. I am happy to be a voice of what many have already been thinking, but not have been saying out loud.
However, I admit that it was a very long chapter to read. I fear that my suggestions for improvements might be lost among my criticisms of Agile. I do not want to be known as someone who just criticizes without making any suggestions for improvement. Therefore, I wanted to write this relatively shorter post where I summarize how I believe we can improve upon the Agile methodology and develop software in a better way.
This chapter is just about my suggestions for improvement. If you want to know why I am making these suggestions or why I am criticizing some aspects of Agile in the first place, you might like to go back and read the original Agile chapter.
These are all my humble opinions of course. If you disagree, please feel free to comment and make alternative suggestions. I am always open to change my mind when presented with a good argument and some new evidence. Nevertheless, I must add that many times throughout my career, I have observed this methodology being applied quite successfully. There were many amazing products that were built following this way of developing software. Products with great features and very few defects.
I strongly believe that this way of development can help us produce great quality software within very reasonable timelines:
Incremental & Iterative Development.
Try dividing up the tasks as much as possible.
However, know that some tasks will not be divisible from a technical standpoint as much as one would like. Try to be flexible when it comes to this.
A very high-level design doc.
Should just state the overall architectural design, without getting into any nitty gritty details.
Should contain the explanation of the various tradeoffs, and the reason why a certain design was chosen over others.
Should be open to change and alteration (to a certain degree), as the project progresses. (i.e. No major changes late in the project.)
Should roughly state the milestones for delivery. (E.g. Milestone 1: Deliver the MVP on Q2. Milestone 2: Deliver v2 with these extra features on Q3 or Q4.)
Any milestones and their promised dates should be flexible and open to change later on as the project evolves due to unforeseeable hurdles.
A backlog of feature requests (and bug reports) that should be flexible and open to change.
Each feature request should be as incremental as possible.
But do not worry if it turns out that you need to implement something else before you can work on this feature, or spend some time learning a new technology before proceeding further. It happens. That is life. Just try to be as flexible as possible.
Keep the code changes small.
Each code change should be as fine grained as possible.
This makes them much easier to review as well. It is much more feasible to review tens or a few hundreds of lines of code rather than thousands of lines of code.
Each code change must be reviewed by another engineer before being committed to the code base.
Keep the code well tested.
Each production code change must be accompanied by additional relevant integration and unit test code changes.
Ideally, a CI/CD pipeline should automatically test each code change thoroughly before merging it with the code base.
No more fixed 2-week sprints.
This especially applies to teams that expect “a working demo” from their engineers every 2 weeks.
Understand that the fixed 2-week model may not work at all times for all kinds of tasks.
Engineers should just pick the next feature or bug report from the backlog, and start working on it.
Hire competent engineers and just let them do their job without asking them “are you done yet?” all the time. Get the hell out of their way.
No more fine-grained estimations.
There should be no estimations for the incremental feature requests. They will be done when they are done, within a week or two or three. Everyone is doing their best, just learn to trust your engineers. Nurture integrity in your team.
Estimations for rough milestones should be on the scale of quarters (e.g. MVP will be delivered Q2 of next year), and should be very flexible (could end up being delivered in Q3)
There are always unexpected surprises and roadblocks in the world of software development. Estimations should account for them.
I have never seen any fine-grained estimations to consistently and accurately succeed in my professional life. Let’s just accept reality.
A certain percentage of each quarter and each development cycle dedicated to ongoing refactoring and paying down ongoing tech debt.
I would say 20% to 30%. Could go up to 100% for a number of quarters if your project is a legacy project that already contains immense amounts of tech debt.
Tech debt happens. It is inevitable. It happens even on brand new projects being developed for the first time.
Weekly meetings with your team of engineers.
To go over if anyone is blocked with anything.
To discuss any new developments or roadblocks in the project as a team.
To privately discuss within the engineering team about how to address any concerns that the business folks / external stakeholders might have about the project. These private discussions help preserve the autonomy of the engineering team.
There should be no need for daily stand-ups. A stand-up at 8 or 9 am everyday is only there for micromanagement purposes.
Be open to impromptu discussions and meetings if an engineer in the team needs to consult with the others. Just don’t overdo them. Don’t forget that engineers need their uninterrupted blocks of time to get their work done.
Meetings with business folks / external stakeholders every 2 or 3 weeks.
Keeping the external stakeholder meetings separate from (and less frequent than) your engineering team meetings help preserve the autonomy of your engineering team.
Try to address their concerns as a unified engineering team.
Be very honest about the tradeoffs in the engineering design and their impacts on the business concerns.
Do not make any promises that you cannot keep.
Do not promise any certain or inflexible deadlines.
Engineering and business teams should work as equals, for the common good of the company.
No one side should try to dominate the other.
A severe power imbalance would have very unhealthy long term consequences for the entire company.
Avoid doing the following:
Crunch times.
Death marches.
Working on nights and/or weekends to finish the project within a certain arbitrary deadline.
Don’t forget that deadlines should be determined by the engineers themselves, not imposed by others. They should also be flexible, since there is always some inevitable uncertainty in software development.
Keep in mind the following at all times:
Trust is a two-way street.
If the engineers see that their management trusts and believes in them, they will put in the effort to earn that trust in any case.
Conversely, if the engineers realize that their management doesn’t trust them, they will stop making any effort whatsoever.
Micromanagement is a great way to show the engineers that they are not trusted.
Trust is built over a long period of time, and can be destroyed in an instant.
You might say this still looks very much like Agile. You might be right. However, gone are the 2-week sprints (with demos at the end), fine grained estimations, and product owners attending every single team meeting. All of this process could be implemented with just a team of engineers.
A lightweight Incremental/Iterative Development Process. That’s all you need to do to achieve good software quality, in my humble opinion.
Before I leave you, I’d like to point out that my book goes into the criticism of many aspects of modern software development and management other than Agile. Please don’t forget to check out the rest of the chapters and the posts here.
And please consider becoming a paid subscriber. That would go a long way to supporting me to write more content in this Substack. You would also have immediate access to the 65,000+ words of book content that is already written here right away. This is quite an amount to go through which I believe should be very informative, and which I hope you will enjoy reading.
Until next time.