Darren Hagman
Darren is a veteran Scrum master with experience in Waterfall and Agile across a number of industries.
Have you ever wanted to have all of the Software Development Methodologies explained in one place? If so, here is an ultimate blueprint about what is Lean, Agile, Scrum and Kanban. This is the first part of the series where we explain all the different methodologies used in Software Development.
Have you ever wanted to have all of the Software Development Methodologies explained in one place? If so, here is an ultimate blueprint about what is Lean, Agile, Scrum and Kanban. This is the first part of the series where we explain all the different methodologies used in Software Development.
Darren is a veteran Scrum master with experience in Waterfall and Agile across a number of industries.
Many methodologies are used in software development today. You may have heard buzzwords such as Waterfall, Agile, Scrum, Kanban, Lean, Extreme Programming, etc.
In this article, I will define these terms, discuss how they are related to each other and how they differ from each other. Many of the aforementioned buzzwords are based on concepts from Lean Manufacturing which was originally based on the Toyota Manufacturing System (TPS) so we will talk about this first.
The term “Lean” has its origins in manufacturing where it was coined to describe a manufacturing model based on the Toyota Production System (TPS) originally developed by Sakichi Toyoda, Kiichiro Toyoda, and Taiichi Ohno who were originally inspired by Henry Ford. TPS was focused on the philosophy of “the complete elimination of all waste” and revolutionized manufacturing in the 1950s thru 1970s. TPS become known as “Lean Manufacturing” in 1990 when “The Machine That Changed the World” was published.
TPS identified three broad types of waste at Toyota:
Excess Processing: from poor tooling or product design
TPS worked to eliminate waste by applying these two core concepts:
As TPS evolved, these core pillars and values built upon the concepts of Jidoka and JIT and became entrenched:
The diagram below shows how the core concepts and core values relate to each other.
The Toyota Product System and Lean Manufacturing evolved over time and were applied in a number of areas including management.
Lean Management applied the core values of continuous improvement and respect for people and distilled it into a set of five prescriptive Lean principles which would be repeated a number of times to continuously improve and eliminate waste:
These principles and other aspects of Lean management were formalized when Womack & Jones published “Lean Thinking” in 1996.
Lean has since been applied to management, software development, and other fields.
In the 1980s and 1990s, the software development industry was approaching a crisis as projects executed using traditional waterfall methodologies were taking longer and longer. This often resulted in a huge lag between a business need being identified and a software solution being delivered. Sometimes this lag was measured in multiple years, or even in decades in certain industries with specific requirements, such as the aerospace industry.
During these multiyear timeframes, requirements, systems, or even entire businesses changed. Often, projects would be canceled part way through or they would complete their scope, only to find out that what they delivered no longer meets the business needs as identified at the beginning of the project.
The Waterfall methodology rewarded stakeholders for thinking of everything as they were forced into writing a contract even though they probably weren’t sure of what they needed. The Waterfall methodology forced decisions to be made during the requirements or design phase, and these decisions were very hard to change without changing the contract and adding costs to the project. A high percentage of software projects failed completely, or delivered late and over budget, or failed to deliver what was needed.
This general frustration led to various thought leaders trying to improve on Waterfall. Early examples include Rapid Application Development (RAD) which focused on reducing waste by shortcutting the requirements and design phases via rapidly developing a prototype and collaborating with business to further develop the requirement. There was also a move toward iterative development where a small project was completed and features were added in continual iterations. While these methodologies helped, they did not solve the core problems associated with Waterfall.
In the 1990s and early 2000s, several authors published books on applying Lean principles to software development. Dr. Robert Charette published “Lean Software Development” in 1993 and “12 Principles of Lean Software Development” in 2003.
Tom and Mary Poppendieck published “Lean Software Development: An Agile Toolkit” in 2003. This book detailed seven principles of Lean Development, which correlates directly to the seven forms of waste in Lean Manufacturing. The similarities and differences between the two different Lean publications and Agile (discussed in the next section) are laid out in the diagram below.
According to Dr. Charette, one of the primary differences between Lean and Agile is that Agile is bottom up, while Lean is top down.
Charette's Lean Software Development | The Agile Manifesto | Poppendieck's Lean |
---|---|---|
|
|
Charette's Lean Software Development | The Agile Manifesto | Poppendieck's Lean |
---|---|---|
|
|
|
Around the same time that Charette and the Poppendiecks published their books, the Agile Framework was created to help solve the same problems. In February 2001, a group of Agile pioneers met at the infamous Snowbird meeting in Snowbird, Utah to try and come up with a solution.
The result was the Agile Manifesto which laid out a set of values and principles for a methodology that attempts to adapt to changing requirements and customer needs, cut waste, and deliver benefits faster using an incremental, iterative approach.
The Agile Manifesto reads as follows:
“We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
That is, while there is value in the items on the right, we value the items on the left more.”
Aligned with the values in the manifesto are the 12 principles behind the Agile Manifesto:
“We follow these principles:
The above values and principles are applications of Lean principles such as Jidoka, JIT, Genchi Genbutsu, Kaizen, Hansei, Heijunka, and reducing waste.
Agile is an umbrella framework that applies to any process that applies the Agile set of values and principles.
Some examples are:
Scrum is a framework that applies Agile principles that was invented separately by multiple people, several of whom signed the Agile Manifesto:
Schwaber and Sutherland collaborated throughout the 1990s to develop and refine the framework in a software development context, speaking at various conferences. Schwaber worked with Mike Beedle to describe the method in the book, “Agile Software Development with Scrum” in 2001.
Schwaber went on to create both of the main Scrum certification authorities:
Over time, several frameworks/certification bodies were created to address scaling of the Scrum framework to larger teams and projects as Scrum was originally designed for small teams (7 plus or minus 2 members):
According to the Scrum Alliance:
Scrum is a simple yet incredibly powerful set of principles and practices that help teams deliver products in short cycles, enabling fast feedback, continual improvement, and rapid adaptation to change.
Scrum is a prescriptive, incremental and iterative framework for developing software that applies Agile principles. The Scrum values and principles are outlined in the charts below and have significant alignment with Lean and Agile values and principles.
Work is divided up into short time-boxed iterations called sprints which are usually 1-3 weeks. This is in stark contrast to the in-depth planning of Waterfall. The work planned for the current sprint is chosen from the top of a prioritized backlog of work items called a Product Backlog (Pull System, Heijunka), and it is fixed once the sprint starts. The goal is to have working software at the end of each sprint, enabling fast feedback.
At the end of the sprint, the team meets to review the completed work done, how the sprint went, and to plan the next sprint. The sprint length, as well as the sprint rituals and cadence, are fixed for each sprint.
Sprints are executed by cross-functional teams containing all the skills needed to complete the work in the sprint. Daily planning and tracking of progress is done using visual artifacts like the scrum board and sprint burndown charts.
The work in a sprint is pulled from a prioritized backlog. Following these methods allow for changing requirements over time and encourages constant feedback from the end users.
The mind map diagram below outlines some of the main concepts of Scum which will be discussed in the upcoming sections.
Scrum has three roles:
As discussed above, Scrum has a defined flow and a set of rituals and activities. The flow of a sprint is as follows.
Before a sprint starts, the Scrum Master facilitates a meeting with the scrum team and the product owner, called the sprint planning meeting, where the product owner identifies the objectives of the upcoming sprint, and the team then plans their work according to the objectives.
This is usually done with the following activities:
The total amount of work or scope committed to for the sprint is measured in story points.
During the sprint, team members pull work items (user stories, tasks, etc.) from the top of the sprint To Do list to work on. Various team members will work on the various work items or their subtasks. They will update the status of an item when appropriate by moving it from one column to the next (typically To Do > In Progress > Testing > Done or some variation thereof) on the Scrum Board until it is done.
Progress is tracked using a burndown chart which shows the amount of work completed and remaining measured in story points. Remaining story points are shown on the Y axis and the remaining time is shown on the X axis. The burndown chart is updated when stories are done.
On a daily basis, the Scrum Master focuses on:
Daily Standup
At the beginning of each day in the sprint, the Scrum Master facilitates a brief, 15-minute meeting with the scrum team and product owner to plan out the day and review sprint progress. This is a short meeting where everyone is standing in front of the Scrum Board and each person in the meeting answers the following questions in 2 minutes or less, referring to specific items on the sprint board:
This allows everyone to see what everyone is working on, see what progress is being made or not made, and identify impediments and/or help needed.
The Scrum Master facilitates two ceremonies to close out of the sprint before planning the next sprint.
Sprint Demo
At the end of the sprint, the Scrum Master facilitates a sprint demo meeting where each completed story is demonstrated on the working software to the product owner and the rest of the team. The product owner will either accept the story if all the acceptance criteria are met, or they will reject the story. If a story is rejected, the shortfalls are identified and the story is put back onto the product backlog in its priority order to be completed at a later time or not at all. Often, the parts of a story that the product owner doesn’t accept are broken off into a separate story(s) and the original story is closed.
The total number of completed story points per sprint (Velocity) is calculated and the sprint is closed. The velocity is used to track the team’s output level and is used to estimate when features or releases will be complete.
Sprint Retrospective
After the sprint demo but before the next sprint planning meeting, the Scrum Master facilitates a sprint retrospective where the team reflects on the sprint that just completed and discusses what went well and what didn’t go well so that they can continuously and incrementally improve process and quality over time (Kaizen, Hansei). There are a plethora of retrospective formats or exercises that can be used to help the team generate discussion.
A list of improvement action items is produced and sometimes they result in items being added to the product backlog, changes to the DoD or team charter, etc.
Product Backlog Creation
Before a sprint can be planned or executed, the product owner needs to create a product backlog of work. The backlog usually starts with feature development items called stories written by the product owner and over time also becomes populated with development or QA tasks, spikes, and defects, etc. potentially created by any member of the team. All the items in the backlog are arranged in priority order.
Backlog Grooming
Once the initial product backlog is created and prioritized, the ongoing backlog groom process takes over. The goal is to always have, at a minimum, enough of the top items in the backlog groomed and estimated so that they are ready to be pulled into a sprint during a sprint planning meeting. This is typically done by having regular ongoing backlog grooming meetings with the product manager and the team facilitated by the Scrum Master.
Prior to the meeting, a list of stories is sent to the team so they can review and prepare for the grooming meeting. At the grooming meeting, each item is discussed in terms of the acceptance criteria, complexity, risk, size, implementation strategy, etc. The acceptance criteria and other details of the story are reviewed and revised until the team, product owner, and Scrum Master have a common understanding of the story. At that point, the story is estimated in story points using a technique called planning poker.
Story Point Estimating
Story points are a unit of effort that uses relative sizing where stories are compared against previous, known, well-understood pieces of work. You are always asking the question “is this story bigger, smaller, or approximately the same size” as some other piece of work.
The Fibonacci scale (1, 2, 3, 5, 8, 13, 21…) is the most commonly used scale where each increment is approximately twice as big as the previous (i.e., a five-point story is more or less twice as big as a three-point story). Sometimes other scales like T-shirt sizes (XS, S, M, L, XL) or even fish (minnow, goldfish, trout, tuna, whale, etc.) are used. Any scale that allows you to compare the size of something relative to another will work.
Story points represent the team’s entire effort to implement a story, including development, testing, design, and other miscellaneous tasks needed to the Definition of Done. The estimate takes into account the amount of work, complexity, and risk. Once the relative size of the story is determined, a size on the scale is assigned as the estimate.
Teams prepare for story point estimating process by first establishing a baseline for estimation by picking a “most medium” sized story that the whole team understands as a reference story. Some additional reference stories that are bigger and smaller are also chosen.
Story point estimating is done during grooming meetings and sometimes during the sprint planning using Planning Poker:
The advantages of story point estimating are:
Release Estimating and Tracking
Story point estimating is also used in a release planning context using the following technique:
The story estimates for all the stories in a release combined with the team’s velocity will allow you to estimate when a release will be complete and track its progress. This is often shown in a release burn-up chart.
The primary advantage of Scrum is the application of Agile values and principles as well as Lean concepts such as Seiri, Jidoka, Just-in-Time, Kaizen, Genchi Genbutsu, Heijunka, Pull System, and Iterations. Applying these principles allow project teams to receive continuous feedback, quickly adapt to changing requirements and uncertainty, reduce waste, increase visibility and transparency, and strive for continuous improvement. By always focusing on the most important items in the product backlog and only working in short iterations that always produce working software, Scrum is more customer focused and allows customers to see what they like (and don’t like) and make changes as needed. The overhead cost in terms of process and management is smaller, thus leading to quicker, cheaper results.
Scrum is a great methodology for projects with requirements that are not clearly known and/or expected to change. This applies to most projects. Scrum is also great for experienced, motivated teams as it allows teams to organize the work themselves and it provides visibility, transparency, and accountability for both progress and problems. All of this will result in teams improving and becoming more productive over time.
Scrum does have some disadvantages and is not the best methodology in some situations:
Kanban is a lighter weight process that applies many of the Lean and Agile values as well as a subset of the Scrum values and principles but there are also some fundamental differences. Kanban focuses on visualization, flow, and limiting work in progress.
Kanban is Japanese for “signboard” or “billboard.” Toyota line-workers used a kanban (i.e., an actual board) to signal extra capacity in various steps in their manufacturing process. In software, this is done with a Kanban board, which is very similar to the Scrum board. There is a prioritized backlog of To Do items and vertical columns for each status that a work item can have. Like Scrum, work items are moved from one status to another; however, in Kanban, the amount of work in progress is strictly limited to a maximum number of items in each status based on team capacity. New work cannot be pulled in until existing work is moved to the next step in the process. In Scrum, work in progress in indirectly limited by controlling the amount of work planned for a sprint.
In Kanban, there are no fixed iterations or sprints, just a constant flow where work items are pulled from one stage to the next. This means that the Kanban board is never reset. It also means that many of the Scrum rituals not used. Kanban teams often have daily standup meetings but they are not prescribed. There are no regularly planned sprint planning meetings, sprint demos or sprint retrospectives, so the process is more lightweight. Some of the activities in those rituals may or may not be performed at an informal level. Continuous improvement is accomplished by tracking and analyzing the flow of items from one state to another and making constant incremental improvements based on what issues are uncovered.
Kanban also doesn’t prescribe the roles that Scrum does. The only role needed is the team member role, however, there is often a project manager that manages the team and the backlog. Often a single Kanban board can serve multiple function based roles and/or teams that only work on items in a certain status. For example, a development team might pull items from To Do to In Progress and move them to Testing, and the test team might test items in Testing and move them to Done.
Many of the backlog management activities for prioritizing and grooming of work items still need to be done to ensure that a given work item is well understood and ready to be worked on, however, estimating the work items is prescribed as optional.
The following table compares Scrum and Agile:
Kanban | Scrum |
---|---|
Continuous Delivery | Timeboxed Sprints |
Less process and overhead | Has prescribed Sprint rituals and roles |
Focuses on completing individual items quickly | Focuses on completing a batch of work quickly |
Measures Cycle Time | Measures Sprint Velocity |
Focuses on efficient flow | Focuses on predictability |
Limits WIP for individual items | Limits WIP at a Sprint level |
Individual work items are pulled | Work is pulled in batches at Sprint Planning |
No prescribed roles | Has prescribed roles (Scrum Master, Product Owner, Team Member) |
Kanban Board can be organized around a single cross-functional team or multiple specialized teams | Scrum Board is organized around a single cross-functional team |
Changes can be made at any time -> more flexible | Changes are only allowed in the Product Backlog. Changes within a sprint are not allowed |
Requires less training | Requires more training |
Good for teams where only incremental improvements are needed | Good for teams where fundamental changes are needed |
In this part, we reviewed a few of the most popular methodologies used for Software Development. By now you should have a good understanding of Lean, Agile, Scrum, and Kanban and their historical roots in Lean Manufacturing and TPS. In the next part of the series, we will continue reviewing and comparing other Software Development methodologies such as Waterfall, JTBD, and SAFe (and other scaling frameworks), as well as hybrid methodologies, so you have all of them conveniently explained in one place.
Agile is an umbrella framework that applies to any process that applies the Agile set of values and principles. Some of the examples are: Extreme Programming, Scrum, and Kanban.
Scrum is a simple yet incredibly powerful set of principles and practices that help teams deliver products in short cycles. Scrum is a framework that applies Agile principles that was invented separately by multiple people, several of whom signed the Agile Manifesto.
Kanban is a lighter weight process that applies many of the Lean and Agile values as well as a subset of the Scrum values and principles but there are also some fundamental differences. Kanban focuses on visualization, flow, and limiting work in progress.
Located in Vancouver, BC, Canada
Member since May 1, 2018
Darren is a veteran Scrum master with experience in Waterfall and Agile across a number of industries.
World-class articles, delivered weekly.
World-class articles, delivered weekly.
Join the Toptal® community.