To track project and product performance, key metrics and measures are required

This article will discuss the “bare minimum” set metrics, product success metrics, and what can be considered an agile measure of success.

Did you ever wonder, worry, or laugh at statements like “You can only measure what you manage”?

You may have been asked “Can you deliver a project on-time, within budget, and within scope?”.

You’ve never seen a company close its doors despite impressive progress made by the software development team in speed, automated testing cover, age and other cutting-edge toolset.

Or are there still companies in business that do at least OK with bad code, outdated frameworks and 99.9% manual coverage?

If you are familiar with any of these, continue reading…

This is a quick review of Scrum.org’s learning paths based on the experience gained in our Agile practice team. It will answer some of the questions and offer recommendations for the efficient use of various metrics, such as agile, project, product or leading.

But, firstly…

What are Agile Metrics?

It’s important to remember that agile metadata is not a useful term. In fact, it can be misleading. All metrics can be agile depending on how they are used. There are some metrics that agile teams use so often that they have a “signature” reputation as “agile metrics”. These include throughput for Kanban and release burndown for Scrum. Personally, I suggest that they be considered a subset, which can be used by any organization or team.

This article focuses on two types of classification: product vs. Project and leading vs. lagging metrics.

Do You Need to Focus on Product or Project Metrics?

Spoiler alert!

Both should be our focus

Offshore service providers can assume that the customer is properly managing the product and its metrics. The vendor team can focus on the “it is supposed” to do: meet the customer’s requirements on time, within budget, and with quality. This assumption can be risky, however, it may not apply to smaller, more well-defined projects that address a business need.

You can do well with relatively simple projects to stay in the project-level metrics lane.

• Almost any case can benefit from budget and scope features.
• It is also important to measure customer satisfaction and team satisfaction in order to achieve both project and product success metrics.
• It would be useful to know the number of defects discovered during development and after release. Sure.
• Team velocity, throughput and item age are all important. Kanban is a set of metrics that helps to minimize waste and maximize value.

This last v-word could be a trap. You want to maximize value and push developers to improve unit test coverage to 100 percent.

For more complex projects, it is important to read “project” as “product” in order to be able to act appropriately. This means that you must live by the agile manifesto principles, and collaborate with business people every day to ensure that the development team’s actions are as aligned with the product’s needs.

We will briefly mention the Product Management Vacuum to show the importance of it and the ways you can fill it.

Product management covers so much more than what is visible to a Scrum Team: all types of marketing research and focus group facilitations, stakeholders interviews and facilitations. Collaborations with UX/UI designers, customer interviews.

There is always the possibility of a gap between the Sprint Backlog and the Company Vision. Scrum is an example of this. However, the principle applies to any project or product. The whole team must be aware and act accordingly for continuous validation that the value being delivered is in line with the vision statement.

It is possible that the customer has already done the Product Management research and only hired you for technical tasks. But it is in your best interest to make sure that the customer’s business is successful, and I recommend you apply agile principles through the Vision-Value-Validation triad which encompasses all of the “short increments-frequent feedback” goodness, as opposed to working off the “classical” assumption that it is possible to plan things the right way upfront, whilst planning a bit of change management just in case 😉

Simply put, ask your customer about their product goals and KPIs. Next, plan and align what is important to the company with what you measure in the work of your team. This will ensure that you optimize the entire product and not just one subsystem. Studies have shown that optimizing a subsystem can lead to problems in the whole system. You should avoid suboptimizations.

Evidence-Based Management Overview

This framework recommends that you base your product success measures on 4 Key Value Areas. Depending on the current product lifecycle (e.g. a promising startup or an old cash cow about to retire), select the most relevant product delivery metrics and adjust the engineering team’s focus. A wealth of examples of product metrics are also included in the Evidence-Based Management guide.

If a product is a high-profitable and has reached its peak marketing potential, it might make more sense to concentrate on the Installed Version Index to create a more efficient and cheaper support and maintenance phase.

On the contrary, it might be crucial to reduce the Time To Market for new products in order to accelerate the conversion of Unrealized Value to Current Value. It is important to give a lot of attention to the ability and potential to innovate.

Engineers may not always see the obvious. Trying to produce state-of the-art software and technical perfection is not always the best way to make money. It is also true that many businessmen don’t realize that startups built quickly and poorly have a higher chance of being “just another startup” as opposed to ones that are built with care and attention to quality and scalability. Managers can help to find the right mix of metrics and measures to bring about some equilibrium and peace between these extremes.

Product and project metrics should be aligned and combined.

Another aspect of measurements: Leading and lagging metrics

You can also view your product and delivery metrics as leading and trailing to ensure they are balanced and meaningful. The former can be viewed as related to inputs or pre-product, while the latter measures outputs/outcomes. The former is easier and more difficult to measure, while the latter is simpler to influence but harder.

This is the illustration from “The Professional Product Owner”

This distinction can be illustrated by trying to lose weight. Lowering the weight is the ultimate goal. The lagging indicator of weight is easy to measure (just step on the scale) but difficult to influence directly (unless you take off a limb). Leading indicators are how much you eat and how often you exercise. Although it is difficult, you do have more control over the leading indicators. Your lagging weight indicator will eventually be affected if you have positive influence on calories taken in and calories burnt. Both are important in product development.

How can you make sure the metrics actually work? Learn more about the Suboptimization Threat

There are still a few things you need to remember about measuring.

You may feel tempted to encourage behavior or practice based upon the metrics. This should be done with caution. It is human nature to try and game any controlling measures. What can be gamed will be played. Experts recommend asking your colleagues for help in devising tricks to fool a metric, before you actually implement it.

The value of measurement will be diminished if a metric is linked to material bonuses. People will work only on achieving the goal and this could lead to suboptimization. Goodhart’s law states that the metrics will improve, but actual performance will fall. The illustration is below.

You should always be ready to change, replace, or review metrics regularly. A metric should not be an end goal. It’s okay for a metric that is no longer relevant to be replaced.

Last but not least, avoid cargo-cutting a measurement: using it without fully understanding its nature. Managers who don’t have enough knowledge about agile frameworks may attempt to compare performance between teams using velocity, a popular “agile metric”. In these circumstances, this would be very misleading.

Only one team can measure velocity. They are the ones who do the planning, estimations and work towards achieving the goals. If your team’s velocity does not grow or fall, this is a sign that your Product Backlog refinement practices need to be improved. As each team is different, it would be wrong and misleading to assume that all teams will “produce” the exact same number of story points. Managers who continue to view velocity as an organizational metric will result in suboptimization that leads to teams trying to manipulate the metric. This could also cause frustration and low morale.

Conclusion

Let’s recap the agile measures that define success.

1. While product and project metrics are important, they can also be difficult to measure.
2. Businesses should adopt “product thinking” rather than “project thinking”, which is a combination of project and product metrics.
3. Suboptimization, cargo-culting and suboptimization are the most frequent problems with metrics.
4. Suboptimization can be avoided by aligning metrics across the entire system and not focusing on sub-systems.
5. It is recommended to combine leading and lagging metrics. Regular reviews, changes, or replacements of metrics are also recommended.
6. It can be very dangerous and should be avoided.
7. Evidence-Based Management is an excellent example of a holistic approach for measuring product development.

It is great fun to give a crash-course on complex subjects, but it is also true that it is difficult to find the right measures of success in real life. It is wise to use the expertise of professionals alongside your own personal development and those of your team.