Moving beyond Product Instrumentation to full Process Instrumentation
It’s tempting to take a birds-eye view with software products and assume they all have a kind of universal similarity, varying by which underlying components are available to be used, how they are combined, or turned on or off. The biggest challenges actually arise from the fact that no two products are exactly the same, and with a lot of complexity hidden in the differences.
Yet complexity can be tamed (or at least reasoned with) when you can see what is happening. Software product instrumentation gives us the ability to monitor or measure the level of a product’s performance, diagnose errors, and to be able to write trace information. In order to have meaningful conversations about developing and managing a product, you have to reference specifics: data and insights about the data. You need visibility, traceability, persistence of information throughout the development lifecycle to assess, validate, inform/decide, take action, and evaluate the outcomes of product decisions.
Product Instrumentation is key to one of the distinguishing characteristics of digital products and services: the ability to evolve continuously while in-market and in use. Product Instrumentation is a key component for informing product strategy and management. Instrumentation supports an empirical approach to defining and prioritizing how a product is going to evolve — what features to refine and what features to add. Instrumentation is essential to the product foundation.
But the value of Instrumentation doesn’t stop there. A broader understanding of Instrumentation — as a mindset — can mean that capability and efficiency are increased as learnings get incorporated into the product and process knowledge base. This is an often overlooked benefit — as not incorporating the learning means you are only getting a partial ROI on the development of digital products and services.
An Instrumentation mindset means expanding Instrumentation to include the development process itself. Process Instrumentation enables greater accuracy to tracking actual value creation, brings greater visibility into how well reality is following the plan, how best to get back on track, and ultimately providing a critical measure of progress towards your business goals.
As a product owner, there are many questions you may want to answer in order to really understand value, progress, and continuous improvement:
How well are we delivering against key business goals?
What is the true measure of quality and reliability being produced?
How accurate are our assumptions about customer value?
Are we working quickly but at the cost of efficiency?
How is our product investment distributed across new features, refactoring, and technical debt?
In order to help answer these questions, a Process Instrumentation approach looks deeper and more broadly into measuring the process itself. It tracks detailed process data and looks broadly across product creation to account for all of the work required to create a product. It connects individual tasks to business value, laddering up from the task level to business goals, such as bottom line revenue. It looks at the decision making process to identify areas that can be improved. This approach brings breadth and depth with the intent of creating a more holistic understanding of value, progress, and improvement.
Improving Process through Instrumentation Process Instrumentation is based on collecting information specifically to evaluate how well the process is working and understand what — if anything — needs to change. Your team is no longer relying only on a project plan as the sole basis of evaluating progress. You now have a wealth of metrics that can help you fine tune your process and — ultimately — deliver a quality product.
Better Visibility into Value Creation Connecting the day to day work and tasks of the product team to the highest level business goals can be one of the most challenging aspects of Process Instrumentation but brings some of the greatest insights. It requires collaboration far beyond the immediate product team and enormous discipline — teams can typically create their product without this accounting. However, without this connection, you can lose essential feedback needed to define, prioritize, and understand the real value of the product work.
A classic example of the difference between project progress and actual value created is evident in tracking velocity through Agile story points. Story points are given to different tasks or objectives, based on a relative scale of estimated difficulty. A rough sense of progress can be measured this way. However, story points don’t tell you how much value is actually being created, since estimates of relative effort are not factual descriptions of complexity involved or the value delivered. For instance, a single task might have actually been much harder than estimated. Once completed, important and complex work was done, but based on the original estimation, it may have only been worth a few points. Treating estimates as absolute measures can be deceptive.
The true measure of the day-to-day progress and value is when it can be tied to the strategy, goals, and market results of the product. For example, connecting epics and associated stories to the goals and desired outcomes of a product enables the team to look at any sprint and identify the business goals it supports, how much work went into the goals, and how well that sprint delivered against measured results. This creates a process lifecycle where the team can look back and identify how efficiently value is being created against the stated strategy and goals of the product. Of course as features are released they are evaluated on how they perform, based on their original expectations or goals.
A simple model connecting tasks to business value looks like this:
Task level user stories connect to feature level epics
Feature epics reference measurable product level goals, such as increased user retention
Product level goals are connected to measurable business results, such as increasing customer LTV
Business results are typically connected to higher level business initiatives or company vision
Connecting day-to-day product development with our business goals and results lets us connect our daily development effort to company value creation. This should be a central measure of how well a product team is doing but is often mired in the complexity of connecting across silos and requires discipline to maintain.
Better Visibility into Progress On a steady-state basis, project management coordinates, tracks, and monitors how the project unfolds against the plan. Often this is measured as degree of completion against initial estimates. Process Instrumentation enables you to go beyond this limited view into actual process management, which is based on understanding how much value was actually created, what accounts for the variance between the project plan and actuals, and whether or not changes need to be made to the process itself.
An Instrumentation mindset brings additional interpretive lenses applied more broadly throughout the process. For instance, a project manager might focus on a specific development milestone associated with the project plan, such as: was a sprint completed on time? If not, what percentage of work remains? While this gives one level of progress, it doesn’t indicate much about the true nature of progress, let alone value and how to improve. Process Instrumentation allows you to define what you measure, how, and why, for instance:
1. Tracking across Lifecycle Stages
To capture the work surrounding feature development, consider the lifecycle of a feature. Although many hours are spent interviewing customers or prototyping design variations, in many organizations, this work may not be codified as a task in a sprint. It can seem disconnected from the feature development. In fact, these bookends of implementation can be the majority of the work and are essential to understanding how well our product development process is working.
Some of the broader work components you can track throughout product lifecycle include:
Defining the business value of a feature
Prototype wireframing
Customer validation interviews
Defining performance metrics of a feature
Capturing usage metrics of a feature
The broadest set of product development tasks should be documented and tracked. Incorporating this effort into the development tracking gives is a much more realistic picture of the actual investment made to yield any level of progress.
2. Bespoke Process Metrics
To understand where you are day-to-day, there are many detailed metrics you can track in order to tell the real story of progress and value creation. Some easily tracked metrics (and the implications) include:
Is your team maintaining proper test coverage of the code base?
What is the true measure of quality being produced?
What bug:feature ratio was delivered?
Is the team working quickly but too inefficiently?
Are there refactoring hot spots when trying to extend features?
Is poor work quality affecting ability to deliver new value?
How often do deployments fail??
Is there a good handle on total product quality and reliability?
How large is each pull request?
Is your team making incremental, easy to test and review changes or are they pushing large, difficult to evaluate changes.
This is not an exhaustive list of what to track, and none of these would individually tell the entire story. One needs to be careful about absolute comparisons. However, taken collectively, they can help build a much more accurate picture of product progress and what might need to be adjusted.
For example, a continuous cycle of feature sprints followed by a sprint of refactoring, combined with an ever growing total bug list for the product might support the notion that the team is rushing to complete new feature sprints, only to need to go back and fix much of that rushed code indicating the initial sprints are too ambitious. Framed this way, it becomes much easier to develop a perspective on progress, backed by data, and adjust.
Ability to Target Improvement for the Product Process One of the challenges for effective Product Management is that often much of the foundational information and knowledge about the product’s development are not captured. As product team compositions change, external factors can change, the lack of understanding what happened along the way can create problems. Often this occurs because this kind of information isn’t captured by the tools being used, as it is about decisions, or analysis about problems that occured happens well after they have been solved.
Many decisions are made during product development. What is often not evident is why a decision was made, as well as the patterns around major decisions. Does managing too many decisions at once create problems or slow down progress? Do decisions lead to clarity and enable progress to surge? Being able to identify what, when, and why key decisions were made is key to understanding why a product is what it is.
Regular retrospectives are a great example of process information that can be instrumented if tracked on a regular basis. Retrospectives let teams and managers know what is working and what needs to be improved. Even simply tracking tallies and resolutions that come out of a retrospective and seeing how they apply to subsequent projects can give a product development function greater insights as to the effectiveness of the processes they use.
Ultimately tracking information needs to be easy, even seamless, and fit into your development process. Not everything needs to be tracked, but there needs to be more than velocity, you need to be able to look at our processes from multiple angles, only then can you correlate our progress with our actions and understand how to improve our approach.
Summary Software products and services are the lifeblood of a digital business, yet not every business has the experience or resources needed to build software products and services that actually work in the context of real-world complexity, scaling, and ROI goals. The best approach is always “eyes-wide open”, which needs to apply to the entire process, not just the initial investment decision. An Instrumentation-mindset helps everyone maintain an “eyes-wide open” stance and ensures the eyes are all seeing the same thing.
The process of developing software products and services becomes more effective when taking an Instrumentation-mindset from the beginning in order to deliver successful outcomes.
The key things that make up an Instrumentation mindset are:
Instrumentation is a way of doing and seeing, not just a tactical checklist item
Instrumentation is about making exploration more successful, not a shortcut that replaces exploration
Instrumentation is key to learning about a product use, performance, and is necessary to make smart decisions about next steps on the product road map
Instrumentation is key to the efficiency of the path taken to create the product, building product knowledge base, and improve product development capabilities
As businesses become more digital and need to make investments in building internal software product development capabilities, getting a Process Instrumentation mindset into the efforts early on is important. The long term value of Process Instrumentation can benefit any and every product or service development effort in which a business invests. Get in touch if you are starting to build internal capabilities. We’d be happy to discuss what we have learned and help you figure out how it can guide and inform your efforts.