Key Performance Indicator must start with business need

In a recent workshop that I attended on Agile Engineering Practices, as we began discussion on differentiating the roles of individuals for acceptance test driven development (ATDD) and test-driven development (TDD), question came up on what key performance indicators (KPI) are to be used. I offered my viewpoint that the approach to deciding what key performance indicator to measure is an incorrect strategy for any organization or business unit.

Let us define what a KPI is. My interpretation of KPI is that it is an objective measure of the efficiency with which a business process is executed to deliver incremental value effectively to a business need. By this definition, I see four contributing factors in two ordered pairs, which are as follows:

1. Business Need & Effectiveness (Objectives and Key Results [OKR] comes here)
2. Business Process & Engagement (Critical Success Factors [CSF] comes here)
3. Delivery and Operational Efficiency (Key Performance Indicators [KPI] comes here)

The OKR is a corporate level initiative that connects with the vision ("As is" to the "To be" state based on BHAG ideas). This may emerge from the portfolio function (identified in the business case) requiring what needs to be done across the organization (identified in the project/program charter) to align stakeholders, commit resources, and engage for success (that I call as ACE for charter). Consequently, each business unit comes up with its own critical success factors (CSF). In my opinion, the CSF is spread in at least 6 areas (Strategy, operations, product, marketing, finance, and people). Now, as programs and projects are chartered to deliver benefits through capabilities, enablers, and features in the product portfolio, the KPI measures the delivery and operational efficiency.  

If we conceptualize all these areas together, then the overlap between each domain (OKR & CSF, OKR & KPI, CSF & KPI) is the Key Result Areas (KRA) that all business units collaborate vertically and horizontally (this is where value comes). Every interaction is loaded with several external and internal risks and so are associated with the risk assessment. This is one of the reasons why products, projects, programs, and portfolio have risk (as measured by Key Result Indicators) as the central vehicle throughout the organizations (OKR & CSF & KPI & KRA).  Without understanding these interconnections, if we drive to use KPIs, then these are anti-KPI patterns because we are not measuring what matters!

 Dr. Rajagopalan's Conceptualization of the relationship among OKR, CSF, KPI, KRA, and KRI

Business needs differ across the organizations and within the business unit in an organization. For example, consider schedule variance (SV). This SV is a project management measure to see how the planned value (BCWP) of a project compares to the actual value for that project (BCWS).

While this SV is a very good measure of an individual progress, in regulated industries that manage portfolio or programs with multiple dependent subprojects where the control on regulatory boards are external constraints, the SV may not be an accurate measure of the performing organization’s project efficiency unless such delays are removed from consideration. So, if the goal is to measure the project health discounting such external influences, then, such measures may not accurately address the business need.

Now, when the right KPI is chosen based on what the organization’s or business unit’s goal is, we can shift the attention to the business process that is in place to address how efficiently it is serving up. For instance, if we use number of forecasted projects successfully launched to actual project launches when proper workflow, documentation, and versioning controls exist, then, the focus can become more objective to measure why tasks slipped, how projects were prioritized, etc.

As a result, whether it is a traditional software development setting or iterative software development, the focus should be on what is that we are trying to measure and why and then have the right processes and tools in place to collect data for analysis. Collecting exhaustive data accurately and applying an incorrect KPI will only lead to inaccurate assessment. For instance, measuring defects logged by tester would motivate the tester to log too many defects instead of consolidating them in one (like grammar, spelling, punctuation, formatting on a single line logged as multiple bugs). Everything countable does not always count, right! 

I would like to start a list of KPIs for a Project, Program, & Portfolio Management setting. This would be a great starter list. Please let me know what else should be added.

1. Schedule Variance and/or Schedule Performance Index
2. Cost Variance and/or Cost Performance Index
3. Number/Percentage of Projects Forecasted to Launched
4. Number/Percentage of milestones reached/missed by project
5. Number of FTE hours / project in a delivery-based schedule (no resource leveling)
6. % of projects coming behind schedule
7. Defect Density (Number of defects logged against requirements)
8. % of failure rate against test cases
9. % of test cases/requirements coverage
10. Number of risks against requirements/test cases
11. Prioritization of requirements/test cases based on risks
12. Task Progress (against DoD)
13. Extent of ATDD/BDD by Business Users (e.g.: exploratory tests)
14. Number of Escaped Defects
15. DORA metrics
16. Cycle Time, Lead Time, WIP
17. Estimation Accuracy
18. Commitment Evaluation (e.g.: Planned vs Actual Velocity)
19. Backlog Growth and Burn rate
20. Risk Reduction Rate (Identified, Treated by Response Type, Residual)
21. % projects on budget
22. % of Challenging Projects/Program or Portfolio
23. % of Change Orders $ to original SOW $
24. Internal Rate of Return
25. Net Present Value
26. Payback Period
27. Net Profit Margin
28. Capacity Utilization Rate (e.g.: Profitability per Project Management Resource)
29. Account Growth Revenue
30. Cash Conversion Rate (Revenue Recognition Frequency)
31. Concept to Cash Cycle Time
32. Marketing Metrics 
33. % of Complaints over Product/Program
34. Customer Satisfaction Index (e.g.: NPS)
35. Training Efficiency

Value of Product Governance & Portfolio Management in Agile Product Development

In a networking event on effective agile transformation, a question came up from a few learners in terms of the practices of project governance in capital project selection was agile or traditional development practice. The same question manifested differently in a PMI-ACP class I was facilitating where a learner expressed the challenges with not doing some proactive requirements gathering exercise around the extent of data exchange methods from the application service provider as part of the Agile approach to favor customer collaboration over contract negotiation. The learner reasoned that the later negotiations to add these requirements only hurt the organization with increased costs, reduced speed to market, and decreased customer satisfaction and organizational profitability.

There seems to be a common pattern that I think is evolving where practitioners of agile are forgetting the importance of Agile Manifesto. While Agile Manifesto was prohibiting extensive upfront requirements gathering for software product development keeping working software as a measure of progress, the Agile manifesto never eliminated documentation requirements. In fact, the stage-gate approaches promoted by Dr. Robert Cooper from the Product Development Institute (Cohn, 2010) takes product development from discover to post-launch review through the stage gates of idea development, business case development, development, testing, validation, and launch introduces multiple stage gates. Each stage gate becomes a major milestone serving as a go/no-go decision-making point.

A formal product selection criterion, also advanced by Project Management Institute’s Program and Portfolio Management Processes, for the organization to evaluate the key performance indicators to measure the return on investment (ROI) and use financial measures likes Internal Rate of Return (IRR), Net Present Value (NPV), & payback period to evaluate the profit potential, operating expenses, resource requirements is not prohibited by Agile Manifesto. Some product development where specialized expertise is outsourced to be competitive in the market, dependencies on external vendors that handle testing or audit procedures, use of distributed agile teams, or adherence to high quality management practices like CMMI may mandate that the standard operating procedure (SOP) is clearly outlined so that the teams productively work towards product development and are not burning unnecessary cycle times with delays and external dependencies. Not considering these things in advance by using Agile Manifesto as the shield to avoid “Contract Negotiation” or “Comprehensive Documentation” is not the failure of Agile.

Mike Cohn (2010), a pioneering contributor to the Agile Manifesto, recommends that any document or artifact required to achieve compliance be part of the product backlog and use of agile project management tool and practices for products in the regulated industries demanding compliance requirements. In a white paper submitted by Primavera, Oracle (2011) recommends the combination of Agile and Enterprise Project Portfolio Management (PPM) systems mitigates the risk in pharmaceutical and financial services industry (p. 3) increasing visibility of projects, resources, schedules and business value for the entire organization (p. 6).

It is therefore evident that our own understanding of Agile framework’s core principles may have to evolve when we apply agile in different industries. But, putting down project governance, product selection, and product portfolio management techniques as ineffective practices promoted by Agile is choosing to ignore the best practices. It is no wonder that topics of scaling agile are evolving leading to the latest description of Scaled Agile Framework (2013) for implementing scaled lean and hybrid agile framework in enterprises.

References

Cohn, M. (2010). Succeeding with Agile: Software development with Scrum. Boston, MA: Addison Wesley

Description of the Scaled Agile Framework (2013). Retrieved October 26, 2013, from http://scaledagileframework.com/purpose/

Oracle (2011). The Yin and Yang of Enterprise Project Portfolio Management and Agile Software Development: Combining Creativity and Governance. Retrieved October 27, 2013, from http://www.oracle.com/us/products/applications/primavera/primavera-eppm-agile-wp-453028.pdf